Code for performing membership attacks and enforcing differential privacy on reinforcement leanring algorithms. The reinforcement leanring algorithms are implemented in rl-starter-files and the environments are from gym-minigrid. We modified these libraries and included them in our repository:
@misc{gym_minigrid,
author = {Chevalier-Boisvert, Maxime and Willems, Lucas and Pal, Suman},
title = {Minimalistic Gridworld Environment for OpenAI Gym},
year = {2018},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/maximecb/gym-minigrid}},
}
@misc{rl_starter,
author = {Lucas Willems},
title = {RL-Starter-Files},
year = {2018},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/lcswillems/rl-starter-files}},
}
@misc{torch_ac,
author = {Lucas Willems},
title = {PyTorch Actor-Critic deep reinforcement learning algorithms: A2C and PPO},
year = {2018},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/lcswillems/torch-ac}},
}
See requirement.txt, please run the code under python 3.7 or later.
Run
pip install -r requirement.txt
If you haven't install gym-minigrid, please do the following:
cd gym-minigrid
pip install -e .
If you haven't install torch-ac, please do the following:
cd torch-ac
pip install -e .
If you have problems with runing the code, please try to build a virtual environment and then run the commands above to install required pakages in the virtual environment:
pip install virtualenv
virtualenv venv
cd venv
source bin/activate
cd ..
-
MiniGrid-FourRooms-v1
- Four room environment where the agent always starts from room 1. -
MiniGrid-FourRooms-v2
- Four room environment where the agent always starts from room 2. -
MiniGrid-FourRooms-v3
- Four room environment where the agent always starts from room 3. -
MiniGrid-MultiRoom-N2-v00
- Multi-room environment with seed range [0, 4], each seed correspondes to a unique setting (eg. room size, room position, agent starting position, target position, etc.) -
MiniGrid-MultiRoom-N2-v0
- Multi-room environment with seed 0. -
MiniGrid-MultiRoom-N2-v1
- Multi-room environment with seed 1. -
MiniGrid-MultiRoom-N2-v2
- Multi-room environment with seed 2. -
MiniGrid-MultiRoom-N2-v5
- Multi-room environment with seed 5. -
MiniGrid-MultiRoom-N2-v6
- Multi-room environment with seed 6. -
MiniGrid-MultiRoom-N2-v7
- Multi-room environment with seed 7.
Train, test, save trajectories on MiniGrid-FourRooms-v1
environment, using PPO algorithm. Please cd rl-starter-files
to run the following examples.
1, Train the agent on the MiniGrid-FourRooms-v1
environment using PPO.
python -m scripts.train --algo ppo --env MiniGrid-FourRooms-v1 --model FourRoom1 --frames 204800
2, Test the agent on the MiniGrid-FourRooms-v1
and save the trajectories.
python -m scripts.train --algo ppo --env MiniGrid-FourRooms-v1 --model FourRoom1 --frames 409600 --test 1
The trajectories are saved in rl-starter-files/storage/FourRoom1/probabilities.csv
1, Visualize agent's behavior:
python -m scripts.visualize --env MiniGrid-FourRooms-v1 --model FourRoom1
2, Visualize the reward plot and other plots:
tensorboard --logdir storage/FourRoom1
Follow the instructions on terminal and open https://localhost:6006/ on the browser to view the plots.
3, Visualize the multiple plots simultaneously:
Create a new directory rl-starter-files/storage/fr_plots/
and copy events.out.tfevents files to fr_plots/
directory. An example is here. Then run:
tensorboard --logdir storage/fr_plots
More examples and visualizations can be found on the README in gym-minigrid and rl-starter-files
To apply the Gaussian Privacy Module to the trained agent and observe the corresponding utility loss, run
python -m scripts.evaluate --env MiniGrid-FourRooms-v1 --model FourRoom1 --sigma 0.1
Note that you can change the name of environment and model. Sigma is the noise variance of the Gaussian Privacy Module.
Membership attack on RL models under MultiRooms environment: Reinforcement_Learning.ipynb
Cifar 10 Dataset that can be imported in Tensorflow or Keras.
@misc{multi30k,
title={Multi30K: Multilingual {English-German} Image Descriptions},
DOI={10.18653/v1/w16-3210},
journal={Proceedings of the 5th Workshop on Vision and Language},
publisher={Association for Computational Linguistics},
author={Elliott, Desmond and Frank, Stella and Sima’an, Khalil and Specia, Lucia},
year={2016}
}
If you haven't install Jupyter Notebook, please go visit here for installation. If you use pip
, you can install it with:
pip install notebook
To run the notebook, go to the directory where your notebook locate at and run the following command at the Terminal (Mac/Linux) or Command Prompt (Windows):
jupyter-notebook
The directory where your jupyter notebook located will be displayed on your browser. Then, please click on the notebook (.ipynb) and run the codes in order.