Reinforcement Learning is the computational approach to learning from interaction (Sutton & Barto). Autonomous agents performing goal-oriented learning based on experience is the holy grail of AI. Recent successes of Reinforcement Learning algorithms include human-level performance on many Atari games, beating world's best Go player, and robots learning dexterity and grasping. Despite these successes, industrial applications of RL outside of organizations with easy access to large-scale compute and software infrastructure remain sparse. Wide-spread applications of RL require more sample efficient algorithms and new software tools for doing distributed computing. Our goal with this tutorial is two-fold:
Provide a solid conceptual foundation for understanding and evaluating the existing state of the art RL methods (both strengths and weaknesses).
Showcase Ray, an emerging distributed execution framework, for experimenting with and deploying large-scale Reinforcement Learning algorithms.
We believe that RL works well on applied problems where: 1.) Gathering simulations and experience is cheap, e.g. games, narrow-domain robotics; 2.) Modellers have access to a compact representation of the environment dynamics to apply approximate dynamic programming. During the tutorial, we will also deep-dive into practical applications of RL such as optimizing tax collections with constrained value iteration, predicting market micro-structure for better trade execution, and others alike.
While portions of the tutorial will focus on conceptual foundations and applied case studies, we will do hands-on implementations of the algorithms on standard Reinforcement Learning environments for debugging and pedagogical purposes.
- Create a directory for this workshop with
mkdir rl-kdd
. cd
into yourrl-kdd
directory. Create avirtualenv
associated with the tutorial. Checkout more detailed instructions for OS specificvirtualenv
creation here.- Activate the previously created
virtualenv
. - Clone this repo.
cd
into the cloned repo. While thevirtualenv
is active,pip install -r requirements.in
to download the associated packages.- If you are having issues with
gym[atari]
installation, make sure you have the following dependencies.
- If you are having issues with
- Test installation works properly by:
- Running
python tests/random_agent.py
. - Running
python tests/rllib_test.py
.
- Running
Side Note: Make sure you are using python 3.6.4
If personal computer is not working for you for any reasons, try out JupyterHub. Windows not supported for Ray so if you have windows then JupyterHub is the default.
Log-In with your JupyterHub at https://appliedrl.hubhero.net/. Notes:
- On First Login, wait a bit (around 5-10 mins). The UI doesn't give instant feedback.
- Don't edit the solutions. Edit the starter code notebooks. If you do edit the solutions, I'd recommend users copy the notebooks over to their home directory if they want to edit them. The git puller will not update a file that has been locally modified, so if you modify notebooks you will have to tell them to copy & delete it if they want to get a newer version. This shouldn't be that big of a deal :)
Once you are into the Hub, open up terminal and pip install matplotlib
and pip install seaborn
.
This tutorial will require a solid foundations in Machine Learning, Python Programming, and basic familiarity with popular Deep Learning frameworks like Tensorflow and Torch.