Details about the challenge are available at http://ai5gchallenge.ufpa.br.
Registration link: https://challenge.aiforgood.itu.int/match/matchitem/39
Datasets and the renderer (CaviarRenderer-ITU-v1) are available at https://nextcloud.lasseufpa.org/s/WYZAMbSbdocs2DL
Additional support material is available at https://nextcloud.lasseufpa.org/s/ZbB3cLN4MGZmwN4
The instructions below will guide you through the process of installing and using the Radio Strike software, including its baseline RL agent and associated resources.
-
First obtain the code and place it in a folder (we will assume here that the folder is called
radiostrike
). For instance, you can clone the repository and name the folder with:git clone https://github.com/lasseufpa/ITU-Challenge-ML5G-PHY-RL.git radiostrike
Later, when issuing commands, remember to first go to the radiostrike folder with:
cd radiostrike
-
Now download the data in folder
episodes_dataset\training
from https://nextcloud.lasseufpa.org/s/WYZAMbSbdocs2DL . This data is required by the Python code to generate the RL episodes. Make the corresponding CSV files (a total of 500 files) available at a folder calledradiostrike\episodes
. For instance, you should haveradiostrike\episodes\ep0.csv
,radiostrike\episodes\ep1.csv
, etc., assuming radiostrike is the folder in which you installed the repository. -
Install required packages. The Python packages that we use in this tutorial (numpy, pandas, etc.) are listed in the file
requirements.txt
.
The baseline agent code provides an example of a simple reinforcement learning (RL) agent for the RadioStrike-noRT-v1 environment, as adopted for the PS-006 ITU Challenge. The provided RL agent uses Stable-Baselines as its framework. Because of that, in our environment we need Python 3.6 and Tensorflow 1.14. Note that the participant is free to adopt a RL library other than Stable-Baselines and another deep learning framework (and eventually use another Python version).
If you are facing installation problems, please see the suggestions in other subsections below. In this subsection we assume you have successfully installed all required packages. Hence, you can train the baseline RL agent with:
python3 train_b-a2c.py -m <model_name> -ep <first_train_ep_id> <last_train_ep_id>
For example, to create an agent and save it in a file named baseline.a2c
, training it on the data from episodes 0 to 399, the command is:
python3 train_b-a2c.py -m baseline.a2c -ep 0 399
Recall that the provided data to create each episode must be available in folder episodes
. After finishing the training, the agent will be stored in the ./model
folder, created automatically by the end of the first run.
To test the baseline RL agent you can run:
python3 test_b-a2c.py -m <model_name> -ep <test_ep_id#1> <test_ep_id#n>
This will give you several output files in the folder ./data
named actions_<model_name>_ep<ep_number>.csv
, in which there will be information regarding the actions performed by the agent
at each simulation step for all the episodes passed. For example, a file can have:
chosen_ue | ue_index | beam_index |
---|---|---|
simulation_pedestrian1 | 2 | 32 |
simulation_car1 | 1 | 63 |
uav1 | 0 | 58 |
You can also get an output with more information by adding --full-output
flag. This way the output file name is not actions_<model_name>_ep<ep_number>.csv
, but output_<model_name>_ep<ep_number>.csv
:
$ python3 test_b-a2c.py -m <model_name> -ep <test_ep_id#1> <test_ep_id#n> --full-output
In this case, the output file will have several extra columns, as follows:
chosen_ue | ue_index | beam_index | x | y | z | pkts_dropped |
---|---|---|---|---|---|---|
simulation_pedestrian1 | 2 | 32 | 2.903349 | 33.66375 | 7.4372497 | 0.0 |
pkts_transmitted | pkts_buffered | bit_rate_gbps | channel_mag | reward |
---|---|---|---|---|
157.0 | 805.0 | 0.6255017349493768 | 0.41372613018058246 | 0.2502006939797507 |
For instance, to use a test set that is disjoint from the previous training set:
python3 test_b-a2c.py -m baseline.a2c -ep 400 500
or
python3 test_b-a2c.py -m baseline.a2c -ep 400 500 --full-output
Besides the RL agent, we also provide "dummy" agents for comparison purposes. There are two and they are called B-Dummy and B-BeamOracle. The B-Dummy agent assumes random action choices for both the scheduled user and which beam index to use. The B-BeamOracle agent follows a sequential user scheduling pattern (ue0-ue1-ue2-ue0-ue1-ue2...) and always uses the optimal beam index for the selected user.
Their operation is equal to the RL agent described above, and they are be called with:
$ python3 test_b-dummy.py -ep <test_ep_id#1> <test_ep_id#n>
or
$ python3 test_b-dummy.py -ep <test_ep_id#1> <test_ep_id#n> --full-output
for the B-Dummy, and
$ python3 test_b-oracle.py -ep <test_ep_id#1> <test_ep_id#n>
or
$ python3 test_b-oracle.py -ep <test_ep_id#1> <test_ep_id#n> --full-output
for the B-BeamOracle.
Their outputs are located in a specific folder inside /data
named ./data/dummy
and ./data/beamoracle
, respectively, and are called actions_b_dummy_<ep_number>
output_b_dummy_<ep_number>
in case of the B-Dummy, or actions_b_beam_oracle_<ep_number>
output_b_beam_oracle_<ep_number>
in case of the B-BeamOracle.
To analyze the output data from the agent and the dummies, several tools for plotting charts are provided in the file plot_tools.py
. You can use it as a class that can be called inside a script or in a quick and dirty way, in which you call it via terminal to plot all the charts available.
Usage:
$ python3 plot_tools.py -m <model_name> -ep <test_ep_id#first> <test_ep_id#last> --dummy --oracle
Example:
$ python3 plot_tools.py -m baseline -ep 0 1 --dummy --oracle
To use it, some information must be provided: first the name of the tested RL model (only the name without extension), second, the episodes to test, and finally, if the user will plot the B-Dummy and the B-BeamOracle results as well.
Obs: To plot any results, they must be obtained first by running the baselines.
Obs: The method .read_data()
is used to read the output data of the baselines from an episode or interval of episodes. In case you use the class instead of the terminal version, remember to use .read_data()
at the beginning of your script, before calling any plot method.
Most of the times one needs to fix dependencies and install specific versions of Python or its packages.
In this case, it is very convenient to use a virtual environment and better control the package versions. Before proceeding with the installation below, please check if you already have Python installed with:
python --version
Obs: Be aware: if you already have Python installed, for instance on Windows, but not in a specific virtual environment, sometimes it is better to uninstall it and start from scratch to avoid conflicts.
Obs: If you are using pip, it is useful to check with pip freeze
the versions of the installed packages.
We will share below some suggestions to install a given Python version. Recall that because we are adopting the Stable-Baselines RL library, we have adopted Python 3.6.8.
We will describe options for both Windows and Linux.
You must first download the Python executable for the version you chose. We are assuming it is Python 3.6 and we will get it from the official Python website (e.g. use this link to directly download for amd64: Python3.6.exe)
Obs: When starting the installation, check the box "Add Python 3.6 to PATH"
Install Pip
python -m pip install --upgrade pip
Install Pipenv
pip install --user pipenv
Because the provided Pipfile lists all dependencies, you can create the environment whith Python 3.6 and install all Radio Strike dependencies using:
pipenv install --python 3.6
Activate your environment
pipenv shell
You can check if you are using the correct python version with:
python3 --version
and also the correct packages:
pip3 freeze
With that you're ready to run the baseline.
To manage python versions we use pyenv. Note that there are many other alternatives to install Python versions. For instance, you can follow link.
To install pyenv, first install the Python build dependencies:
$ sudo apt-get update; sudo apt-get install make build-essential libssl-dev zlib1g-dev \ libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \ libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
then, install pyenv
with:
$ curl https://pyenv.run | bash
and add the following in ~/.bashrc
if you use bash, or ~/.zshrc
with you use zsh, to enable pyenv
in the terminal:
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"
Restart your shell so the path changes take effect:
$ exec $SHELL
In our case (to run our Stable Baselines), we download Python 3.6.8:
$ pyenv install 3.6.8
For setting up the environment and performing package management, we assume here pipenv.
$ pipx install pipenv
For other distributions refer to pipenv installation guide.
Because the provided Pipfile lists all dependencies, you can create the environment whith Python 3.6 and install all Radio Strike dependencies using:
pipenv install --python 3.6
Activate your environment
pipenv shell
You can check if you are using the correct python version with:
python3 --version
and also the correct packages:
pip3 freeze
With that you're ready to run the baseline.
Download the renderer CaviarRenderer-ITU-v1 from https://nextcloud.lasseufpa.org/s/WYZAMbSbdocs2DL .
AirSim needs to find a specific configuration file that we created for Radio Strike. Hence, you should download the AirSim settings file called settings.json and put
this file in a folder called Airsim
, within the folder Documents
. According to the AirSim
documentation, this
AirSim subfolder is located at Documents\AirSim on Windows and ~/Documents/AirSim on Linux systems.
At this point the CaviarRender can be executed, to visualize the results in the 3D scenario the Render mode needs to be enabled in the configuration menu of CaviarRender.
In order to have a faster execution, it is possible to disable the Unreal Engine graphics by changing the AirSim settings ViewMode.
To do that, change "ViewMode": "" to "ViewMode": "NoDisplay" in the file settings.json
mentioned above.
CaviarRenderer-ITU-v1 performs two distinct things:
- After having trained a RL agent, CaviarRenderer-ITU-v1 is used to visualize the RL agent actions in the 3D scenario. It indicates the beam directions and also the scheduled user (by drawing a bounding box around it). To execute it:
With the 3D scenario running it is possible to visualize the results using the caviar_render.py script. This script runs one episode per execution and it is necessary to set the episode path in the begin of the script and also the output files with the RL results.
python3 caviar_render.py
The output of this script is a video with the simulation result. It is currently not feasible to visualize the 3D scenario along the RL training stage. But this is in our "to do" list.
- CaviarRenderer-ITU-v1 can also be used to create additional episode data, storing the information as .CSV files. You can of course use it to create more episodes for your research, but you cannot use any extra episodes when training your model for the PS-006 ITU Challenge. The rules restrict the models to be trained only with the provided episodes.
The first step to generate new episodes is to create a set of trajectories to be used by the AirSim's UAVs.
The trajectories available here can be used, or new trajectories can be generated using the random_generator.py script. It is possible to define the number of trajectories to be generated by the script.
cd waypoints/
python3 random_generator.py
With the trajectories ready, the CaviarRender should be executed and, in its menu, choose to disable its Render mode. Now it is possible to run the ep_generator.py script.
python3 random_generator.py
The number of episodes, sample rate, trajectories paths and others configurations can be changed in the being of the ep_generator.py script.
The dataset is provided in .csv
files, that are in the folder episodes
(i.e ep0.csv
, ep1.csv
etc). Each episode
has approximately 3 minutes of duration, with information stored with a sampling interval of 10 ms. The csv is composed by the following columns:
timestamp | obj | pos_x | pos_y | pos_z | orien_w | orien_x |
---|
orien_y | orien_z | linear_acc_x | linear_acc_y | linear_acc_z | linear_vel_x | linear_vel_y |
---|
linear_vel_z | angular_acc_x | angular_acc_y | angular_acc_z | angular_vel_x | angular_vel_y | angular_vel_z |
---|
There are three different types of objects: uav
, simulation_car
and simulation_pedestrian
. Only the uav
type has information in all columns, while the others have only information regarding their position and orientation.
Airsim uses a coordinate type other than unreal, called NED coordinates system, i.e., +X is North, +Y is East and +Z is Down, and the units are in S.I. different from the Unreal Engine which uses centimeters. For more information see Airsim Documentation.
An episode has information regarding all mobile objects in a scene (all pedestrians, etc.). To keep it simple, we assumed that the baseline RL agent uses only information from the three users (uav1
, simulation_car1
and simulation_pedestrian1
). These are the user equipment (UEs) being served by the base station (BS). Hence, the baseline data was obtained by filtering the original episodes to discard the information about all other mobile objects (which are scatterers, not UEs).
The participant must provide:
-
The predicted output for the test set as a CSV (comma-separated values) file.
Obs1: This output can be generated with
test_b-a2c.py
, you just need to pass your model in the argument (the name "b-a2c" comes from the baseline, but you're free to use other RL strategies).Obs2: This output must contain at least the actions of the agent, namely the scheduled user ID and the chosen beam index, for each time step from the test set.
-
The participant must also provide two video files:
- one with the rendering of the RL agent for the episode with the largest value for the return
G
; - and another for the episode with the smallest value of
G
.
Obs1: These two videos should correspond to the output of CaviarRenderer-ITU-v1 (or later version). If desired, the participant can concatenate the two into a single video.
Obs2: Another option is to expand the video (or videos) by promoting the designed RL agent, describing its architecture or any other thing the participant thinks is interesting. This promoting video must be recorded in Portuguese or English (the organizers can generate the videos for the participants if needed).
- one with the rendering of the RL agent for the episode with the largest value for the return
The participant with the best return G
over the test episodes wins. In case of a draw regarding the average return G
among distinct participants, the tie will be broken using the best video(s).
Note that for the evaluation, the return G
is going to be calculated with the same reward as the baseline agent (
B-A2C
)
where ,
, and
correspond, respectively, to the total amount (summation for all users) of transmitted, dropped and buffered packets at time
.
Obs1: The reward is restricted to the range
. At each time
, a single user can be served, but
accounts for the number of packets in all three buffers. Hence,
only if all buffered packages of the scheduled user are transmitted, while the buffers of the other two users were empty.
Obs2: The participant is free to use other rewards, states and RL methods (DQN, A3C, etc.) in the agent design and training.