Extend Jupyter notebook use in the tutorials and containerize agents

[BjoernLudwigPTB]

Since the notebooks themselves do not help any further understanding. We will transfer the markdown parts into proper documentation pages and maybe manually add the output from running the scripts at the bottom.

[BjoernLudwigPTB]

@bangxiangyong @anupam-prasad We should overthink that. During the weekend it appeared to me, that Jupyter Notebooks are actually THE way to deploy the agents in a browser environment via a Docker container. We wouldn't even have to touch the agents' code to get a fully working browser based environment with just one single command for machines with Docker and docker-compose installed. docker-compose up is all that will be needed to be able to access for example the agents' notebooks at localhost:8888 and the dashboard after starting it at localhost:8050/dashboard for example or anywhere else at a given free port on localhost or in a web environemnt even configurable to be accessed at any other URL or port.

[bangxiangyong]

Thats a good comment, i do not have practical experience with docker thus yet, i'll need a simple demo to see how it works..

[BjoernLudwigPTB]

Thats a good comment, i do not have practical experience with docker thus yet, i'll need a simple demo to see how it works..

We will include that in the docs, but it boils down to installing Docker and docker-compose, clone the repo and then run docker-compose up before visiting the browser to get going in the notebooks and the dashboard. Basically we replace the necessities to install Python and agentMET4FOF by the necessities to install Docker, docker-compose and clone the repo. The gains will be possibility of web deployment and basically being completely platform independent, as long as Docker runs on the host machine (which delivers the web applications to any client in the given network).

[BjoernLudwigPTB]

By the way: since there is Docker for Windows and the Docker container which contains the agents would be Linux-based, the speed drawback on Windows should be no issue anymore.

[BjoernLudwigPTB]

In the Docker setup we should at least enable the possibility to connect to an existing agent network inside the container. We could do that by starting an agent network at a designated port, which we then expose to the host system and make it available at a certain port.

[BjoernLudwigPTB]

We will prioritize the notebooks to introduce nbval.

[BjoernLudwigPTB]

Now we worked quite a while on integrating nbvals error checking and got stuck. The checking works fine for a run on the command line locally after installing agentMET4FOF in editable mode with the extras_reuqire dependencies:

$ pip install -e .[tutorials]
Obtaining file:///home/ludwig10/code/agentMET4FOF
[...]
Successfully installed agentMET4FOF-0.6.4
$ pytest --nbval --current-env --sanitize-with nbval_sanitization agentMET4FOF_tutorials/tutorial_1_generator_agent.ipynb -v    
========================================= test session starts =========================================
platform linux -- Python 3.8.8, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/ludwig10/code/envs/agentMET4FOF-py38/bin/python
cachedir: .pytest_cache
rootdir: /home/ludwig10/code/agentMET4FOF
plugins: dash-1.20.0, timeout-1.4.2, nbval-0.9.6, cov-2.11.1
collected 1 item                                                                                                                                                                                          

agentMET4FOF_tutorials/tutorial_1_generator_agent::ipynb::Cell 0 PASSED                                                                                                                             [100%]

========================================== 1 passed in 5.01s =========================================

But it does not work with tox and it does not produce the expected result. Actually we noticed that the actual output we see, when executing the notebook with the backend osBrain is not what nbval receives as output from its' runs. Everything after SET STATE: Running is missing in nbval's run. With backend mesa it's even worse, since notebooks with that backend do not even contain this output in the cell on manual runs. That means, that with both backend in the current setup, it is not possible to use this method to check against expected output. We could though check for the expected output of the setup processes and the opener at least. For now, we keep our test results in the respective branch and PR #229 but still will have to think about how to proceed.

[BjoernLudwigPTB]

I finally did it, heureka! I am serving the agents in a Jupyter Notebook over the internet (at the moment temporarily at https://kletterni3d.de [do not ask why 😜]) and the dashboard alongside at http://kletterni3d.de/dashboard as soon, as it is started in one of the tutorial notebooks for instance. The code behind this is the current state of #215 . This is really exciting!

@bangxiangyong That's why I suggest to soon decide to merge #254 . It is already part of #215 .

[BjoernLudwigPTB]

Due to the security implications I switch off the server for now. If someone wants to try it out, let me know! 🤓

[BjoernLudwigPTB]

I was able to build the image in the pipeline and upload it from there to a dev release, which I created just to share the image. You can download the file docker_image_agentMET4FOF_jupyter.tar.gz as one of the release assets and then import it locally with

$ docker load -i <your/local/download/location/>docker_image_agentMET4FOF_jupyter.tar.gz

, run it locally with

$ docker run -p 8888:8888 -p 8050:8050 --rm agentmet4fof

After that, you will find the usual jupyter notebook output with a token on your command line, with a URL you can open in your browser. After starting a tutorial, you can then visit the dashboard at http://localhost:8050.

Let us know, if everything worked out for you!