Skip to content

Monitoring GPU, RAM and CPU usage for slurm partitions and users

Notifications You must be signed in to change notification settings

ecs-vlc/iridis-light-monitoring

Repository files navigation

slurm-monitoring

Monitoring GPU, RAM and CPU usage for slurm partitions and users.

This is an app written in Python using flask. It gathers information using the standard slurm functions (squeue, scontrol etc.) The implementation assumes some fixed parameters such as maximum resource available to reduce the number of requests sent to the slurm server.

To use it go to http://degas.ecs.soton.ac.uk:7777/ with your VPN activated.

In the case that resources can be locked by a user and not actually being used, this can be monitored too. This is when a job requests resources from a node containing GPUs, but does not actually use them.

alt text alt text

Things to do in the future: 0. Add the new nodes and partitions a100.

  1. Create a dummy user to connect to iridis5.
  2. Talk to isolutions to setup a DNS address, sth like: iridis-light-monitoring.ecs.soton.ac.uk
  3. Add more stuff and improve the code.
  4. Improve the interface. For example on mobile, it looks a bit weird.

If the app does not work:

  1. ssh into degas@soton.ac.uk
  2. Inspect the output: journalctl -u iridis-light-monitoring.service --no-pager
  3. Make the code changes needed.
  4. Restart the service : sudo systemctl restart iridis-light-monitoring.service

About

Monitoring GPU, RAM and CPU usage for slurm partitions and users

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published