Distributed Ensemble (MPI Support)

[Robadob]

The implementation of MPI Ensembles within this PR is designed for each CUDAEnsemble to have exclusive access to all the GPUs available to it (or specified with the devices config). Ideally a user will launch 1 MPI worker per node, however it could be 1 worker per GPU per node.

It would be possible to use MPI shared-memory groups to identify workers on the same node and negotiate division of GPUs, and/or for some workers to become idle, however this has not been implemented.

Full notes of identified edge cases are in the below todo list.

---

• Setup CMake
• Update CUDAEnsemble
  • Handle the 3 possible error configs
  • Do something about programmatic output logs (currently each copy will provide full RunLog, with those not handled empty of data, checking step counter > 0 is a hack, this presumably also affects failed runs under normal error modes)
    • Each runner has their own logs in RunLog (This requires an API break, as RunLog vector does not identify run index or RunPlan (it was initially assumed that vector would be parallel with RunPlanVector used as input). [Agreed with Paul 2023-07-26]
    • Rank 0 gets all the logs, others get empty
    • All runners get all logs
  • Catch and handle local MPI runs inside CUDAEnsemble ? (we want users to avoid using multiple MPI runners that have access to same GPU)
  • Should we expose world_size/rank to HostAPI? (I don't think it's necessary)
• Design a test case
• Test on Mavericks/Waimu (single node)
• Test on Bede (multi node)
• Document (readme)
• Document (userguide) Distributed Ensembles (MPI) FLAMEGPU2-docs#156
• Update telemetry to account for MPI
• Do something with ensemble example? (e.g. set CudaEnsemble::Config().mpi=false;)

• Do we need to handle a race condition with RTC cache?

Closes #1073
Closes #1114

---

Edit for visibility (by @ptheywood) - needs to be clear in the next release notes that this includes a breaking change to the return type of CUDAEnsemble::getLogs from a std::vector to a std::map

[Robadob]

I've created a simple test case that will either run on a local node (if worker count <= device count) or across multiple nodes (I could probably extend this to ensure it's 1 worker per node, but that would require some MPI comms to setup the test).

The issue with MPI testing is that MPI_Init(), MPI_Finalize() can only be called once per process. With CUDAEnsemble auto cleaning up and triggering MPI_Finalize() which waits for all runners to also call it. A second MPI test case cannot be run.

Perhaps an argument for Pete's CMake test magic, as I understand that runs the test suite for each individual test. Alternative would be to add a backdoor, that tells Ensemble not to finalize when it detects tests (and add some internal finalize equivalent to ensure sync).

Requires discussion/agreement.

Simplest option would be to provide a CUDAEnsemble config to disable auto finalize, and expose a finalize wrapper to users.

The only possible use-case I can see for distributed ensemble calling CUDAEnsemble::simulate() multiple times, would be a large genetic algorithm. If we wish to support that, then it will be affected by this too.

---

Changes for err test

Add this to FLAMEGPU_STEP_FUNCTION(model_step)

    if (FLAMEGPU->getStepCounter() == 1 && counter%13==0) {
        throw flamegpu::exception::VersionMismatch("Counter - %d", counter);
    }

Add this to the actual test body, adjust through Off, Slow, Fast.

        ensemble.Config().error_level = CUDAEnsemble::EnsembleConfig::Fast;

[Robadob]

https://mpitutorial.com/tutorials/running-an-mpi-cluster-within-a-lan/

Setting up MPI to run across mav+waimu seems a bit involved, probably better to try Bede. I would hope the fact it works on single node is evidence that it will work though.

[Robadob]

Happy for this to be tested on Bede and merged whilst I'm on leave. Functionality should be complete, may just want to test on Bede and refine how we wish to test it (e.g. make it ctest exclusive and include error handling test).

[ptheywood]

I'll review this and test it on bede while you're on leave, and try to figure out a decent way to test it (and maybe move mpi_finalize to cleanup or similar, though again that would mean it can only be tested once).

[Robadob]

As discussed with @ptheywood (on Slack), will move MPI_Finalize() to cleanup(), and replace with MPI_Barrier() (to ensure synchronisation before all workers leave the call to CUDAEnsemble::simulate().

This will require adjustments to the documentation and tests.

[mondus]

Need to add MPI to requirements list in README if option is enabled.

[ptheywood]

Also need to consider how this will behave with telemetry: flag indicating mpi use, number of ranks?, how to do the list of devices from each node etc.

(this is a note mostly for me when I review this in the near future)

[Robadob]

I can throw in a telemetry envelope at the final barrier if desired so rank 0 receives all gpu names.

[Robadob]

I've now added MPI to readme requirements and ensured all tests pass with local MPI and sans MPI.

Changes resolved