For each ROS2 core concept, we developed several tests for evaluating its performances.
iRobot ROS2 Performance Evaluation Framework
For a whole system performance evaluation, we have developed benchmark applications
iRobot ROS2 Benchmark applications
These are the main metrics that we are interested in:
- Latency
- Reliability
- CPU usage
- Memory usage
Due to our specific application scenario and to the new ROS2 multithread capabilities, we made most of the evaluations running all nodes in a single process. Evaluations have been performed on standard x86_64 laptops as well as on embedded platforms.
If you want to repeat the experiments, eventually with different parameters, follow the Build Instructions.
The goal is to develop a set of experiments which are easily reproducible and provide useful insights on the ROS2 performances. This allows to easily repeat them as soon as a new ROS2 distribution is released.
For each ROS2 distribution we aim to:
- Test all the ROS2 features.
- Find limits/bugs.
- Perform regression tests with respect to the previous distribution.
Early results are suggesting us that this is the correct way to go, as we are spotting the strengths and weaknesses of ROS2 and helping the ROS2 community to grow, by signaling and fixing bugs to the ROS2 core implementation.
We tested ROS2 on many different systems. In order to do that we implemented tools that allow users to easily create any number of nodes with the same characteristics as well as tools for manually crafting your own graph topology and measure its performances.
This repository also contains some Docker-based tools for cross-compiling ROS2 SDK and single packages for RaspberryPi.
You can find detailed instructions in the cross-compiling directory.
Here you find links to the experiments for the most recent ROS2 distribution.
Each README file contains a description of the experiment, instructions for reproducing it and the results obtained.
| Experiment | README |
|---|---|
| Benchmarking applications | README.md |
| Discovery time | README.md |
| Pub/Sub Latency | README.md |
| Pub/Sub Reliability | README.md |
| Pub/Sub Memory usage | README.md |
| Pub/Sub CPU usage | README.md |
| Client/Service systems | README.md |
| Bouncy-Crystal regression test | README.md |
ApexAI provides an alternative valid performance evaluation framework, which allows to test different type of messages. Our implementation is inspired by their work.
- Evaluation of ROS2 Communication Layer
- ROS2 latency and throughput
- Evaluating QoS performances using ROS demos
- Exploring the performance of ROS2
- Time-Sensitive Networking for robotics
- Real-time Linux communications: an evaluation of the Linux communication stack for real-time robotic applications
- Towards a distributed and real-time framework for robots: Evaluation of ROS 2.0 communications for real-time robotic applications
- Time Synchronization in modular collaborative robots
- Reduce delay and jitter in wireless ROS1 networks