Allow control over which executor is used for recording and/or consider changing the default

[adamdbrw]

Description

rosbag2 uses SingleThreadedExecutor and runs spin() for recording. This is not great for some cases, notably when recording streams with high number of executables per second, since spin() consumes a lot of CPU in such case: ros2/rclcpp#1637. This is especially the case with rosbag2 record, which:

• should not care that much about minimizing latency for callback execution (a few ms added is probably ok).
• certainly does not want to minimize latency at the cost of high CPU use, since it disturbs the recorded system running on the same machine (e. g. an automotive stack)

Changing the spin() in recorder to spin_some() or spin_all() + a very small sleep strictly improves performance in all important metrics (for myself at least) (a drop to half the CPU use).

Related Issues

ros2/rclcpp#1637
#679

Completion Criteria

API is extended to allow different executors (or different use of executors, some lambdas?) and/or default spinning is changed.

Testing Notes / Suggestions

rosbag2 performance benchmarks can be used to run a desired setup of publishers easily.

[emersonknapp]

I think it would be acceptable to just change the default behavior to the better-performing one.

Adding a new configuration option adds complexity that I don't think we need yet.

[adamdbrw]

@emersonknapp I proposed a minimal (in LOC) but impactful PR

[MichaelOrlov]

• Closing as wouldn't be done since it was proved here The callbacks of subscribers for records are executed in different threads #679 (comment) that Multithreaded executed with the current cache mechanism will not give a tangible gain in performance.

• The root cause rather in the suboptimal implementation of the SingleThreadedExecutor which is going to be improved soon by not recreating waitset entity after each callback.

• Also performance issue in the should_split_bagfile() was addressed for the MCAP file format.