Jittered output from depth_image_proc register node

[wouterio]

Hi,

We have a depthai camera shooting color images at 5Hz and depth images at 10Hz. The camera node is connected to a depth_image_proc register node that registers the color and depth images. The register node is connected to a third node that does something with the registered depth images. We added diagnostics to the camera node and the third node for logging message frequency and delay.

This pipeline works well for about 40~70 mins. After that it starts dropping messages. I added a plot below showing the delays from the camera node's outgoing messages, the third node's incoming registered depth images (orange line), and the third node's outgoing messages (purple line). I couldn't add the message delays from the register node itself because it doesn't support diagnostics, so adding the third node's incoming message delays was the next best option.

A bit after 4300 seconds, the plot starts showing numerous gaps between the third node's incoming registered depth messages, as well as big spikes in message delays. The camera node keeps pretty stable. I also don't think that the third node causes trouble as it doesn't take much computation. So it looks like something goes wrong in the register node? But we're unsure what it could be. Any ideas?

Thanks in advance.

Also see: https://robotics.stackexchange.com/questions/113272/jittered-output-from-depth-image-proc-register-node

[mikeferguson]

Given the simplicity of most of the code in these components, I expect this is not something to be fixed in image_pipeline itself but actually something in one of the following:

• DDS layer
• message_filters
• rclcpp executor
• something else running on the system (did the system load spike at the same time? there's nothing in these nodes that ensures real time compliance)

[wouterio]

Hi Mike, thanks for your response. I'll have a look at your suggestions. For now I know we use two DDS configurations. The one we use for our robots is slightly more complicated. The other is just the plain ros2-out-of-the-box config. Both show the same jittered pattern. I guess the register node using an approximate time synchronizer?

There is something with the cpu load going on, but I'm unsure about the chicken and the egg. It makes sense that the register node might drop messages during high cpu load, but there's no clear relation.

In the plots underneath you see that the cpu load (top plot) at some point starts spiking (for unknown reason) and at some point the register nodes starts dropping messages (bottom plot):

But here you see massive cpu load (each cpu plotted individually) while the register node is just fine.

And here just relatively low cpu load while the register node drops messages:

So it's unclear to me how the cpu load relates to the register node dropping messages. But most of the time you'll see the spiky cpu load in combination with dropping messages. And still in all these plots it looks like the camera runs stable.

Not sure if this is still the right place to dive into this matter. Perhaps I should post this at the messages_filters issue tracker as well.