Add a document for ros_time

[tfoote]

This adds a document for the ros time abstraction.

This also has a few cleanups of the config, and supports local testing using the official jekyll docker image.

[wjwwood]

Why?

[tfoote]

That's how jekyll is designed to be used, it's mostly to allow github-pages to render properly for project prefixes.

See: https://jekyllrb.com/docs/github-pages/ and
https://byparker.com/blog/2014/clearing-up-confusion-around-baseurl/

[wjwwood]

That must have changed (again), because that wasn't the case before. Just make sure double check that the live site is working correctly after merging.

[wjwwood]

Thanks for taking some time to write this out.

Maybe we need to have a section about pausing? Was that supported in ROS 1's simulated time? How should it work, i.e. will it rely on publishes to /clock to stop or is there a separate mechanism for pausing? Are there corner cases like what happens if you pause and then the next time update is in the past?

Also I think we could spend some time thinking about blocking of the time function. For example, should it block while time going backwards callbacks are being executed or just return the latest time? I think it might be undesirable to have the get current time function block under some (asynchronous) conditions.

[jacquelinekay]

What is expected to happen when I have two publishers to the /clock topic that are interleaved (same frequency, different start times)? I've never tried this in ROS 1. What can be done to prevent unexpected behavior here, or at least warn the user?

[gbiggs]

If there is a way to gather information about the topology of the system, specifically who is publishing on what topics, then at the very least it should be possible to add something to roswtf to catch this problem.

[tfoote]

Two conflicting publishers is going to lead to bad results. It certainly can be rolled into roswtf, and the subscribers should probably warn as well.

[jacquelinekay]

I tweaked some spelling and grammar in branch ros_time_jackie: https://github.com/ros2/design/tree/ros_time_jackie

[dirk-thomas]

The whole aspect of time synchronization seems not relevant to me in this article. The main goal is to allow the time to go slower / faster / be paused. Therefore I would propose to remove this reference here as well as in the next section.

[gbiggs]

Doesn't the idea of allowing the rate of change of time to be altered inherently introduce the idea of a synchronised clock across each node? At least for nodes that do calculations involving time.

[tfoote]

Our default behavior of falling back to the system time requires them to be in sync. If we did not want to require that we would have to publish /clock all the time. This is here to remind people that in the default case synchronization is necessary.

[dirk-thomas]

Please rebase to validate against the latest linters.

[gbiggs]

You need to define "real time" here. I assume that you're talking about running in "world time" (I don't actually know if there's an agreed-upon phrase to describe that), not meeting deadlines.

[tfoote]

I changed all usages to be 'real time' and added a note to differentiate 'real time' from requirements for deterministic computing requirements.

[wjwwood]

The layout of the text makes "these cases" sort of ambiguous. I'd suggest describing which cases directly here instead.

[wjwwood]

I had some comments, but they're mostly questions clarifying the language. This is looking pretty good. Thanks for working on it @tfoote.

[tfoote]

@wjwwood I added some clarifications and a todo

[ahundt]

You may want to consider reading papers on time synchronization, particularly if you want to make ros2 usable for those with realtime constraints. One example use case is dealing with data from a pair of stereo cameras which need to be sent signals to remain synchronized to an accuracy within nanoseconds for the most accurate depth data possible. In particular, it would be ideal if you can expose time uncertainty in addition to the time itself.

Another useful use case is google's global time synchronization database paper.

Finally, here are general time synchronization google scholar search results.
https://scholar.google.com/scholar?hl=en&q=time+synchronization&btnG=&as_sdt=1%2C21&as_sdtp=&search_plus_one=form

[wjwwood]

You may want to consider reading papers on time synchronization,

I'm not sure what leads you to believe we haven't, but for those who are interested, we looked at many things in addition to the Google paper you linked to (in case anyone else is interested in the subject):

• The NTP RFC's:
  • https://www.ietf.org/rfc/rfc5905.txt
  • http://www.ietf.org/rfc/rfc5906.txt
  • http://www.ietf.org/rfc/rfc5907.txt
  • http://www.ietf.org/rfc/rfc5908.txt
• A new algorithm called TICSync:
  • http://www.robots.ox.ac.uk/~mobile/Papers/2011ICRA_arh.pdf
• The chrony documentation:
  • http://chrony.tuxfamily.org/comparison.html
  • http://tldp.org/HOWTO/Clock.html
• Clock Synchronization for Real-Time Industrial systems:
  • http://cdn.intechopen.com/pdfs-wm/12259.pdf

I think our perspective, which we're trying to convey with this document, is that the ROS 1 time interface is pretty good, in that it has been used successfully in many use cases thus far. However, we found a few places where it could be improved, like exposing the update mechanism through a programmatic extension point (whereas in ROS 1 you could only update it with the /clock topic). The other thing we're thinking of adding is the ability for the user to get a callback when the simulated time is changed or goes backwards.

In my opinion providing an interface similar to the interface provided by the POSIX API, or the C++ chrono API, is a good goal for the simulated time interface that way it can be a drop in replacement when using a normal clock in many situations. Any other extensions to control or slew time, or measure uncertainty in time, should in my opinion either be a transparent side-effect (like NTP or chrony) or through a side channel separate from the normal interface.

If you've got experience with time synchronization, then please help us by making concrete suggestions as to how we might change the interface and provide more functionality or flexibility without compromising the existing interface's usability and familiarity.

As to your comments you've already made:

particularly if you want to make ros2 usable for those with realtime constraints.

I don't see any way the time interface needs to be changed in order to support realtime scenarios, but if you have a suggestion, please make it. We've given a lot of thought to this, so for example, getting the time with the rcl interface (our C API for ROS 2), it is accessed with by an atomic int64, so on modern processors where ATOMIC_LLONG_LOCK_FREE is true the access of the current time, simulated or not, will be lock free and thread safe. Also we provide a portable steady clock and a system clock in addition to the simulated clock (assumed to behave like the system clock but flow differently).

One example use case is dealing with data from a pair of stereo cameras which need to be sent signals to remain synchronized to an accuracy within nanoseconds for the most accurate depth data possible.

I believe that is best solved by triggering the camera's electronically (with a GPS PPS for example) and using sequence numbers to synchronize them, but that's up to the person integrating the cameras and not the role of the framework, I think. In all my experience and research the only thing I feel comfortable saying is that there is no single system that meets everyone's timing needs. So I think the appropriate role of ROS is to provide a single decent way of simulating time (for common cases like playing back log data and integrating with simulators) and let others use a different method if they need. I'm not aware of any proposed time system that cannot be used in combination with ROS 1 or the proposed design for ROS 2. If you're aware of a system that could not be used with ROS 1 or used with the proposed clock simulation mechanism for ROS 2, then please let us know by describing it and what about the current design is in conflict.

In particular, it would be ideal if you can expose time uncertainty in addition to the time itself.

Can this not already be exposed with a side channel, separate from the normal "get time" mechanisms? Can you propose a mechanism that better supports this use case while also supporting use cases without this metric (I'd argue far more common)? Or a more fundamental question: Does the framework really need to solve this problem?

[ahundt]

Thanks for the detailed response! I'm just trying to contribute based on reading the document and request for contributions on the ros2 website, I wasn't trying to assume what you may/may not know. Sorry if it came across in another way! :-)

could ros2::Duration duration = ros2::time::now() internally store a time and an uncertainty parameter and provide access to it without disrupting current APIs? For example in addition to duration.toSec() one could also call duration.uncertainty().

Can this not already be exposed with a side channel, separate from the normal "get time" mechanisms?

The difficulty with a side channel is once you have hundreds of packages using one time function it is incredibly difficult to change it all out to use another! That's why I made my suggestion.

http://www.robots.ox.ac.uk/~mobile/Papers/2011ICRA_arh.pdf

I was actually looking for that TICSync paper while I was writing this post but couldn't recall the name! I'm glad you already knew about it.

Or a more fundamental question: Does the framework really need to solve this problem?

In absolute terms, of course not. But if it did at such a low level of the system from the beginning? Boy would that make a huge difference when engineering real systems!

I believe that is best solved by triggering the camera's electronically (with a GPS PPS for example) and using sequence numbers to synchronize them, but that's up to the person integrating the cameras and not the role of the framework, I think.

Yes, that's definitely the way to sync the frames when you have people with the hardware and expertise to do so! If you've not got that... the next best thing would be if the error in your measurements could be easily quantified by ROS2 Time. :-)

Thanks for your consideration!

[wjwwood]

Thanks for the detailed response! I'm just trying to contribute based on reading the document and request for contributions on the ros2 website, I wasn't trying to assume what you may/may not know. Sorry if it came across in another way! :-)

My comments were meant to come across neutral, not negatively. I also apologize if they seemed to be. We do appreciate your interest and questions.

could ros2::Duration duration = ros2::time::now() internally store a time and an uncertainty parameter and provide access to it without disrupting current APIs? For example in addition to duration.toSec() one could also call duration.uncertainty().

Can this not already be exposed with a side channel, separate from the normal "get time" mechanisms?

The difficulty with a side channel is once you have hundreds of packages using one time function it is incredibly difficult to change it all out to use another! That's why I made my suggestion.

From a logistical perspective, I think it would be better to not integrate a query like this into the core Time type. For one reason, who's to say that this is the only metadata about time we'd want to store and query (which goes back to my argument about there not being a silver bullet solution that would work for everyone). Another issue is that I don't know if we could always deliver this information to the user (how do you calculate or get the uncertainty for the system clock). I'd rather see an interface like (taking TICSync as an example) auto uncertainty = ticsync::get_uncertainty(), or if necessary a function that gets both the uncertainty and the time at the same moment:

ticsync_ros::ROSTimeWithUncertainty ticsync_time = ticsync_ros::now();
float uncertainty = ticsync_time.uncertainty();
float now = ticsync_time.ros_time.to_secs();

To your point that such an API as I just suggested would not work because you'd need to change all the code that already uses plain "ROS time", I'd say that I don't believe that is a limitation because even if uncertainty were in the core API we cannot force people to use it (I don't think anyways). I'm also not convinced that everyone would need to use the API which contain extra metadata like uncertainty.

Now, I don't even know if uncertainty is something TICSync could give you, but procedurally, I think having layered API's like I demonstrated above is the way to go until one API becomes the obvious solution for pretty much everyone.

That being said, I'm open to discussing individual extensions to the Time API if you can argue why it needs to be in the core API and cannot not be easily layered on top and used by those who need it. A good example of this is simulated time. The ROS Time type which can either give simulated time or time from the system clock is needed to facilitate simulation and log playback situations. It's possible a similar set of use cases surrounding data synchronization or interpolation would need something like uncertainty. But to be elevated to the core API I think it needs to two qualities: it needs to affect the behavior of the ROS time constructs for all code that uses it and it needs to have a reasonable fallback behavior when not in use. The simulated time object (ROS Time) both needs to affect many existing calls to the time API (but not all which is what "wall time" is for) and it behaves reasonably when simulated time is not being used.

[gbiggs]

On 2016/04/17, at 13:54, William Woodall notifications@github.com wrote:

Thanks for the detailed response! I'm just trying to contribute based on reading the document and request for contributions on the ros2 website, I wasn't trying to assume what you may/may not know. Sorry if it came across in another way! :-)

My comments were meant to come across neutral, not negatively. I also apologize if they seemed to be. We do appreciate your interest and questions.

could ros2::Duration duration = ros2::time::now() internally store a time and an uncertainty parameter and provide access to it without disrupting current APIs? For example in addition to duration.toSec() one could also call duration.uncertainty().

Can this not already be exposed with a side channel, separate from the normal "get time" mechanisms?

The difficulty with a side channel is once you have hundreds of packages using one time function it is incredibly difficult to change it all out to use another! That's why I made my suggestion.

From a logistical perspective, I think it would be better to not integrate a query like this into the core Time type. For one reason, who's to say that this is the only metadata about time we'd want to store and query (which goes back to my argument about there not being a silver bullet solution that would work for everyone). Another issue is that I don't know if we could always deliver this information to the user (how do you calculate or get the uncertainty for the system clock). I'd rather see an interface like (taking TICSync as an example) auto uncertainty = ticsync::get_uncertainty(), or if necessary a function that gets both the uncertainty and the time at the same moment:

ticsync_ros::ROSTimeWithUncertainty ticsync_time = ticsync_ros::now();
float uncertainty = ticsync_time.uncertainty();
float now = ticsync_time.ros_time.to_secs();
To your point that such an API as I just suggested would not work because you'd need to change all the code that already uses plain "ROS time", I'd say that I don't believe that is a limitation because even if uncertainty were in the core API we cannot force people to use it (I don't think anyways). I'm also not convinced that everyone would need to use the API which contain extra metadata like uncertainty.

Now, I don't even know if uncertainty is something TICSync could give you, but procedurally, I think having layered API's like I demonstrated above is the way to go until one API becomes the obvious solution for pretty much everyone.

That being said, I'm open to discussing individual extensions to the Time API if you can argue why it needs to be in the core API and cannot not be easily layered on top and used by those who need it. A good example of this is simulated time. The ROS Time type which can either give simulated time or time from the system clock is needed to facilitate simulation and log playback situations. It's possible a similar set of use cases surrounding data synchronization or interpolation would need something like uncertainty. But to be elevated to the core API I think it needs to two qualities: it needs to affect the behavior of the ROS time constructs for all code that uses it and it needs to have a reasonable fallback behavior when not in use. The simulated time object (ROS Time) both needs to affect many existing calls to the time API (but not all which is what "wall time" is for) and it behaves reasonably when simulated time is not being used.

I think that the layered API approach suggested by Will is a good solution to this problem.

I am curious, though, how often does uncertainty information about a time source change?

Geoff

[ahundt]

@gbiggs it can change from one second to another but can be dramatic in real use cases. For example, ping a server on the other side the world for a year straight from a cell phone and check out the resulting stats to see how much it could vary for a similarly connected robot driving around various continents. You don't really have to do it, just thinking about everywhere a heavy traveler goes in a year and the reliability of cellphone networks will likely be convincing. :-)

[wjwwood]

For example, ping a server on the other side the world for a year straight from a cell phone and check out the resulting stats to see how much it could vary for a similarly connected robot driving around various continents.

But that's travel time, not time differences between the machines. Those two machines would need to have their time sources synchronized through a different mechanism like NTP or with the PPS of GPS (making them each a "stratum 0" time sources https://en.wikipedia.org/wiki/Network_Time_Protocol#Clock_strata). I see the measuring of the message travel time (or of ping) is a different issue altogether... Even if you're talking about the average ping time over a year, that's not got anything to do with the time on the local system versus some relative other time source.

[ahundt]

But that's travel time, not time differences between the machines. Those two machines would need to have their time sources synchronized through a different mechanism like NTP or with the PPS of GPS (making them each a "stratum 0" time sources https://en.wikipedia.org/wiki/Network_Time_Protocol#Clock_strata). I see the measuring of the message travel time (or of ping) is a different issue altogether... Even if you're talking about the average ping time over a year, that's not got anything to do with the time on the local system versus some relative other time source.

I agree ping != time difference. Sorry for the confusion, I was merely trying to illustrate some source of uncertainty.

In other words, while the number may be very small, even the machine next to the atomic clock on stratum 0 has an uncertainty value that can be estimated. That uncertainty will also change by some small amount with every measurement. Even the uncertainty measurement isn't 100% guaranteed to be certain or fixed. I could suddenly unplug and replug the connection to the atomic clock, or more likely my GPS signal could drop out, and that would cause a change (increase) in uncertainty. That uncertainty would decrease when the connection is re-established to the GPS signal. Correct?

[tfoote]

Indeed uncertainty can change and it can be valuable to know about it for certain algorithms. However in most cases such as realtime mentioned at the top of the thread it's mostly about guaranteeing a minimum performance bound.

Applications requiring very high precision such as the triggering between stereo pairs usually do that with a dedicated electrical circuit in hardware. There's a level that that is unachievable using standard networking protocols, especially if we need our system to work over arbitrary wireless networks.

Building in the ability to the core of the system to measure, estimate, and communicate uncertainty will add overhead. And I think that this ability is separable from the core implementation. I think that it would be great if there was an implementation of TICSync which could be put in as a source of system time, and can be hooked in as a custom time source.

One of the major challenges is simply picking a way of representing uncertainty. Depending on your application there are many different ways to represent uncertainty and depending on your application you'll use different approximations. Many use cases use the first order Gaussian model. But there are many more models with higher fidelity which can provide more information, such as a probability distribution function, which can range in representation from discrete values histogram to a continuous higher order function.

Depending on your representation there is completely different math for operating on the uncertainty, especially propagating it and using it's results making it impossible for us to pick the one implementation for everyone to use for their application.

As we talk about representing the uncertainty of the measurements I would suggest that for the areas where the uncertainty is important that a compound datatypes be created. Like we have PoseStamped and PoseWithCovarianceStamped we could have a TimeWithGuassianUncertainty message. It would contain the time representation and the uncertainty representation. And then methods can be defined to operate on those messages. If these messages are picked up and used in more packages it will become a standard. For other applications maybe a min and max range or bounds or more complicated uncertainty representation will be defined and become adopted by the system.

To support this we would just need a system time source which can be queried for it's uncertainty and extending the clock API to be able to query it. With that any data producer which wants to publish the clock uncertainty datatype can query the clock time and uncertainty and then publish the datatype.

But until we have a clock implementation that can provide an uncertainty measurement I don't think it's worth trying to build out this infrastructure. Once we have an implementation which can provide an estimate of the uncertainty we can extend the Clock api to query it and define messages to contain the datatype.

PS note that this is a closed issue so you're only getting people who were subscribed and get the notifications. Unless we want to reopen(which we can't since it's already merged) this it should probably be on new thread, optimally a concrete proposal that can be discussed.

[ahundt]

sure, where do you advise?

[jacquelinekay]

@ahundt you could move the conversation to the sig-ng mailing list, or if you want to start a new design proposal incorporating some of the ideas from this discussion, you could make a pull request to this repository with a new article.