Use yarp-telemetry to store the yarprobotinterface log

[S-Dafarra]

This is more a request for discussion.

Recently, in order to debug a long-standing issue (robotology/icub-tech-support#673), we have a strong need to store the log file coming from the robot yarprobotinterface. The common tool to do so would be the yarplogger. Unfortunately, this has the disadvantage that it can only store a limited amount of lines, and usually above the 10000 lines it is pretty unusable. In the specific case mentioned above, this limitation is even stronger considering that we are trying to debug a problem that may occur after hours of robot usage.

During a recent demo, @maggia80 had the idea of using yarp-telemetry to store the textual data produced by yarprobotinterface. With this issue I would like to discuss this possibility.

@Nicogene @AlexAntn @traversaro @pattacini

[randaz81]

The limit of 10000 lines can be changed to a different number (or even unlimited) in the 'preferences' menu of the gui. I'd happy to discuss any other limitations /improvements if you have other needs.

[drdanz]

I don't think that using yarp-telemetry is a good idea for the log.
The whole concept of yarp telemetry was to log a limited amount of structured data, not strings.
Also consider that log needs to be sent out immediately (especially warnings and errors), and not stored on a circular buffer, and wait for the flush.

[S-Dafarra]

The limit of 10000 lines can be changed to a different number (or even unlimited) in the 'preferences' menu of the gui. I'd happy to discuss any other limitations /improvements if you have other needs.

Yes, I am aware of that. In the past, we had some stability issues when setting a large limit. Associated discussion robotology/community#558

I don't think that using yarp-telemetry is a good idea for the log.
The whole concept of yarp telemetry was to log a limited amount of structured data, not strings.
Also consider that log needs to be sent out immediately (especially warnings and errors), and not stored on a circular buffer, and wait for the flush.

Note that this could be just for storing the logs, we were imagining it as another listener on the associated log port. It could just take care of saving the log in chunks, without the user intervention. A string, in the end, is basically a vector of integers. On the other hand, I see that the length is not fixed, but I suppose we can circumvent this limitation

[Nicogene]

Adding the logging to the telemetry is doable but it would complicate a little bit the architecture, since all the other quantities are logged via attaching remote control boards or via nwc (odometry).
As far as I know, it does not exist an nwc or interfaces for logging, and we should open some ports in the device read the logs from them, and store them, implementing the same thing that yarplogger does.
Please correct me if I am saying nonsense, I have not great insight into how the logger works.

[drdanz]

Note that this could be just for storing the logs, we were imagining it as another listener on the associated log port.

In this case, it doesn't make any sense to use yarp-telemetry, unless you are interested in reading the logs in matlab.
Such a listener should just save the log on a database or similar, no need for a circular buffer or for being able to write the data to a matlab file, even less for a device.

The yarp logger is already split into a library and an executable (actually 2 executables, we also have the console version).
It wouldn't take much to take the library, and use sqlite to write the log on a database file, and replace the logger.
The consideration to take in this case is the same as for having 2 loggers in the same network though.

[S-Dafarra]

I see! I would say that the end goal would be to have the possibility to store logs continuously, when needed, without the intervention of the user.

I guess the choice of the method comes as a second thing. I was pretty aware of how yarp-telemetry works, while I am less confident with the yarplogger infrastructure. I am ok with any solution.

The consideration to take in this case is the same as for having 2 loggers in the same network though.

This is a good point. Dropping here the discussion for future references robotology/community#478