[Metricbeat] Expand docker/blkio data

[fearful-symmetry]

What does this PR do?

See #13798. This passes along additional cgroup metrics already gathered but not used by metricbeat. Those new fields are wait_time, service_time and queued.

I'm still not 100% sure about this. We treat the new time fields the same way we do everything else--we sum them and treat them as a total. I'm not sure if this is a way to go, if we should be using average time instead, or if there's a different field type we should be using of long. Also, I'm having trouble finding docker hosts where any of these new fields are even populated in the underlying docker stats, which isn't helping things.

How to test this PR locally

Pull down on a macos/linux docker host, run TestData

[unixsurfer]

@fearful-symmetry what do you mean by we sum them and treat them as a total ?

According to the documentation of blkio-controller Linux kernel returns the total time for all IO operations for blkio.io_wait_time and blkio.io_service_time metrics, so I don't think we need to do any aggregation. Am I missing something here?

[fearful-symmetry]

@unixsurfer docker reports these stats per device/driver, hence why we also have major and minor numbers:

{"major":259,"minor":2,"op":"Sync","value":4813430784},
{"major":259,"minor":1,"op":"Sync","value":4806131712}

Metricbeat sums these down to a total per op.

[jsoriano]

Well, if these times are quantities of time spent on something, I think it is ok to sum them, the same as we sum the bytes read or written.

[jsoriano]

I'm having trouble finding docker hosts where any of these new fields are even populated in the underlying docker stats, which isn't helping things.

@fearful-symmetry they are not being populated in my system, on what distro do you see these metrics? maybe they require cgroups v2?

{"major":259,"minor":2,"op":"Sync","value":4813430784},
{"major":259,"minor":1,"op":"Sync","value":4806131712}

If these metrics are only exposed on the Sync op we will need to modify getNewStats to take this into account. Though for this metric probably the Total is enough.

[jsoriano]

Codewise it looks good to me, some possibly pending things:

• Confirm that these metrics are reported at least with "op" Total, so they are counted in getNewStats.
• Document in the description of the fields what are the requirements for these metrics to work. cgroups v2?
• Changelog entry for the new fields.

[fearful-symmetry]

@jsoriano

hey are not being populated in my system, on what distro do you see these metrics? maybe they require cgroups v2?

Those metrics are from a different field, I'm just going on documentation and code, which suggests they're all the same. It appears that the time fields are broken down the same as the other fields.

Document in the description of the fields what are the requirements for these metrics to work. cgroups v2?

Yah, I have...no idea.

[unixsurfer]

Well, if these times are quantities of time spent on something, I think it is ok to sum them, the same as we sum the bytes read or written.

To sum them per device? IMHO, we can simply return the time per operation and don't return the sum for all operations.
If we sum them then a huge spike in Write operation will be hidden and we wouldn't be able to see it.

[exekias]

This is looking good, any chance you can update existing tests to include the changes? It will also need a changelog

[fearful-symmetry]

jenkins, test this