Rabbitmq exchanges metricset

[jsoriano]

Continues with #6607

[spotlesscoder]

will #6606 also be added?

[jsoriano]

Yes, the idea is to finish with both PRs

[jsoriano]

Don't merge it yet, I want to try to generate a data.json with more metrics.

[ruflin]

WFG

[ruflin]

Normally in case of empty objects they should not be added. Perhaps an issue in the schema? Not a blocker of the PR but worth mentioning.

[jsoriano]

I haven't found how to avoid it, they are definned as optional schema.Objects, there would be other way to define them?

                "messages": c.Dict("message_stats", s.Schema{
                        "publish": s.Object{
                                "count": c.Int("publish", s.Optional),
                                "details": c.Dict("publish_details", s.Schema{
                                        "rate": c.Float("rate"),
                                }, c.DictOptional),
                        },
                        ...
                }

[jsoriano]

@ruflin would it be ok to define the schema with dotted field names?

                "messages": c.Dict("message_stats", s.Schema{
                        "publish.count": c.Int("publish", s.Optional),
                        "publish.details": c.Dict("publish_details", s.Schema{
                                "rate": c.Float("rate"),
                        }, c.DictOptional),

[ruflin]

I wonder if this will have any affect on the output in the data.json?

For the empty objects: Can you try to set the surrounding doc as optional too? Just had a look at ack and I think it's not set to optional.

[jsoriano]

I have been playing around with rabbitmq, trying to understand these metrics, and it seems that for exchanges, only publish_in and publish_out metrics are populated. Indeed, in the management web view only these metrics are exposed (and used in the code).

But according to documentation all metrics for queues, exchanges and channels fill the same message_stats object with "fields for which some activity has taken place". So I'm not completely sure, but this would lead to change some things in this PR.

I see two approaches here:

1. Use the same schema and fields for queues, exchanges and channels metricsets
2. Remove all fields that are not populated for each type of object

The advantage of the first option is that we'd be supporting any documented field if at some moment it is added to some specific object type, but probably if they are not now is because they don't make sense, for example, if queues and channels have the responsibility of redispatching, or controlling acks, probably exchanges are never going to have activity and metrics about that.

I'd go for the second option, if only for the advantage of letting the users know what metrics to expect in each metricset. We can always revisit this decision if some new field is added in the future, but we avoid confusion by now.

@CodingSpiderFox what do you think, does it make sense? could you check in your deployments if your exchanges have more metrics apart of the ones for publish_in and publish_out?

[spotlesscoder]

@jsoriano I agree with your choice. Absolutely makes sense.

I will check for the metrics in my deployments when I'm back to work

[spotlesscoder]

Unfortunately, I can't check the deployments at the moment as I'm very busy with other stuff :/

[jsoriano]

Ok, I think this can be merged. It is added as beta and in principle it doesn't break any existing functionality in any case. @CodingSpiderFox please create a new issue, or add a coment here, if at some moment you can check the metrics in your deployments.

[spotlesscoder]

Nice!

[jsoriano]

jenkins, test this again, please

[spotlesscoder]

Which feature version of metricbeat does contain these new fields?

[jsoriano]

@CodingSpiderFox these fields will be available in 6.4.