Nested terms aggregations in legend

[willemdh]

Is your feature request related to a problem? Please describe.
We currently use Grafana for Metricbeat data visualisations, because it allows nested (group by)aggregations. This is very useful for some types of metrics, as it allows to show in 1 graph the min, max, avg and current cpu usage of a combination of host.name, process.name and process.id. It would be awesome if this would some day become possible in Kibana.

Describe the solution you'd like
Allow nest terms aggregations in tsvb, so we don't need to use Grafana anymore. To visualise the same amount of info, we would need like 3 graphs in Kibana vs 1 graph in Grafana..

Describe alternatives you've considered
Tried with all possible visualisations in Kibana, but never succeed to visualise nested metric aggregations with a legend containing avg, min, max and current.

Additional context

[markov00]

Hey @willemdh thanks for your request.
Let me try to understand a bit better your request: you are interested in visualize 4 values (min, max, avg,current CPU usaged) grouped by hostname, than by process name and then by process.id.

You dataset, for a single time bucket, will be something similar to:

hostname	process name	process id	avg	min	max	current
host1	chrome	chrome_tab_1	10	2	12	9
host1	chrome	chrome_tab_2	10	2	12	9
host1	firefox	firefox_tab_1	10	2	12	3
host1	chrome	firefox_tab_2	10	2	12	6

This mean that you want to visualize 4 metrics * each host-processname-processid.
If that is right, I was wondering if you have tried to use the default Line/Area visualization of Kibana specifying your 4 metrics and using multiple split series (that are the same as your group by). Multiple split series create nested aggregations like the one desired.

[willemdh]

@markov00 Tx for the info, I did try to make this in Kibana, but is is a very unsatisfying experience. To try proove my point I retried your suggestion.

Some (imho important) features that seem to be missing in Kibana:

• The legend in Kibana is very limited and doesn't allow what I'm doing in Grafana, each combination is listed as a separate list item instead of a dynamic table as in Grafana. Yes, I could make a separate datatble visualisation, but that would essentially double the work.
• A lot more steps are required to visualise the same things in Kibana as in Grafana
• General performance while making the graph is much better in Grafana (while using the same backend elasticsearch)
• As Grafana allows to add visualisations to rows, this enables us to list a lot more metrics on one Grafna dashboard compared to one Kibana dashboard. As the visualisations in a collapsed row are not loaded on dashboard opening, this does not use extra resources
• Graph area filling is not configurable.
• Items in grafana legends are multi-clickable and have more controls. It's possibe to use control/shift click to select multiple items in the legend. It also allows sorting of the min, max, avg, current values in the legend by default.
• Using multiple y-axis (left / right) with different types of metrics (% / bytes) is buggy and does not show correctly

I'm not trying to bash Kibana here, I love Kibana. But for visualising metrics Grafana is currently the best option. I'm really hoping that one day we don't have to use 2 tools for visualising our elasticsearch data.

Please let me know if you need any more info. Please try yourself to create the above type of visualisations in both Grafana and Kibana and I'm sure you will understand what I mean.

Tx

Willem

[markov00]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.

[willemdh]

Of course this is still valid. After all these years we are still using Grafana to visualise metrics...

[michieldetailleur]

I have to agree with Willem here

…

On Wed, 8 May 2024 at 12:39, willemdh ***@***.***> wrote: Of course this is still valid. After all these years we are still using Grafana to visualise metrics... — Reply to this email directly, view it on GitHub <#223 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIBZMDTPSGYINA7D7Z2UC3ZBH6GNAVCNFSM4HSGQK62U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJQGAZDQMRWGA2Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[markov00]

Thanks @willemdh @michieldetailleur I will reopen this issue. We are trying to reduce the amount of request we get from customers/users etc so that's why we just put it into an icebox. This just means that is not currently one of our highest priorities, but we don't want to just close it, but keep is around/searchable.

I'm reopening this issue but not sure if we can deliver that any time soon.
We are, btw improving the current chart legend #2096 providing some of the features you mentioned.

[markov00]

@willemdh Reading again the entire thread I feel that #2096 will solve the main issue request.
Lens can already do nested terms aggregations, the missing part is to show, on the legend and for a chosen metric the chart statistics: min,max, avg, current like you were showing in Grafana.
This is in progress in Kibana elastic/kibana#176583

about the following:

A lot more steps are required to visualise the same things in Kibana as in Grafana

I see, we are two different tools, we are doing our best but there always space to improve. It will be great to get more info about that, what are the pain points you see in our UX.

General performance while making the graph is much better in Grafana (while using the same backend elasticsearch)

Agree, and we are also working on that actively (See for example some caching efforts elastic/kibana#180440).

As Grafana allows to add visualisations to rows, this enables us to list a lot more metrics on one Grafna dashboard compared to one Kibana dashboard. As the visualisations in a collapsed row are not loaded on dashboard opening, this does not use extra resources

This effort is in our pipeline elastic/kibana#1547 definitely a looong waited feature, but is finally planned.

Graph area filling is not configurable.

And still not configurable. If you can give me a bit more details why do you need this, what is your use case, I can expand a bit about this decision.

Items in grafana legends are multi-clickable and have more controls. It's possibe to use control/shift click to select multiple items in the legend.

Same in Kibana, using shift click. Probably was implemented after this issue was created

It also allows sorting of the min, max, avg, current values in the legend by default.

Will probably be a subsequent feature of the current Legend table effort.

Using multiple y-axis (left / right) with different types of metrics (% / bytes) is buggy and does not show correctly

Do you have more details there?

[willemdh]

A lot of questions. I'm on a holiday atm. Just out of my head. Adding rows, table legend would be really great. About the axis, not sure if this still applies to current Lens state, but I think so. Something that is really hard to visualise properly in Kibana is network flow data and diskio metrics which contains deltas. A non negative derivative function in Lens is non existent, definitely not usable with multiple aggregations. About the graph areas, I mean things like opacity, better coloring options, but also line width and other options for connecting graph points. Tx for the shift click tip, I'll test it asap. Also visualising ingress on the positive y axis and egress on a negative y axis is also not an option in Lens I think? Using macros referencing the terms aggregations in the legend is also not an option I think in Lens.