[O11y][HAProxy] Migrate stat data-stream visualizations to lens in kibana 8.4.0

[rajvi-patel-22]

• Enhancement

What does this PR do?

• Migrate stat data-stream visualizations to lens for HAProxy integration package

• Statistics for HAProxy Lens migration:

HAProxy Dashboards	Before Migration		After Migration
	Lens	Visualization	Lens	Visualization
[Metrics HAProxy] HTTP backend	0	8	8	0
[Metrics HAProxy] Overview	0	4	4	0
[Metrics HAProxy] Frontend	0	2	2	0
[Metrics HAProxy] HTTP frontend	0	3	3	0
[Metrics HAProxy] HTTP server	0	6	6	0
[Metrics HAProxy] Backend	0	6	6	0
	0	29	29	0

Checklist

• I have added an entry to my package's changelog.yml file.
• I have verified that panels are populated with data.
• I have verified that panels are not distorted after being migrated to lens.
• I have updated screenshots of dashboard.
• I have verified that data count are matching and panel aggregations are same as before.

Author's Checklist

• Migrated panels should be removed from visualization folder.
• Migrated visualizations are populating in current Kibana version 8.4.0 itself.

Related issues

• Relates Lens Migration of OBS integrations #5221

Issues

Below 10 panels are not migrated due to lens limitations:

Dashboard	Visualization	Reason
[Metrics HAProxy] HTTP backend	Downtime seconds [Metrics HAProxy]	elastic/kibana#153326
	Connections per server [Metrics HAProxy]	elastic/kibana#153326
	Active servers in backend [Metrics HAProxy]	elastic/kibana#153326
[Metrics HAProxy] HTTP server	Healthcheck [Metrics HAProxy]	Complex painless script
[Metrics HAProxy] Backend	Downtime seconds [Metrics HAProxy]	elastic/kibana#153326
	Connections per server [Metrics HAProxy]	elastic/kibana#153326
	Active servers in backend [Metrics HAProxy]	elastic/kibana#153326
[Metrics HAProxy] Overview	Frontends	elastic/kibana#153326
	Backends	elastic/kibana#153326
	Servers visualizations	elastic/kibana#153326

Note: Please find below screenshots of dashboards (before migration) to compare migrated panels:

[Metrics HAProxy] Overview:

[Metrics HAProxy] HTTP backend:

[Metrics HAProxy] Backend:

[Metrics HAProxy] HTTP Frontend:

[Metrics HAProxy] HTTP server:

[Metrics HAProxy] Frontend:

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-04-12T05:59:38.615+0000

• Duration: 15 min 59 sec

Test stats 🧪

Test	Results
Failed	0
Passed	25
Skipped	0
Total	25

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (1/1)	💚
Files	100.0% (1/1)	💚
Classes	100.0% (1/1)	💚
Methods	90.476% (19/21)	❕
Lines	90.556% (163/180)	❕
Conditionals	100.0% (0/0)	💚

[rajvi-patel-22]

@agithomas, As we have mentioned in the description, there are 10 panels which have not be migrated to the lens. We have found work around for Healthcheck [Metrics HAProxy] panel from [Metrics HAProxy] HTTP server dashboard which was blocked. The functionality can be replicated using conditional operations which are added in Kibana v8.6.0 (Github Issue).

Currently, we have migrated everything to 8.4.0. In order to migrate this new panel, we will have to upgrade to the newer Kibana version (v8.6.0).

Can you please let us know how should we proceed further?

[agithomas]

Currently, we have migrated everything to 8.4.0. In order to migrate this new panel, we will have to upgrade to the newer Kibana version (v8.6.0).

@rajvi-elastic , are you saying with 8.6.0 migration, we would have 100% lens migration for HAProxy dashboards?

[rajvi-patel-22]

@rajvi-elastic , are you saying with 8.6.0 migration, we would have 100% lens migration for HAProxy dashboards?

No @agithomas . Not all the 10 remaining panels could be migrated to the lens. Only one panel which is blocked due to Complex painless script could be migrated. All the other 9 panels are blocked due to elastic/kibana#153326.

With v8.6.0 we will be only able to migrate one panel from the list. Other 9 panels would be still visualizations.

[rajvi-patel-22]

@agithomas, There is one screenshot of the dashboard of the filebeat module. It seems redundant to me, we already have updated the latest screenshot here. If it's not needed now then please remove this from the integration.

[agithomas]

All the other 9 panels are blocked due to elastic/kibana#153326

@SubhrataK , is the last bucket issue highlighted here and referred in other integrations can now be considered is not an issue anymore ?

@rajvi-elastic , please confirm if the issue is similar to other integrations and there are no edge cases here.

[agithomas]

@agithomas, As we have mentioned in the description, there are 10 panels which have not be migrated to the lens. We have found work around for Healthcheck [Metrics HAProxy] panel from [Metrics HAProxy] HTTP server dashboard which was blocked. The functionality can be replicated using conditional operations which are added in Kibana v8.6.0 (Github Issue).

Currently, we have migrated everything to 8.4.0. In order to migrate this new panel, we will have to upgrade to the newer Kibana version (v8.6.0).

Can you please let us know how should we proceed further?

@lalit-satapathy , what is your take on this? The benefit of migrating to 8.6.0 from 8.4.0 is only for just one panel. My personal preference is to retain this one panel in old vis style and migrate when there exist a proper solution instead of workaround.

[rajvi-patel-22]

@SubhrataK , is the last bucket issue highlighted here and referred in other integrations can now be considered is not an issue anymore ?

@rajvi-elastic , please confirm if the issue is similar to other integrations and there are no edge cases here.

No @agithomas, The issue that is mentioned here is not similar to what we discussed in IBM MQ. We are still discussing this last bucket issue with Kibana team and they have scheduled a call in next week.

[rajvi-patel-22]

As discussed with @SubhrataK,

1. For the issue Not having feature to set last bucket value, we are good with the metrics chart. For the line/area chart we need auto functionality in reduced timerange. So for the time series visualization we still need to discuss with the kibana team.

2. For the Splitting a chart into multiple smaller charts by a breakdown dimension is not supported in the lens, we are going to migrate into table chart. Also need to re-arrange the dashboard like first metrics, then line/area/bar charts, and the table charts be at last.

3. Split Active server in backend into 2 panels which represents Active server and Total server.

4. Create line chart to represent Downtime seconds.

[rajvi-patel-22]

@agithomas, Can you please help me understand the aggregation that has been used in Connections per server panel. The panel is using last value functionality along with drilldowns. In order to migrate this visualization in lens, it would be better to have an understanding of why average and derivatives are applied on haproxy.stat.connection.total rather than directly using the value of haproxy.stat.connection.total :

CC: @SubhrataK

[agithomas]

@agithomas, Can you please help me understand the aggregation that has been used in Connections per server panel. The panel is using last value functionality along with drilldowns. In order to migrate this visualization in lens, it would be better to have an understanding of why average and derivatives are applied on haproxy.stat.connection.total rather than directly using the value of haproxy.stat.connection.total :

CC: @SubhrataK

It appears that the field is of metric_type counter. Avg() function may not be the agg function to use here. It is important to find out the rate of change on counter field and hence derivative() function is used.

You may consider changing from avg() to max() while making the lens migration.

[rajvi-patel-22]

@lalit-satapathy , what is your take on this? The benefit of migrating to 8.6.0 from 8.4.0 is only for just one panel. My personal preference is to retain this one panel in old vis style and migrate when there exist a proper solution instead of workaround.

@agithomas, The solution that we have implemented is the same as auto lens migration implements (Edit visualization in lens option). In older TSVB visualization, the ternary operator was used which can be replicated in lens using ifelse condition v8.6.0. In both the cases the counts are accurate and If we migrate to 8.6.0 then we will achieve 100% migration.

Please let us know whether we should migrate to the 8.6.0 or not?

[agithomas]

@lalit-satapathy , what is your take on this? The benefit of migrating to 8.6.0 from 8.4.0 is only for just one panel. My personal preference is to retain this one panel in old vis style and migrate when there exist a proper solution instead of workaround.

@agithomas, The solution that we have implemented is the same as auto lens migration implements (Edit visualization in lens option). In older TSVB visualization, the ternary operator was used which can be replicated in lens using ifelse condition v8.6.0. In both the cases the counts are accurate and If we migrate to 8.6.0 then we will achieve 100% migration.

Please let us know whether we should migrate to the 8.6.0 or not?

Please proceed.

[rajvi-patel-22]

@agithomas, The solution that we have implemented is the same as auto lens migration implements (Edit visualization in lens option). In older TSVB visualization, the ternary operator was used which can be replicated in lens using ifelse condition v8.6.0. In both the cases the counts are accurate and If we migrate to 8.6.0 then we will achieve 100% migration.
Please let us know whether we should migrate to the 8.6.0 or not?

Please proceed.

@agithomas, All the panels have been migrated to lens (Kibana v8.6.0).

[agithomas]

packages/haproxy/img/haproxy-metrics-http-frontend.png have negative values for traffic volume. Please correct them.

[rajvi-patel-22]

packages/haproxy/img/haproxy-metrics-http-frontend.png have negative values for traffic volume. Please correct them.

@agithomas, this panel uses the differences. This has the similar aggregations as oracle panel..

[agithomas]

packages/haproxy/img/haproxy-metrics-http-frontend.png have negative values for traffic volume. Please correct them.

@agithomas, this panel uses the differences. This has the similar aggregations as oracle panel..

Well, it is understandable from Oracle Integration snapshot, it is not clear here that what is represented is a difference from previous value. In oracle, the actual value was shown and then the line showing the difference is an overlay. I understand that this is how things were in the past .

@SubhrataK , can you please have a quick check on this and recommend a suitable title change or a better representation of panel data such as - data + difference value

[agithomas]

packages/haproxy/img/haproxy-metrics-http-frontend.png have negative values for traffic volume. Please correct them.

@agithomas, this panel uses the differences. This has the similar aggregations as oracle panel..

Well, it is understandable from Oracle Integration snapshot, it is not clear here that what is represented is a difference from previous value. In oracle, the actual value was shown and then the line showing the difference is an overlay. I understand that this is how things were in the past .

@SubhrataK , can you please have a quick check on this and recommend a suitable title change or a better representation of panel data such as - data + difference value

@rajvi-elastic , please create a backlog issue for this. It may be best we take it up separately in the interest of time.

[agithomas]

Rest things are good.

Version change from 8.0.0 to 8.6.0 needs approval, deferring the PR approval until then.

[rajvi-patel-22]

@rajvi-elastic , please create a backlog issue for this. It may be best we take it up separately in the interest of time.

@agithomas, I have created the backlog issue.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

[agithomas]

@rajvi-elastic ,

please wait till #8204 is merged .

[rajvi-patel-22]

Another PR has been raised for HAProxy lens migration.