By value migration script

[flash1293]

Right now, a lot of integrations dashboards are still built "by value". This means there are separate saved objects for each visualization. To ease migration to "by value" dashboards which don't have this issue, a helper script should be provided that performs the following tasks given a folder:

• Check for visualization saved objects in individual json files
• Check for dashboard saved objects in individual json files and look up which dashboards they belong to
• For each dashboard which has visualizations that can be inlined:
  • Migrate the dashboard and the visualizations to the latest version
  • Inline the visualizations into the panels json of the saved object, transforming the state similar to the different serialization routines for by value and by reference
• Overwrite the dashboard files
• Delete the visualization files

This allows maintainers to run the script, check whether the dashboards still work as expected and create a PR.

[elasticmachine]

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

[flash1293]

cc @ruflin

[ruflin]

@elastic/ecosystem Lets collaborate with the team from @flash1293 on this script. @joshdover You are likely interested in this too as it will speed up the installation of packages.

[flash1293]

The https://github.com/flash1293/legacy_vis_analyzer script is already doing the easy part of this (finding dashboards and visualizations and associating them by id)

[mtojek]

Hey Joe,

FYI We use elastic-package export dashboards to dump all Kibana saved objects and put them in separate directories, so in this case, I have a few questions:

1. Does this format change (always inline saved objects)?
2. Do you plan to deprecate the current one (always separate saved objects)?
3. Should we adjust the export command we use to dump objects differently?

[flash1293]

@mtojek Thanks for reaching out.

Does this format change (always inline saved objects)?

The old format is still supported, this is just an addition (and it's compatible with all tooling). Using the inlined variant should be the default choice though and there should be a specific reason if separate saved objects are used.

Do you plan to deprecate the current one (always separate saved objects)?

No, there are still good use cases to support this (e.g. one-off visualizations not part of a dashboard or visualizations which should always stay in sync across multiple dashboards, even after being edited by the user).

Should we adjust the export command we use to dump objects differently?

That's not necessary, everything will stay compatible. There are already some dashboards which are fully by value: https://github.com/elastic/integrations/tree/main/packages/ti_abusech/kibana

[mtojek]

The old format is still supported, this is just an addition (and it's compatible with all tooling). Using the inlined variant should be the default choice though and there should be a specific reason if separate saved objects are used.

@ruflin @joshdover I don't know what was the reason to separate Kibana objects into different directories. Maybe we should iterate on this?

[ruflin]

By the time the package-spec was created, by value did not exist yet. So this was the only option. I think moving forward, the majority of our dashboards with the visulisations inside should be just in the kiabna/dashboards directory. But there will still be some cases where users maybe only want to ship a visualization or index-pattern as a standalone asset.

[flash1293]

It's less about the directory structure within the integrations repo and more about how it translates into saved objects within Kibana. The current approach of having a single saved object per visualization has some downsides:

• It's slower to create all of the saved objects instead of increasing the payload of the dashboard saved object
• Every individual visualization will show up in the "Visualize Library" view within Kibana - as there are often thousands of these visualizations, the library becomes very hard to use for visualizations the user created themselves (e.g. they become hard to find)
• On loading the dashboard, an additional roundtrip is necessary to fetch the dashboard saved object, then the visualization saved objects
• On cloning the dashboard to create a custom version of it, the individual visualizations are not cloned but instead still reference the same visualizations in the library, so if a user starts changing them on the cloned dashboard, the original integrations dashboard will be changed as well

The reason the structure with separate visualization saved objects is used because a lot of these dashboards (or at least the first version of them) predate the "by value" (inlining of visualizations into the dashboard) feature.

[flash1293]

One small note to add - saved searches can't be done "by value" yet, so they will definitely stay, at least in the mid-term. map, lens and visualization saved objects can be inlined.

[flash1293]

Another good point by @joshdover about why inlining is beneficial: On using the global search bar, it's harder than necessary at the moment to find the dashboard because there will always be lots of visualizations matching the same search.

[mtojek]

By the time the package-spec was created, by value did not exist yet. So this was the only option. I think moving forward, the majority of our dashboards with the visulisations inside should be just in the kiabna/dashboards directory. But there will still be some cases where users maybe only want to ship a visualization or index-pattern as a standalone asset.

It means that in most cases it would be recommended to switch to the inline form dashboards objects. That's what I expected and hence was my follow-up question regarding adjustments in elastic-package. If we decide to migrate all dashboards and don't update the export command, the old format will still sneak in.

I guess that we can evaluate which object types can be included in the dashboards based on the spec definitions.

[flash1293]

@mtojek It would be nice to introduce some kind of linting rule to prevent accidentally adding by reference visualizations in the future (without completely preventing it). However, I'm not sure how common it will be - if an integration author is going to put together a dashboard for a new integration by going to Kibana and configuring it through the UI, they will end up with inlined by value visualizations not with separate saved objects, because this is what's happening by default when adding panels to a dashboard.

I have set up a tracking dashboard for visualize->lens and by value migration:

IMHO we can keep an eye on that for now and introduce more tooling in case it becomes an issue.

[flash1293]

@ruflin @cmacknz I put together a first version of the vis inliner here: https://github.com/flash1293/legacy_vis_analyzer (do not look at the code, it's horrible). Could you give it a test run to see whether it works?

[ruflin]

@lalit-satapathy Pulling you in here as this is also interesting for your team. @cmacknz I assume based on your 👍 you'll give it a try.

[ruflin]

I opened an issue in package-spec elastic/package-spec#316 to help with migration to "by value" by notifying engineers about it.

[cmacknz]

Yes, I'll give the migration a try. I have upcoming PTO though so it might not happen for 2 weeks or so but it is on my list of things to do. If we want to try it sooner we'll have to find somebody else to try it out.

[flash1293]

@mtojek Do you have some time to give https://github.com/flash1293/legacy_vis_analyzer a try? Always happy to hop on a zoom to talk through it.

[mtojek]

I created an issue for this: elastic/elastic-package#791

[flash1293]

Script is available, if there are problems/bugs we'll open a new issue