Extract Kibana dashboards #7224
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently when a PR is opened to change or add a dashboard the PR's are hard to review even if only a small detail was changed. The reasons is that the Kibana json objects contain json as string. The content inside these strings is valid JSON if it is decoded. With this PR the dashboards in Metricbeat are modified that they contain the full decoded JSON objects instead of the string. Additional the JSON entries are sorted. This makes also the content of the visualisations readable, will create minimal diffs and allows to even apply small fixes on the code base. In addition we can now validate if it's valid JSON and introduce automatic scripts to remove potentially unneeded fields from the dashboards to clean them up. All the existing dashboards were decoded with the following command: ``` python ../libbeat/scripts/unpack_dashboards.py --transform=decode --glob="/Users/ruflin/Dev/gopath/src/github.com/elastic/beats/metricbeat/module/*/_meta/kibana/6/dashboard/*.json" ``` This command has to be applied to dashboards exported from Kibana before adding them to the module directory. To make sure the import still works as before, on collection of the dashboard they are converted back into the encoded format. This change currently only applies to Metricbeat dashboards but could be applied to all.
ruflin
added
discuss
|
Here is a follow up PR to also adjust the exporter script: #7241 |
jsoriano
approved these changes
Jun 4, 2018
ruflin
added a commit
to ruflin/beats
that referenced
this pull request
Jun 4, 2018
In elastic#7239 support to export a dashboard is a added to each Beat. This is expected to be used by users. The export_dashboard script from the Beats repository is expected by the Devs and contributors which want to add new dashboards. In elastic#7224 the dashboards are modified to be stored with decoded json objects for better versioning. This change modifies the export dashboard script so it generates the same decoded output so no additional conversion is needed. Note: The export_dashboard script could need some cleanup but I didn't tackle this in this PR and leave it for later to keep the change to a minimum.
jsoriano
pushed a commit
that referenced
this pull request
Jun 4, 2018
In #7239 support to export a dashboard is a added to each Beat. This is expected to be used by users. The export_dashboard script from the Beats repository is expected by the Devs and contributors which want to add new dashboards. In #7224 the dashboards are modified to be stored with decoded json objects for better versioning. This change modifies the export dashboard script so it generates the same decoded output so no additional conversion is needed.
ruflin
added a commit
to ruflin/beats
that referenced
this pull request
Jun 5, 2018
Follow up from elastic#7224 for Filebeat.
ruflin
added a commit
to ruflin/beats
that referenced
this pull request
Jun 5, 2018
Follow up from elastic#7224 for Auditbeat.
ruflin
added a commit
to ruflin/beats
that referenced
this pull request
Jun 5, 2018
Apply decoding of dashboards also to Heartbeat. See elastic#7224 for more details.
This was referenced Jun 5, 2018
jsoriano
pushed a commit
that referenced
this pull request
Jun 5, 2018
Apply decoding of dashboards also to Heartbeat. See #7224 for more details.
jsoriano
pushed a commit
that referenced
this pull request
Jun 5, 2018
Follow up from #7224 for Auditbeat.
jsoriano
pushed a commit
that referenced
this pull request
Jun 5, 2018
Follow up from #7224 for Filebeat.
ruflin
added a commit
to ruflin/beats
that referenced
this pull request
Jun 8, 2018
Currently all dashboards end up under `_meta/kibana` in each Beat. The problem with this directory is that for some Beats it contains the automatically collected and generated files and for others the original. This prevents us from having decoded JSON files for all dashboards across Beats. An other benefit of having the `kibana.generated` directory is that now `_meta/kibana` can become the place for dashboards across modules in one Beat without having to worry about overwriting / removing these dashboards. Community Beats must convert their dashboards with the script under elastic#7224. Further changes: * Decode Packetbeat and Winlogbeat dashboards * Move encoding step to libbeat so it is applied automatically to all dashboards in the kibana.generated directory. * Remove _meta/kibana from .gitignore as it can now be used to contain dashboards in all Beats. * Adjust Makefiles for the new logic. * Adjust tests to depend on kibana.generated directory
jsoriano
pushed a commit
that referenced
this pull request
Jun 11, 2018
Currently all dashboards end up under `_meta/kibana` in each Beat. The problem with this directory is that for some Beats it contains the automatically collected and generated files and for others the original. This prevents us from having decoded JSON files for all dashboards across Beats. An other benefit of having the `kibana.generated` directory is that now `_meta/kibana` can become the place for dashboards across modules in one Beat without having to worry about overwriting / removing these dashboards. Community Beats must convert their dashboards with the script under #7224. Further changes: * Decode Packetbeat and Winlogbeat dashboards * Move encoding step to libbeat so it is applied automatically to all dashboards in the kibana.generated directory. * Remove _meta/kibana from .gitignore as it can now be used to contain dashboards in all Beats. * Adjust Makefiles for the new logic. * Adjust tests to depend on kibana.generated directory
3 tasks
webmat
pushed a commit
to webmat/beats
that referenced
this pull request
Aug 27, 2018
See elastic#7251 and elastic#7224 for more details.
adriansr
pushed a commit
to adriansr/beats
that referenced
this pull request
Sep 26, 2018
See elastic#7251 and elastic#7224 for more details.
adriansr
pushed a commit
to adriansr/beats
that referenced
this pull request
Oct 2, 2018
See elastic#7251 and elastic#7224 for more details.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently when a PR is opened to change or add a dashboard the PR's are hard to review even if only a small detail was changed. The reasons is that the Kibana json objects contain json as string. The content inside these strings is valid JSON if it is decoded.
With this PR the dashboards in Metricbeat are modified that they contain the full decoded JSON objects instead of the string. Additional the JSON entries are sorted. This makes also the content of the visualisations readable, will create minimal diffs and allows to even apply small fixes on the code base. In addition we can now validate if it's valid JSON and introduce automatic scripts to remove potentially unneeded fields from the dashboards to clean them up.
All the existing dashboards were decoded with the following command:
This command has to be applied to dashboards exported from Kibana before adding them to the module directory.
To make sure the import still works as before, on collection of the dashboard they are converted back into the encoded format.
This change currently only applies to Metricbeat dashboards but could be applied to all.