[Fleet] Improve performance of data stream API #97058
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jen-huang
added
release_note:fix
v8.0.0
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
jfsiii
approved these changes
Apr 13, 2021
💚 Build SucceededMetrics [docs]
History
To update your PR or re-run it, just comment with: cc @jen-huang |
jen-huang
added
the
auto-backport
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Apr 14, 2021
* Improve performance of data stream API * Remove extra logger, replace filter with reduce * Remove unused import
💔 Backport failed
Successful backport PRs will be merged automatically after passing CI. To backport manually run: |
jen-huang
added a commit
to jen-huang/kibana
that referenced
this pull request
Apr 14, 2021
* Improve performance of data stream API * Remove extra logger, replace filter with reduce * Remove unused import # Conflicts: # x-pack/plugins/fleet/server/routes/data_streams/handlers.ts # x-pack/plugins/fleet/server/services/epm/packages/get.ts
kibanamachine
added a commit
that referenced
this pull request
Apr 14, 2021
jen-huang
added a commit
that referenced
this pull request
Apr 14, 2021
* Improve performance of data stream API * Remove extra logger, replace filter with reduce * Remove unused import # Conflicts: # x-pack/plugins/fleet/server/routes/data_streams/handlers.ts # x-pack/plugins/fleet/server/services/epm/packages/get.ts
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
Apr 14, 2021
…ax_primary_shard_size * 'master' of github.com:elastic/kibana: (99 commits) added missing optional chain for bracket notation (elastic#96939) [Discover][DocViewer] Fix toggle columns from doc viewer table tab (elastic#95748) [TSVB] Fix per-request caching of index patterns (elastic#97043) [Datatable] Fix filter cell flakiness (elastic#96934) Unskip heatmap suite and fixes flakiness (elastic#96941) [Fleet] Improve performance of data stream API (elastic#97058) [ML] Data Frame Analytics: remove beta badge (elastic#96977) [App Search] Migrate expanded rows for meta engines table in Engines Overview (elastic#96251) Instances latency distribution chart tooltips and axis fixes (elastic#95577) [Monitoring] Using primary average shard size (elastic#96177) [Workplace Search] Hide Kibana chrome on 3rd party connector redirects (elastic#97028) ## [Security Solution] Fixes `Exit full screen` and `Copy to cliboard` styling issues (elastic#96676) Index pattern field editor - Add warning on name or type change (elastic#95528) [App Search] Add small engine breadcrumb utility helper (elastic#96917) Copy esArchiver commands from ./reassign.ts to fix tests (elastic#97012) [Security Solution][Detections] Updates MITRE Tactics, Techniques, and Subtechniques for 7.13 (elastic#97011) Index patterns server - throw correct error on field caps 404 (elastic#95879) Use `EuiThemeProvider` in lists plugin tests and stories (elastic#96129) [npm] upgrade caniuse database (elastic#97002) chore(NA): moving @kbn/apm-utils into bazel (elastic#96227) ... # Conflicts: # x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/serialization/policy_serialization.test.ts # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/form/schema.ts
phillipb
added a commit
to phillipb/kibana
that referenced
this pull request
Apr 14, 2021
…to-metrics-tab * 'master' of github.com:elastic/kibana: (61 commits) [Usage collection] Usage counters (elastic#96696) UI actions readme (elastic#96925) [TSVB] Enable brush for visualizations created with no index patterns (elastic#96727) [Data telemetry] Add Async Search to the tests (elastic#96693) added missing optional chain for bracket notation (elastic#96939) [Discover][DocViewer] Fix toggle columns from doc viewer table tab (elastic#95748) [TSVB] Fix per-request caching of index patterns (elastic#97043) [Datatable] Fix filter cell flakiness (elastic#96934) Unskip heatmap suite and fixes flakiness (elastic#96941) [Fleet] Improve performance of data stream API (elastic#97058) [ML] Data Frame Analytics: remove beta badge (elastic#96977) [App Search] Migrate expanded rows for meta engines table in Engines Overview (elastic#96251) Instances latency distribution chart tooltips and axis fixes (elastic#95577) [Monitoring] Using primary average shard size (elastic#96177) [Workplace Search] Hide Kibana chrome on 3rd party connector redirects (elastic#97028) ## [Security Solution] Fixes `Exit full screen` and `Copy to cliboard` styling issues (elastic#96676) Index pattern field editor - Add warning on name or type change (elastic#95528) [App Search] Add small engine breadcrumb utility helper (elastic#96917) Copy esArchiver commands from ./reassign.ts to fix tests (elastic#97012) [Security Solution][Detections] Updates MITRE Tactics, Techniques, and Subtechniques for 7.13 (elastic#97011) ...
|
Great improvement!
When is this "upfront" exactly happening? On package installation? Another way I was thinking of it to only retrieve it the first time the user actually clicks on a data stream and then cache it. |
|
@ruflin One call upfront means when the user hits |
madirey
pushed a commit
to madirey/kibana
that referenced
this pull request
May 11, 2021
* Improve performance of data stream API * Remove extra logger, replace filter with reduce * Remove unused import
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Resolves #97044. This PR improves the performance of
GET /api/fleet/data_streamsendpoint. Previously, we were retrieving dashboard information for each package, one dashboard ID at a time. This caused the endpoint to become slower and slower as the number of data streams/installed packages grew. And in some cases with unluckly timing against other ES operations/searches, this can cause the endpoint to time out or throw a500search_phase_execution_exceptionerror.With this PR, we retrieve dashboard information for all installed packages with one call upfront, and map the results for fast lookup in downstream operations. In my testing of 116 data streams, with 4 packages installed, this reduced the average response time from 2.5s to 180ms, more than 10x improvement:
Before
After
