[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist #193205
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ymao1
changed the title
ymao1
added
release_note:skip
|
Pinging @elastic/response-ops (Team:ResponseOps) |
|
@elasticmachine merge upstream |
|
@elasticmachine merge upstream |
pmuellr
approved these changes
Sep 19, 2024
| await this.esContext.esAdapter.createDataStream(this.esContext.esNames.dataStream); | ||
| } else { | ||
| // apply current mappings to existing data stream | ||
| await this.esContext.esAdapter.updateConcreteIndices(this.esContext.esNames.dataStream); |
There was a problem hiding this comment.
Do we end up testing this path in FTR? I'm thinking no, but not positive.
Could we do this with a jest integration test? Just start a Kibana, then start another or kill the first and restart. Maybe we could add a debug log to ensure we made it here?
There was a problem hiding this comment.
Added a debug log when the PUT mappings call is successful and added a jest integration test that starts up Kibana, tests for the Creating datastream info log, then restarts Kibana and tests for the Updating concrete indices info log and the success debug log
…nto event-log/update-mappings
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]Public APIs missing comments
History
To update your PR or re-run it, just comment with: cc @ymao1 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Sep 20, 2024
…and index template already exist (elastic#193205) Resolves elastic#192682 ## Summary As of 8.8, we started writing all event log documents to the `.kibana-event-log-ds` index. Prior to this, we created a new index template and data stream for every version (`.kibana-event-log-8.7` for example) so any mapping updates that were added for the version were created in the new index on upgrade. With the static index name and serverless, we need a way to update mappings in existing indices. This PR uses the same mechanism that we use for the alerts index to update the index template mappings and the mappings for the concrete backing indices of a datastream. ## To Verify Run ES and Kibana in `main` to test the upgrade path for serverless a. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E path.data=../test_el_upgrade` and Kibana `yarn start --ssl` b. Create a rule and let it run to populate the event log index c. Switch to this PR branch. Make a mapping update to the event log index: ``` --- a/x-pack/plugins/event_log/generated/mappings.json +++ b/x-pack/plugins/event_log/generated/mappings.json @@ -172,6 +172,9 @@ }, "rule": { "properties": { + "test": { + "type": "keyword" + }, "author": { "ignore_above": 1024, "type": "keyword", ``` d. Start ES and Kibana with the same commands as above e. Verify that the `.kibana-event-log-ds` index is created and has the updated mapping: - https://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template - https://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings I also verified the following: 1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when event log indices were versioned) to now 2. Run ES and Kibana in 8.15 to test the upgrade path from the previous release to now However, I had to create an 8.x branch and cherry pick this commit because `main` is now on 9.0 and we can't upgrade directly from older 8.x version to 9.0! --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit e2798de)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Sep 20, 2024
…tream and index template already exist (#193205) (#193589) # Backport This will backport the following commits from `main` to `8.x`: - [[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)](#193205) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ying Mao","email":"ying.mao@elastic.co"},"sourceCommit":{"committedDate":"2024-09-20T13:55:48Z","message":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)\n\nResolves #192682 Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","v9.0.0","Feature:EventLog","backport:prev-minor","v8.16.0"],"title":"[Response Ops][Event Log] Updating event log mappings if data stream and index template already exist","number":193205,"url":"#193205 Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)\n\nResolves #192682 Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"#193205 Ops][Event Log] Updating event log mappings if data stream and index template already exist (#193205)\n\nResolves #192682 Summary\r\n\r\nAs of 8.8, we started writing all event log documents to the\r\n`.kibana-event-log-ds` index. Prior to this, we created a new index\r\ntemplate and data stream for every version (`.kibana-event-log-8.7` for\r\nexample) so any mapping updates that were added for the version were\r\ncreated in the new index on upgrade.\r\n\r\nWith the static index name and serverless, we need a way to update\r\nmappings in existing indices. This PR uses the same mechanism that we\r\nuse for the alerts index to update the index template mappings and the\r\nmappings for the concrete backing indices of a datastream.\r\n\r\n## To Verify\r\n\r\nRun ES and Kibana in `main` to test the upgrade path for serverless \r\na. Check out `main`, run ES: `yarn es snapshot --license trial --ssl -E\r\npath.data=../test_el_upgrade` and Kibana `yarn start --ssl`\r\n b. Create a rule and let it run to populate the event log index\r\nc. Switch to this PR branch. Make a mapping update to the event log\r\nindex:\r\n\r\n```\r\n--- a/x-pack/plugins/event_log/generated/mappings.json\r\n+++ b/x-pack/plugins/event_log/generated/mappings.json\r\n@@ -172,6 +172,9 @@\r\n },\r\n \"rule\": {\r\n \"properties\": {\r\n+ \"test\": {\r\n+ \"type\": \"keyword\"\r\n+ },\r\n \"author\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\",\r\n```\r\n d. Start ES and Kibana with the same commands as above\r\ne. Verify that the `.kibana-event-log-ds` index is created and has the\r\nupdated mapping:\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template\r\n-\r\nhttps://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings\r\n\r\nI also verified the following:\r\n1. Run ES and Kibana in 8.7 to test the upgrade path from 8.7 (when\r\nevent log indices were versioned) to now\r\n2. Run ES and Kibana in 8.15 to test the upgrade path from the previous\r\nrelease to now\r\n\r\nHowever, I had to create an 8.x branch and cherry pick this commit\r\nbecause `main` is now on 9.0 and we can't upgrade directly from older\r\n8.x version to 9.0!\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e2798def07d50595806748dd64cccaa216c5e234"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Ying Mao <ying.mao@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #192682
Summary
As of 8.8, we started writing all event log documents to the
.kibana-event-log-dsindex. Prior to this, we created a new index template and data stream for every version (.kibana-event-log-8.7for example) so any mapping updates that were added for the version were created in the new index on upgrade.With the static index name and serverless, we need a way to update mappings in existing indices. This PR uses the same mechanism that we use for the alerts index to update the index template mappings and the mappings for the concrete backing indices of a datastream.
To Verify
Run ES and Kibana in
mainto test the upgrade path for serverlessa. Check out
main, run ES:yarn es snapshot --license trial --ssl -E path.data=../test_el_upgradeand Kibanayarn start --sslb. Create a rule and let it run to populate the event log index
c. Switch to this PR branch. Make a mapping update to the event log index:
d. Start ES and Kibana with the same commands as above
e. Verify that the
.kibana-event-log-dsindex is created and has the updated mapping:- https://localhost:5601/app/management/data/index_management/templates/.kibana-event-log-template
- https://localhost:5601/app/management/data/index_management/indices/index_details?indexName=.ds-.kibana-event-log-ds-2024.09.17-000001&filter=.kibana-&includeHiddenIndices=true&tab=mappings
I also verified the following:
However, I had to create an 8.x branch and cherry pick this commit because
mainis now on 9.0 and we can't upgrade directly from older 8.x version to 9.0!