[Response Ops][Alerting] Install component templates for FAAD that covers legacy rule registry technical fields #150358
Assignees
Labels
Feature:Alerting/Alerts-as-Data
Issues related to Alerts-as-data and RuleRegistry
Team:ResponseOps
Label for the ResponseOps team (formerly the Cases and Alerting teams)
Comments
ymao1
added
Team:ResponseOps
|
Pinging @elastic/response-ops (Team:ResponseOps) |
1 task
ymao1
added a commit
that referenced
this issue
Feb 27, 2023
…for framework alerts as data (#150384) Resolves #150358 ## Summary In a previous [PR](#145581) we started installing a common component template for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for its component templates. In this PR we are doing the following: * Renaming the installed `alerts-common-component-template` to `.alerts-framework-mappings`. * Creating and installing `.alerts-legacy-alert-mappings` component template when `enableFrameworkAlerts: true` on alerting plugin setup * The combination of the two component templates creates the same set of mappings as the rule registry technical component template * Creating and installing `.alerts-ecs-mappings` component template when `enableFrameworkAlerts: true` on alerting plugin setup (when `enableFrameworkAlerts: false`, the rule registry continues to install this component template * Using the `@kbn/ecs` package provided by core to generate the ECS field map. The rule registry will continue to install the existing ECS field map which is actually a subset of ECS fields * Adding `useLegacy` and `useEcs` flags that allow rule types to specify whether to include the legacy alerts component template and the ECS component template when registering with framework alerts-as-data. * Moved some common functions to alerting framework from the rule registry ## Things to note * When generating the ECS field map, we are now including the `ignore_above` setting from the `@kbn/ecs` package. This changes the ECS component template to include those settings. I tested updating an index with just `"type":"keyword"` mappings to add the `ignore_above` field to the mapping and had no issues so this seems like an additive change to the mapping that will hopefully prevent problems in the future. * The rule registry ECS component template also includes the technical fields which is redundant because the technical component template is automatically installed for all index templates so the framework ECS component template only contains ECS fields. | Previous mapping | Updated mapping | | ----------- | ----------- | | `{ "organization": { "type": "keyword" } }` | `{ "organization": { "type": "keyword", "ignore_above": 1024 } }` | ## To Verify ### Verify that the generated component templates are as expected: Get the following **While running `main`:** 1. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 2. Get the technical component template `GET _component_template/.alerts-technical-mappings` 3. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: false`:** 4. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 5. Get the technical component template `GET _component_template/.alerts-technical-mappings` 6. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true`:** 7. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 8. Get the technical component template `GET _component_template/.alerts-technical-mappings` 9. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` 10. Verify that component templates exist for `.alerts-framework-mappings` and `.alerts-legacy-alert-mappings` **Compare the ECS component templates** Compare 1 and 4 (ECS component template from `main` and installed by rule registry in this branch). The difference should be: * no difference in ECS fields * because the rule registry ECS component template also includes technical fields, you will see the 2 new technical fields in this branch Compare 4 and 7 (ECS component template from rule registry & alerting framework in this branch). * some new ECS fields for alerting installed template * each `keyword` mapped field for alerting installed template should have `ignore_above` setting * no `kibana.*` fields in the alerting installed template **Compare the technical component templates** Compare 2 and 5 (technical component template from `main` and installed by rule registry in this branch). The difference should be: * 2 new `kibana.alert` fields (`flapping_history` and `last_detected`) Compare 5 and 8 (technical component template from rule registry & alerting framework in this branch). * there should be no difference! **Compare the index mappings** Compare 3 and 6 (index mapping from `main` and installed by rule registry in this branch). The difference should be: * 2 new `kibana.alert` fields (`flapping_history` and `last_detected`) Compare 6 and 9 (index mapping from rule registry & alerting framework in this branch). * some new ECS fields * each `keyword` mapped ECS field should have `ignore_above` setting ### Verify that the generated component templates work with existing rule registry index templates & indices: 1. Run `main` or a previous version and create a rule that uses both ECS component templates & technical component templates (detection rules use both). Let it run a few times. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. Verify that the mapping on the existing `.internal.alerts-security.alerts-default-000001` has been updated to include the latest ECS mappings and the two new technical fields. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
wwang500
pushed a commit
to wwang500/kibana
that referenced
this issue
Feb 27, 2023
…for framework alerts as data (elastic#150384) Resolves elastic#150358 ## Summary In a previous [PR](elastic#145581) we started installing a common component template for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for its component templates. In this PR we are doing the following: * Renaming the installed `alerts-common-component-template` to `.alerts-framework-mappings`. * Creating and installing `.alerts-legacy-alert-mappings` component template when `enableFrameworkAlerts: true` on alerting plugin setup * The combination of the two component templates creates the same set of mappings as the rule registry technical component template * Creating and installing `.alerts-ecs-mappings` component template when `enableFrameworkAlerts: true` on alerting plugin setup (when `enableFrameworkAlerts: false`, the rule registry continues to install this component template * Using the `@kbn/ecs` package provided by core to generate the ECS field map. The rule registry will continue to install the existing ECS field map which is actually a subset of ECS fields * Adding `useLegacy` and `useEcs` flags that allow rule types to specify whether to include the legacy alerts component template and the ECS component template when registering with framework alerts-as-data. * Moved some common functions to alerting framework from the rule registry ## Things to note * When generating the ECS field map, we are now including the `ignore_above` setting from the `@kbn/ecs` package. This changes the ECS component template to include those settings. I tested updating an index with just `"type":"keyword"` mappings to add the `ignore_above` field to the mapping and had no issues so this seems like an additive change to the mapping that will hopefully prevent problems in the future. * The rule registry ECS component template also includes the technical fields which is redundant because the technical component template is automatically installed for all index templates so the framework ECS component template only contains ECS fields. | Previous mapping | Updated mapping | | ----------- | ----------- | | `{ "organization": { "type": "keyword" } }` | `{ "organization": { "type": "keyword", "ignore_above": 1024 } }` | ## To Verify ### Verify that the generated component templates are as expected: Get the following **While running `main`:** 1. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 2. Get the technical component template `GET _component_template/.alerts-technical-mappings` 3. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: false`:** 4. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 5. Get the technical component template `GET _component_template/.alerts-technical-mappings` 6. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true`:** 7. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 8. Get the technical component template `GET _component_template/.alerts-technical-mappings` 9. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` 10. Verify that component templates exist for `.alerts-framework-mappings` and `.alerts-legacy-alert-mappings` **Compare the ECS component templates** Compare 1 and 4 (ECS component template from `main` and installed by rule registry in this branch). The difference should be: * no difference in ECS fields * because the rule registry ECS component template also includes technical fields, you will see the 2 new technical fields in this branch Compare 4 and 7 (ECS component template from rule registry & alerting framework in this branch). * some new ECS fields for alerting installed template * each `keyword` mapped field for alerting installed template should have `ignore_above` setting * no `kibana.*` fields in the alerting installed template **Compare the technical component templates** Compare 2 and 5 (technical component template from `main` and installed by rule registry in this branch). The difference should be: * 2 new `kibana.alert` fields (`flapping_history` and `last_detected`) Compare 5 and 8 (technical component template from rule registry & alerting framework in this branch). * there should be no difference! **Compare the index mappings** Compare 3 and 6 (index mapping from `main` and installed by rule registry in this branch). The difference should be: * 2 new `kibana.alert` fields (`flapping_history` and `last_detected`) Compare 6 and 9 (index mapping from rule registry & alerting framework in this branch). * some new ECS fields * each `keyword` mapped ECS field should have `ignore_above` setting ### Verify that the generated component templates work with existing rule registry index templates & indices: 1. Run `main` or a previous version and create a rule that uses both ECS component templates & technical component templates (detection rules use both). Let it run a few times. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. Verify that the mapping on the existing `.internal.alerts-security.alerts-default-000001` has been updated to include the latest ECS mappings and the two new technical fields. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
bmorelli25
pushed a commit
to bmorelli25/kibana
that referenced
this issue
Mar 10, 2023
…for framework alerts as data (elastic#150384) Resolves elastic#150358 ## Summary In a previous [PR](elastic#145581) we started installing a common component template for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for its component templates. In this PR we are doing the following: * Renaming the installed `alerts-common-component-template` to `.alerts-framework-mappings`. * Creating and installing `.alerts-legacy-alert-mappings` component template when `enableFrameworkAlerts: true` on alerting plugin setup * The combination of the two component templates creates the same set of mappings as the rule registry technical component template * Creating and installing `.alerts-ecs-mappings` component template when `enableFrameworkAlerts: true` on alerting plugin setup (when `enableFrameworkAlerts: false`, the rule registry continues to install this component template * Using the `@kbn/ecs` package provided by core to generate the ECS field map. The rule registry will continue to install the existing ECS field map which is actually a subset of ECS fields * Adding `useLegacy` and `useEcs` flags that allow rule types to specify whether to include the legacy alerts component template and the ECS component template when registering with framework alerts-as-data. * Moved some common functions to alerting framework from the rule registry ## Things to note * When generating the ECS field map, we are now including the `ignore_above` setting from the `@kbn/ecs` package. This changes the ECS component template to include those settings. I tested updating an index with just `"type":"keyword"` mappings to add the `ignore_above` field to the mapping and had no issues so this seems like an additive change to the mapping that will hopefully prevent problems in the future. * The rule registry ECS component template also includes the technical fields which is redundant because the technical component template is automatically installed for all index templates so the framework ECS component template only contains ECS fields. | Previous mapping | Updated mapping | | ----------- | ----------- | | `{ "organization": { "type": "keyword" } }` | `{ "organization": { "type": "keyword", "ignore_above": 1024 } }` | ## To Verify ### Verify that the generated component templates are as expected: Get the following **While running `main`:** 1. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 2. Get the technical component template `GET _component_template/.alerts-technical-mappings` 3. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: false`:** 4. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 5. Get the technical component template `GET _component_template/.alerts-technical-mappings` 6. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true`:** 7. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 8. Get the technical component template `GET _component_template/.alerts-technical-mappings` 9. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` 10. Verify that component templates exist for `.alerts-framework-mappings` and `.alerts-legacy-alert-mappings` **Compare the ECS component templates** Compare 1 and 4 (ECS component template from `main` and installed by rule registry in this branch). The difference should be: * no difference in ECS fields * because the rule registry ECS component template also includes technical fields, you will see the 2 new technical fields in this branch Compare 4 and 7 (ECS component template from rule registry & alerting framework in this branch). * some new ECS fields for alerting installed template * each `keyword` mapped field for alerting installed template should have `ignore_above` setting * no `kibana.*` fields in the alerting installed template **Compare the technical component templates** Compare 2 and 5 (technical component template from `main` and installed by rule registry in this branch). The difference should be: * 2 new `kibana.alert` fields (`flapping_history` and `last_detected`) Compare 5 and 8 (technical component template from rule registry & alerting framework in this branch). * there should be no difference! **Compare the index mappings** Compare 3 and 6 (index mapping from `main` and installed by rule registry in this branch). The difference should be: * 2 new `kibana.alert` fields (`flapping_history` and `last_detected`) Compare 6 and 9 (index mapping from rule registry & alerting framework in this branch). * some new ECS fields * each `keyword` mapped ECS field should have `ignore_above` setting ### Verify that the generated component templates work with existing rule registry index templates & indices: 1. Run `main` or a previous version and create a rule that uses both ECS component templates & technical component templates (detection rules use both). Let it run a few times. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. Verify that the mapping on the existing `.internal.alerts-security.alerts-default-000001` has been updated to include the latest ECS mappings and the two new technical fields. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In #145581 we started installing FAAD resources (behind a feature flag). We installed a common component template that covers all the
kibana.alert.*fields that would be used by the framework. This is a subset of the technical component template that is currently used by index templates in the rule registry. We also chose a slightly different naming convention than what is used by the rule registry but have since decided that we should use the same convention.This issue is for
alerts-common-component-templateto.alerts-framework-mappingsThe text was updated successfully, but these errors were encountered: