[Response Ops][Alerting] Install resources needed for framework alerts-as-data

[ymao1]

Resolves #145100

Summary

• Adds alerting config for enableFrameworkAlerts
• When xpack.alerting.enableFrameworkAlerts=true, AlertsService is initialized during the alerting plugin setup phase
• Adds optional alerts definition to the RuleType definition which allows a rule type to specify a context and a field mapping for that context
• AlertsService.initialize()
  • installs an ILM policy that will be used by all alerts-as-data indices - alerts-default-ilm-policy
  • installs a component template that will be used by all alerts-as-data indices - alerts-common-component-template, including all the mappings for fields needed by the framework
• Rule type registration - when a rule type with an alerts definition is registered, the context and field map are registered with the AlertService. When AlertsService initialization is completed successfully, context specific resources are installed.
• Context specific resources:
  • installs context specific component template that reflects the registered field map - alerts-${context}-component-template
  • installs context specific index template for the default namespace - .alerts-${context}-default-template
  • creates context specific concrete write index for the default namespace - .alerts-${context}-default-000001
• Resource installation retries for transient ES errors and short-circuits when a timeout value is reached

Notes

• We explore the possibility of creating a single index template per context and re-using it for space-aware concrete indices but a rollover alias (which must be space-aware) must be attached to an index template so it is not feasible to only have a single index template
• There will be a followup issue for create space-specific index templates/indices as needed to support detection rules desire to separate alerts by space.

To Verify

• set xpack.alerting.enableFrameworkAlerts: true in your kibana.yml
• modify x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.ts to define a custom field mapping for the stack context

--- a/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.ts
+++ b/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.ts
@@ -187,5 +187,14 @@ export function getRuleType(
     },
     producer: STACK_ALERTS_FEATURE_ID,
     doesSetRecoveryContext: true,
+    alerts: {
+      context: 'stack',
+      fieldMap: {
+        'kibana.alert.threshold': {
+          required: false,
+          type: 'long',
+        },
+      },
+    },
   };
 }

• start up ES and Kibana and verify that the correct resources are installed. The following should be created:
  • ILM policy - alerts-default-ilm-policy
  • Common component template - alerts-common-component-template
  • Stack component template - alerts-stack-component-template
  • Stack index template for default space - .alerts-stack-default-template
  • Stack write index for default space - .alerts-stack-default-000001
• verify that the index template uses both component templates and that the expected mappings are applied to the index
• try making non-destructive, additive only changes to the common component template or the stack context field mapping and verify that the concrete index mappings are updated.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[ymao1]

@elasticmachine merge upstream

[CoenWarmer]

AO owned changed LGTM

[pmuellr]

Structure looks great! Added some nit comments ...

[pmuellr]

I'm always a little worried about touching existing indices. Did we have some cases of updating the event log indices that caused problems? Closed indices or something like that? It's tough because if we are kinda "lenient" for some of these errors, they'll just come back and bite us later.

Feels like the code below is probably the best thing now - log an error early exit.

[pmuellr]

Could we set a flag in the pluginStop$ observer to check if we're shutting down instead? Or make the message a const and use it both to throw the error message AND check it here?

Actually, what happens if we throw the error during shutdown, and just skip the conditional?

[pmuellr]

could add some meta here too, maybe? I guess _meta.managed is defined by the platform, can we add more?

[pmuellr]

Let's remember to ask folks to add this to their kibana.dev.ymlfiles, after the merge, so we'll get some early testing of the initialization bits.

[ymao1]

@elasticmachine merge upstream

[ymao1]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 3d54034
• Webpack Bundle Analyzer

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
apm	1364	1365	+1
cases	588	589	+1
infra	1221	1222	+1
observability	495	496	+1
securitySolution	3534	3535	+1
synthetics	1336	1337	+1
triggersActionsUi	495	496	+1
total			+7

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/rule-data-utils	94	96	+2
alerting	456	457	+1
total			+3

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
alerting	38	39	+1

Unknown metric groups

API count

id	before	after	diff
@kbn/rule-data-utils	97	99	+2
alerting	466	467	+1
total			+3

ESLint disabled line counts

id	before	after	diff
alerting	71	77	+6

Total ESLint disabled count

id	before	after	diff
alerting	72	78	+6

History

• 💚 Build #101861 succeeded 19b8818
• 💔 Build #101805 failed cb680ef
• 💔 Build #101774 failed b113946
• 💛 Build #101228 was flaky 3ff7e81
• 💚 Build #100797 succeeded 4080def
• 💚 Build #100695 succeeded 9176eec

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ymao1