[meta] Consolidate generated ES index templates

[ebeahan]

Overview

With ECS aligning versioning with the Elastic stack releases starting in 8.0, the ECS tooling could be simplified to generate a single set of Elasticsearch index template artifacts. This single generated version would align with the stack release of the same version.

For example, ECS 8.0 produces sample Elasticsearch index mappings that are compatible with Elastic Stack 8.0.

Advantages

• Simplify the ECS tooling. No additional code to generate multiple versions of the index templates to support multiple versions of ES or multiple versions of the ES index template APIs.
• Reduce the copies of index templates in the repo; simplified user experience.
• Prevent users from trying to use the curated artifacts with the wrong APIs: example
• The sample templates are maintained as examples. They should be used as a starting point for experimentation. Users are still free to generate their own variations of index templates but shouldn't rely on ECS maintaining multiple copies.

Concerns

• Should the tooling allow users to maintain their own legacy index template artifacts (e.g. for users who maintain customer mappings)?
• The logstash-output-elasticsearch plugin depends on the ECS projects generated, legacy index template artifacts as part of their build for ECS compatibility mode. We will need to remove this dependency before we can remove the generated, legacy templates from ECS.

[ypid-geberit]

I was forwarded from #1600 to this issue with my idea to "Support to generate ECS fields as dynamic index template". I would propose to keep discussion of this new idea/feature in #1600. Otherwise, you are welcome to give feedback here.

[djptek]

Elasticsearch (7.x) Grok is now ECS compliant see:

elastic/elasticsearch#76885

https://github.com/elastic/elasticsearch/tree/7.x/libs/grok/src/main/resources/patterns/ecs-v1

https://github.com/elastic/elasticsearch/blob/dfc9df8c26dd4bdfd5728c23a2fa35d1b8f86cea/docs/reference/scripting/grok-syntax.asciidoc#migrating-to-elastic-common-schema-ecs

Will there be a dependency to ensure the Templates are aligned with the Grok Patterns?

[kgeller]

Removal of ES 6 support for ECS 8: #1680

[kgeller]

ES generated artifacts folder structure update: #1700
Subsequent Logstash update: logstash-plugins/logstash-output-elasticsearch#1060

[yaauie]

After the above restructuring, ECS's 8.0 branch is still generating ES7-style legacy templates (top-level "mappings" and "settings" fields), and needs to be updated to generate ES8-style legacy templates (top-level "template" field with "mappings" and "settings" sub-fields).

For reference, before the stack-alignment changes, Logstash's Elasticsearch output plugin had been providing ECS 8 support by transforming ES7-style generated templates into ES8-style during vendoring.

[ebeahan]

The distinction isn't between ES7 and ES8; it's between v1 index template API ("legacy" or the _template endpoint) and v2 index template (_index_template). My understanding is that both APIs will remain in ES8, although the legacy templates are considered deprecated.

After the above restructuring, ECS's 8.0 branch is still generating ES7-style legacy templates (top-level "mappings" and "settings" fields), and needs to be updated to generate ES8-style legacy templates (top-level "template" field with "mappings" and "settings" sub-fields).

Which index template endpoint is the Logstash Elasticsearch output plugin using for ECS 8.0 support: _template or _index_template?

With the templates ECS is generating, we've been following this pattern::

• A single, monolith legacy template - compatible with _template
• A composable and component templates (per field set) compatible with _index_template

We have not been maintaining a single, monolith template compatible with _index_template. Is that what the Logstash Elasticsearch output plugin expects?

[yaauie]

🤦🏼 that is on us. The Elasticsearch output plugin for Logstash uses the _index_template API when ushing templates to Elasticsearch 8+.

Thank you for the clarification. Knowing that the generated artifacts are stable in shape is sufficient for the Logstash team to move forward.

[ebeahan]

The Elasticsearch output plugin for Logstash uses the _index_template API when ushing templates to Elasticsearch 8+.

Thanks for clearing that up. 👍

@yaauie still unclear about one thing. Does the ES output plugin still need ECS to produce a single v2 index template with ALL ECS fields in one single template? I believe the answer is "yes," but I want to confirm.

[yaauie]

No. We can work from the "legacy" v1 templates, and in fact doing so simplifies our codepath. We already translate the "legacy"-style index templates for ECS 1.x when vendoring them for use with Elasticaearch 8, and can use the same code-path with the stack-aligned "legacy"-style templates. Moving the Stack-aligned monolith template to V2 adds complexity because we would need to translate those back into legacy for use with ES 7.x (due to our plugins implementation).

[kgeller]

Conclusion:

Since ECS is aligning versioning with the Elastic stack releases starting in 8.0, we have simplified the ECS tooling to generate two sets of Elasticsearch index template artifacts. We will now produce 1) a composable template compatible with the v2 _index_template API and 2) a legacy index template compatible with v1 "legacy" _template API.

Previously, at every ECS version, we were generating legacy templates compatible with ES 6 and ES 7, as well as a composable template. We have now consolidated down to, for every ECS version, a single ES 8 compatible legacy template and a single composable template.