Support to generate ECS fields as dynamic index template

[ypid-geberit]

Summary

The ECS tools currently generate ES index templates which use explicit mapping to express the ECS fields. I would like to discuss if dynamic templates could make sense in some cases.

Motivation:

Dynamic templates pro:

• It becomes less likely that you run into the total fields limit (ref: Mapping limit settings).
• Less confusing to users as only the fields will be shown in Kibana (for example KQL) that are actually used in a certain index set.
• More flexible for the Elastic admin as one index template can be used for multiple use cases. The index template can have the full ECS fieldsets. Indices will only create the needed fields.

Explicit mapping pro:

• It is against the currently proposed USAGE.
• Performance? I would think this is negligible as it only happens the first time a new field is indexed. So basically. The first event indexed into a new index will create most fields.

Detailed Design:

I can implement a POC after some feedback that show me that I have not overlooked something why dynamic templates are a no-no. For background, I have used dynamic templates for years for our logging use case without issues.

[kgeller]

Hi @ypid-geberit !

The ECS team has discussed recently how we want to consolidate down the artifacts that we create / maintain for users and have created a meta issue to start the discussion there.

I think that, combined with the notion that dynamic mappings are more intended for experimentation and exploration (see docs), we will probably stick with explicit mappings for the time being.

Please share any/all thoughts you have on the future of our generated templates on that meta issue. We haven't made any decisions yet and would love input from the community!

[ypid-geberit]

@kgeller This is a new feature/idea. I would prefer to keep this separate as the discussion could get long. It is not yet supported.

experimentation and exploration

To be precise about my idea. I intend to generate a very tight set of dynamic mapping rules that are basically equivalent of the current explicit mapping.

Is there a better way to address the points I identified as advantages of dynamic templates in #1600 (comment)?

[kgeller]

@ypid-geberit we can continue here! I will check with the wider ECS team to make sure I am not missing anything. In the meantime, I will re-open this, and also link you to another user's dynamic template generator in case it's of any use https://github.com/mbudge/dynamic-template-generator.

[ypid-geberit]

Thanks so much for linking to https://github.com/mbudge/dynamic-template-generator, I would probably never have found that! Looks like a good starting point for trying this.

Also thanks for reopening and checking.

[djptek]

@ypid-geberit while I can appreciate the potential flexibility that adding dynamic fields could add in the right use cases, there is one showstopper to incorporating these into ECS, which is the potential for performance issues in the wrong use cases. Those performance issues could be critical for our users.

Additionally, a generic format can never be 100% optimal, so we use benchmarking to validate that ECS performs comparably (or better) as the Elastic Stack and ECS Format evolve. We would no longer be able to benchmark if the mappings were to leverage dynamic fields.

I would recommend that dynamic fields be implemented as custom fields, alongside, but not within ECS

I think this has been a truly valuable discussion and thank you for your input, it will be really useful for anyone who has the same question to find it here in the future, thanks @ypid-geberit !