Formal Concept of "Dynamic" Object Fields within ECS

[BenB196]

Summary

I would like to propose an addition to ECS that formalizes the concept of "dynamic" object fields within ECS.

Motivation:

The motivation behind this has to do with limitations imposed within Kibana that prevent certain features from working unless they are formally recognized within ECS. (mainly affecting the Security features of Kibana, i.e.: Filter Alerts by Field, Close Existing Alerts via Exception rule, etc.)

Note: That the above limitations make sense as to not cause an overlap or conflict within the field mappings, and I'm not 100% sure if the change I'm proposing here is the correct one, but I wanted to get a conversation started about it.

Detailed Design:

Currently ECS has statically defined object mappings, my proposal is to add a sub-object specifically dedicated to hosting user created fields that guarantee that they won't conflict with the statically defined object mappings.

• Field names:
  • Something like dynamic_ecs
• Example values for the fields:
  • geo.dynamic_ecs.*:
    • geo.dyanmic_ecs.currency
    • geo.dyanmic_ecs.shape
    • geo.dynamic_ecs.custom_field_123
    • geo.dynamic_ecs.proxy
  • user.dynamic_ecs.*:
    • user.dynamic_ecs.interal_role
    • user.dynamic_ecs.custom_object.*:
      • user.dynamic_ecs.custom_object.secondary_email
• Suggested appropriate datatypes
  I would like this new field to be a data type that is an object, but that has dynamic fields disabled, requiring the user to specify the fields which they intend to use within the mapping.
• Any example events that map to the proposed use case(s)

While not an example event, here are two use-cases where I think there is a value add for this type of feature.

1. I currently enrich IP information with additional fields that provide value when investigating events, but I can't take full advantage of these with in the Security app as they aren't ECS compliant, and they are fields which might not be worth adding into the official ECS.

Example Fields:

• geo.proxy : bool (is this IP a known proxy IP)
• geo.hosting : bool (is this IP a known hosting IP)
• geo.mobile : bool (is this IP a known mobile IP)
• geo.reverse : keyword (reverse record at the time of the event)

With this change I could do something like:

• geo.dynamic_ecs.proxy
• geo.dynamic_ecs.hosting
• geo.dynamic_ecs.mobile
• geo.dynamic_ecs.reverse

And these fields would be considered "ECS" compliant so that things like Kibana could take advantage of them.

2. I currently enrich user information with internal fields, and like use case 1, I can't easily take full advantage of these within Kibana Security

Example Fields:

• user.organization.role : keyword (the current role/job the user has)
• user.organization.department : keyword (the current department of the user)
• user.organization.manager_id : keyword (the Id of the current manager of the user)

With this change I could do something like:

• user.dynamic_ecs.organization.role
• user.dynamic_ecs.organization.department
• user.dynamic_ecs.organization.manager_id

And these fields would be considered "ECS" compliant so that things like Kibana could take advantage of them.

Note: I don't really like the term dynamic here, as I don't believe that the ability for the object to truly be dynamic in the mapping should be supported here. I mean it more in the sense that users can specify their own mappings within this object, and they would be considered ECS compliant.

[kgeller]

Hi @BenB196 ! If I follow correctly, you're seeking a solution to being able to use your own custom fields in the security solution, and you're proposing "dynamic" objects to do so?

As far as having a space for "dynamic" objects to go, this idea has come up before and a similar idea was explored in this RFC.

When it comes to being able to use your own custom fields in those security features like Filter Alerts by Field, Close Existing Alerts via Exception rule and others, I will defer to product as I am less familiar @jamiehynds .

[BenB196]

Thanks @kgeller regarding the RFC, it looks like it somewhat covers the scope of my request. The one thing that I think it differs slightly on, is the RFC states that object would be a top-level field. From my understanding of the a top-level field within the context of ECS, is that it can't be used under another object (i.e.: user object). If this is an incorrect understanding, please correct me.

[kgeller]

So top-level just indicates where it sits in the schema. If the field were to have re-uses, then you'd be able to nest it under another object.

Example would be the user field set, which is top level, but can be nested under the objects listed here.

[djptek]

Hi @BenB196 this is related, though not the same as a recent issue raised regarding Dynamic Templates which highlights the conflicts inherent in dynamic vs static behaviour

If the use case requires

user created fields that guarantee that they won't conflict with the statically defined object mappings

perhaps this can be addressed using the ECS toolset for adding Custom Fields