[Security Solution][Timeline] Dragging a nested field to timeline does not generate the correct query

[rylnd]

Describe the bug:
Dragging a nested field to the timeline does not filter events to those matching the field.

Kibana/Elasticsearch Stack version:
7.11

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):
Timeline

Steps to reproduce:

1. Generate a document with a nested field
2. Drag a value from the nested field to the timeline
3. Observe that the field has added to the timeline, but the events have not been refined

Current behavior:
All events are shown; no filtering occurs

Expected behavior:
Timeline events are filtered to those matching the nested field/value

Screenshots (if relevant):
Nested field in signals table:

nested field in timeline (no indication of nested type):

relevant portion of the inspected timeline query:

relevant portion of the timeline network request:

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[stephmilovic]

Partially resolved here. This is what still is broken:

[MadameSheema]

Reopening since the issue has not been closed yet.

[MadameSheema]

Tagged as critical after a conversation with @MikePaquette

[ghost]

Thanks @MadameSheema for providing the api index hits details to have the sample nested data .

we have followed the steps and successfully got the nested data in indicator match rule on 7.12.0 BC3 , However issue is still occuring incorrect result is returning for nested data . we will retest this issue on 7.12.0 BC4.

Build Details:

Version: 7.12.0 BC3
Commit:08417cbd6c15e4c866651a7dcdfeded58845206d
Build:39134

Snap-Shoot:

thanks !!

[rylnd]

Reopening until @karanbirsingh-qasource can confirm this fix on a BC.

[MadameSheema]

@karanbirsingh-qasource can you please prioritise the validation of this fix? Thanks

[ghost]

Hi @MadameSheema and @rylnd

We have validated this issue on 7.12.0 BC4 and Observed that issue is Still occurring incorrect result is returning.

Build Details:

Version: 7.12.0-BC4
Platform: Production
Commit:99ac38d70e426f589bb61a034c96e602d759cfab
Build:39242
https://staging.elastic.co/7.12.0-336ff10d/summary-7.12.0.html

Screenshot:

Timeline:

Thanks!!

[MadameSheema]

@deepikakeshav-qasource please validate the fix on the next BC. Thanks :)

[XavierM]

yes, I am pretty sure it is working on the data provider but not on the filter in/out just under KQL

[ghost]

Hi @MadameSheema,

We have validated this issue on 7.12.0 BC5 build and Observed that issue is Fixed. Correct result is returning under Timeline

Build Details:

Version:7.12.0 BC5
Platform: Production
Commit:b7f9a41f486a2910ef22a1274ec734219c35ca3e
Build:39309
Artifact Page: https://staging.elastic.co/7.12.0-583fca05/summary-7.12.0.html

Screenshot:

Hence, we are closing this issue and adding "QA Validate" Label to it.

Thanks!!

[XavierM]

What I mean here, if you are using the menu context on a nested field, the filter out/in won't work as right now. However @kqualters-elastic is working on a fix for 7.12.1 Please, Kevin add this bug to you PR. So QA can test it.

[ghost]

Hi @MadameSheema,

We tested the latest 7.12.0 doc link and found that documentation for this ticket is not yet updated.

We will test again once the changes are updated in Doc.

Thanks!

[kqualters-elastic]

After some debugging with @XavierM we've come to the conclusion that this is actually working as expected, for both draggable filtering as well as adding/removing filters via the context menu.

[ghost]

Hi @MadameSheema

we have checked that this issue start occuring again on 7.15.0 BC1 . The Timeline result did not filter correctly on adding nested field threat.enrichments.indicator.domain and mydata.domain

Please find below complete details:

Build Details:

Version:7.15.0
Commit:d791226d9385122f33f4a5ca38fa5369012fbec3
Build:43636

Screen-Cast:

nested-data.mp4

Dev Nested Data HIT:

PUT indicator

PUT /indicator/_mapping
{
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "threatintel": {
            "properties" : {
            "indicator": {
              "properties": {
                "domain": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
          }
        }
            }
}
}
}

POST /indicator/_doc
{
"@timestamp": "2021-02-22T21:00:49.337Z",
"threatintel" : {
            "indicator" : {
              "domain" : "117.242.211.13"
            }
}
}

PUT mydata

PUT /mydata/_mapping
{
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "mydata": {
              "properties": {
                "domain": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
          }
        }
}
}

POST /mydata/_doc
{
  "@timestamp": "2021-02-22T21:00:49.337Z",
  "mydata": {
    "domain": "117.242.211.13"
  }
}

[MadameSheema]

@kqualters-elastic @XavierM any update regarding this? thanks

[ecezalp]

did a little investigation around this bug, and here are my findings. To sum up, I believe the currently observed behavior isn't related to nested field types, but it is about non-ECS-compliant fields sneaking into the timeline view, which does not seem to be supported for investigation.

Setup (reproduction)

1. add new indices to kibana.dev.yml

// kibana.dev.yml

xpack.securitySolution.signalsIndex: .siem-signals-89784
kibana.index: .kibana-89784

note: there is no significance to number 89784, it's just the number of this issue.
note: in kibana.dev.yml there are no other significant configurations (for instance, ruleRegistry is not enabled)

2. start elasticsearch & kibana and visit the security app
3. go to Kibana devtools and make the requests from Karanbir's earlier comment, pasted below for convenience

requests

PUT indicator

PUT /indicator/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "threatintel": {
      "properties": {
        "indicator": {
          "properties": {
            "domain": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

POST /indicator/_doc
{
  "@timestamp": "2021-08-23T21:00:49.337Z",
  "threatintel": {
    "indicator": {
      "domain": "117.242.211.13"
    }
  }
}

PUT mydata

PUT /mydata/_mapping
{
  "properties":{
    "@timestamp":{
      "type":"date"
    },
    "mydata":{
      "properties":{
        "domain":{
          "ignore_above":1024,
          "type":"keyword"
        }
      }
    }
  }
}

POST /mydata/_doc
{
  "@timestamp": "2021-08-22T21:00:49.337Z",
  "mydata": {
    "domain": "117.242.211.13"
  }
}

4. create an indicator match rule with the following configuration

5. create a custom query rule with the following configuration

6. observe that one alert for each rule is generated

7. Open alert summary flyout from either one of the alerts, click on the Table tab, find mydata.domain field, and add the column to the Alerts table

8. Click on the timeline icon next to mydata.domain field, and notice that no alerts show up

Findings / Further Investigation

1. Considering that the alert generated by the custom query rule also did not show up, I think that we can eliminate the possibility that the bug has to do with nested field types.

2. it doesn't seem possible to manually add the mydata.domain field on the timeline once it's removed.

3. it is also not possible to manually add the mydata.domain field on the Alerts table once it the fields are reset

4. other fields such as signal.rule.name or threat.enrichments.matched.field work as expected on timeline, both with manual KQL entry and from the timeline button on the Alerts table once the field is added through the fields button.

Conclusion

It looks like we don't have timeline support for non-ECS-compliant fields. We do seem to get data on the Alerts table if the user adds a column from the Table tab of the Alert Summary Flyout, but it's not normally possible to add the column using the fields button on the table. I can envision two solutions that could help resolve this issue:

1. remove "add to timeline" button from non-ECS-compliant fields if they make their way on the Alerts table
2. remove "column toggling" functionality on the Alert Summary Flyout for non-ECS-compliant fields

I would appreciate a review of my findings and a product decision regarding the next steps if possible.

Thanks for reading 👍

[rylnd]

@karanbirsingh-qasource @MadameSheema I agree with @ecezalp 's findings above; it looks like this is working as expected for mapped fields, nested or no.

One contributor to the behavior seen here is the fact that threat.indicator.domain is no longer an ECS CTI field, and has moved instead to threat.indicator.url.domain. We had been using this for many CTI examples; I would suggest updating scenarios to use the new, analogous field instead. In @karanbirsingh-qasource's video, it should be possible to e.g. send threat.enrichments.matched.atomic to the timeline, which verifies that this is not a bug with nested fields.

If there is a bug here, it's the fact that we add unmapped fields to the alerts table, and then send those columns to the timeline, but I believe a separate ticket should be opened to address that case.

[ghost]

did a little investigation around this bug, and here are my findings. To sum up, I believe the currently observed behavior isn't related to nested field types, but it is about non-ECS-compliant fields sneaking into the timeline view, which does not seem to be supported for investigation.

Setup (reproduction)

1. add new indices to `kibana.dev.yml`

// kibana.dev.yml

xpack.securitySolution.signalsIndex: .siem-signals-89784
kibana.index: .kibana-89784

note: there is no significance to number 89784, it's just the number of this issue.
note: in kibana.dev.yml there are no other significant configurations (for instance, ruleRegistry is not enabled)

1. start elasticsearch & kibana and visit the security app

2. go to Kibana devtools and make the requests from Karanbir's earlier [comment](https://github.com/elastic/kibana/issues/89784#issuecomment-902621705), pasted below for convenience

requests

1. create an indicator match rule with the following configuration

1. create a custom query rule with the following configuration

1. observe that one alert for each rule is generated

1. Open alert summary flyout from either one of the alerts, click on the Table tab, find mydata.domain field, and add the column to the Alerts table

1. Click on the timeline icon next to mydata.domain field, and notice that no alerts show up

# Findings / Further Investigation

1. Considering that the alert generated by the custom query rule also did not show up, I think that we can eliminate the possibility that the bug has to do with nested field types.

2. it doesn't seem possible to manually add the `mydata.domain` field on the timeline once it's removed.

1. it is also not possible to manually add the `mydata.domain` field on the Alerts table once it the fields are reset

1. other fields such as `signal.rule.name` or `threat.enrichments.matched.field` work as expected on timeline, both with manual KQL entry and from the timeline button on the Alerts table once the field is added through the fields button.

Conclusion

It looks like we don't have timeline support for non-ECS-compliant fields. We do seem to get data on the Alerts table if the user adds a column from the Table tab of the Alert Summary Flyout, but it's not normally possible to add the column using the fields button on the table. I can envision two solutions that could help resolve this issue:

1. remove "add to timeline" button from non-ECS-compliant fields if they make their way on the Alerts table

2. remove "column toggling" functionality on the Alert Summary Flyout for non-ECS-compliant fields

I would appreciate a review of my findings and a product decision regarding the next steps if possible.

Thanks for reading 👍

Thanks @ecezalp for looking into the issue and provide the detailed observation for the same .

As per the suggestion from Ryland i am opening up the seperate ticket for the above conclusion enhancement planned for the add unmapped fields to the alerts table, and then send those columns to the timeline .

Here is the ticket link for it #110002

[ghost]

Thanks @rylnd for sharing the update of the change of the ECS CTI Field from the earlier one we were using.

The issue is Fixed ✔️ for the threat.enrichments.matched.atomic filed .

we have updated the old related test-case for it with the new filed that is threat.enrichments.matched.atomic .

• C608370

Observations:

Please find below detailed observation for the threat.enrichments.matched.atomic filed

• User is able to add threat.enrichments.matched.atomic filed from Field icon in utility bar
  

• User is able to add threat.enrichments.matched.atomic field under timeline as Add field
  

• Timeline result is getting filtered out correctly for this field
  

Hence we are closing issue and opened new ticket for the separate issue of unmapped fields to the alerts table.

thanks !!

c.c @MadameSheema

[MadameSheema]

Thanks @ecezalp @rylnd

@rylnd taking into account this issue opened by @peluja1012 #90346, unmapped fields should be displayed in the alert details view, even we have test covering that area. It has been any change on the behaviour of unmapped fields I'm not aware of?