[Security Solution] Event/Alert Details JSON View displays nested objects inaccurately

[rylnd]

The Event/Alert Details flyout displays nested objects (type: nested) as a single object, and in doing so loses information about the true underlying structure of those nested documents.

Affects Kibana 7.11+

Display in JSON View:

Document in the signals index:

Note here that threat.indicator.url is unmapped.

Related to #90222, but distinct enough that I filed a separate issue.

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[MadameSheema]

@karanbirsingh-qasource can you please validate the fix of this issue on 7.12.BC3? Thanks :)

[ghost]

Hi @MadameSheema & @rylnd

Thanks for the update.

we have validated this issue on 7.12.0 BC5 and Found it Fixed. However we have one Question regarding the ticket.

Build Details:

Version: 7.12.0 BC5
Commit:b7f9a41f486a2910ef22a1274ec734219c35ca3e
Version : 39309

Question:

Q1. On our Sample data Event/Alert Details JSON View , the JSON data is encoded , however we have Unescaped that same data and check the order and structuring was correct and matching to the one of GET Document signal .

So can you please confirm if this json data being encoded under Event/Alert Details can be also be concern part of this reported issue or not?

"threat": {
   "indicator": "{\"domain\":\"117.242.211.13\",\"matched\":{\"atomic\":\"117.242.211.13\",\"field\":\"mydata.domain\",\"id\":\"-TxzWXgBajxX6dLQFGo-\",\"index\":\"indicator\"}}"
 }
}

"threat_mapping": "{\"entries\":[{\"field\":\"mydata.domain\",\"type\":\"mapping\",\"value\":\"threatintel.indicator.domain\"}]}",

• Event/Alert Detal JSON View : Detection_Alert_JSON_view.zip
  Below displayed json data is Unescaped to the original encoded json data

"threat": {
  "indicator": "{"domain":"117.242.211.13","matched":{"atomic":"117.242.211.13","field":"mydata.domain","id":"-TxzWXgBajxX6dLQFGo-","index":"indicator"}}"
}
}

• Document in the signals index:GET Dection Signal index.zip

    "threat" : {
      "indicator" : [
        {
          "domain" : "117.242.211.13",
          "matched" : {
            "atomic" : "117.242.211.13",
            "field" : "mydata.domain",
            "id" : "-TxzWXgBajxX6dLQFGo-",
            "index" : "indicator"
          }
        }
      ]
    }

Observations:

Steps followed to created Nested Data Detection Alert

1. With the help of devtools, generate the following indexes and docs:

PUT indicator
PUT /indicator/_mapping
{
"properties": {
"@timestamp": {
"type": "date"
},
"threatintel": {
"properties" : {
"indicator": {
"properties": {
"domain": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
}
}
}
POST /indicator/_doc
{
"@timestamp": "2021-02-22T21:00:49.337Z",
"threatintel" : {
"indicator" : {
"domain" : "117.242.211.13"
}
}
}

PUT mydata
PUT /mydata/_mapping
{
"properties": {
"@timestamp": {
"type": "date"
},
"mydata": {
"properties": {
"domain": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
}
POST /mydata/_doc
{
"@timestamp": "2021-02-22T21:00:49.337Z",
"mydata": {
"domain": "117.242.211.13"
}
}

2. Create an indicator match rule with the following details:

   index patterns: mydata
   indicator index patterns: indicator
   Indicator mapping field: mydata.domain
   Indicator index field: threatintel.indicator.domain
   Runs every: 10 seconds
   Additional look-back time: 30000h

3. Wait for the rule to generate the alert

4. With the customize columns option add: threat.indicator.domain

thanks !!

[MadameSheema]

@rylnd correct me if I'm wrong but that is correct, right?

[rylnd]

The escaping of nested objects is expected; there are several other examples of this:

There is one outstanding issue here, which is that nested objects are missing the encapsulating array in the escaped string format. This is demonstrated in @karanbirsingh-qasource's example; you can see that the indicators: [{}] was indexed but indicators: {} was displayed.

I would say that the structure is not truly correct as asserted above, but as each object is presented and it's still valid JSON this is acceptable for now. Here's the basic format if you have multiple nested objects:

"threat": {
    "indicator": "{},{}"
  },

@MadameSheema I see you changing labels/assignees here. Would you like to close this issue, keep it open for the issue above, or something else?

[MadameSheema]

Thanks @rylnd let's keep it opened as an enhancement know to finish the work on it ;)

[XavierM]

@rylnd and @monina-n should talk about how we are going to represent nest fields inside of the t-grid.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)