[Security Solutions][Timeline] Non indexed fields are not showing up in the table view

[FrankHassanabad]

Describe the bug:

In timeline when you have an unindexed field after creating an alert and you select it in the details view it looks like the values are not showing up in the table. Tested 7.11.1-BC-1

Steps to reproduce:

1. Create a cloud instance from ece
2. Download auditbeats and set it up.
3. Create a rule that looks for "hosts: *"
4. Go to the output of the rule and select one of the rows.
5. Select a field that is un-indexed to view in the table. Look at the table and notice that the value is not there.

Expected behavior:
The table shows the unindexed values in the columns.

Kibana version:
7.11.1-BC-1

Elasticsearch version:
7.11.1-BC-1

Server OS version:
any

Browser version:
any

Browser OS version:
any

Original install method (e.g. download page, yum, from source, etc.):
cloud

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[MadameSheema]

@karanbirsingh-qasource can you please validate the fix of this issue on 7.11.2BC2 and 7.12BC3? thanks :)

[ghost]

Hi @MadameSheema

we have validated this issue on 7.12.0 BC3and 7.11.2 BC2 and on both "user.effective.group.id" field is not present either in the detection alert list column and nor in the Table [ right details fly out ].

so could you please confirm if is this is expected fix of this ticket to remove that field [i.e. non ecs that is not indexed and is also an object ] from both area (alert list column and table column).

Additionally we have found related field"user.group.id"( below find observation for same)

Build Details:

Version: 7.12.0 BC3
Commit: 08417cbd6c15e4c866651a7dcdfeded58845206d
Build:39134

Version: 7.11.2 BC2
Commit:bdba929767160a3272f5144acd9270d7bdaaea7c
Build:38015

Snapshot:

Observation:

-Created a custom rule with host.name:* query
-Validated any one of the alert entry details

7.12.0 JSON Details of Alert

{
  "_id": "d7754bbdbc0beb5685e24ffdab4e067182da9b22efa92d252088e4941931013a",
  "_index": ".siem-signals-default-000001",
  "_score": "1",
  "_type": "_doc",
  "@timestamp": "2021-03-10T09:01:04.512Z",
  "agent": {
    "ephemeral_id": "cda430cf-164a-404a-b220-b0f3de9b429a",
    "hostname": "qasource",
    "id": "00ca492e-bbc9-47a7-aa93-d159bc585274",
    "name": "qasource",
    "type": "filebeat",
    "version": "7.12.0"
  },
  "data_stream": {
    "dataset": "elastic_agent",
    "namespace": "default",
    "type": "logs"
  },
  "ecs": {
    "version": "1.8.0"
  },
  "elastic_agent": {
    "id": "34e99100-816e-11eb-8402-556c045a42e6",
    "version": "7.12.0"
  },
  "event": {
    "dataset": "elastic_agent",
    "kind": "signal"
  },
  "host": {
    "architecture": "x86_64",
    "hostname": "qasource",
    "id": "4143c277-074e-47a9-b37d-37f94b508705",
    "ip": "10.0.5.175",
    "mac": "00:50:56:b1:af:f1",
    "name": "qasource",
    "os": {
      "build": "19042.867",
      "family": "windows",
      "kernel": "10.0.19041.867 (WinBuild.160101.0800)",
      "name": "Windows 10 Pro",
      "platform": "windows",
      "type": "windows",
      "version": "10.0"
    }
  },
  "input": {
    "type": "log"
  },
  "log": {
    "file": {
      "path": "C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-d93154\\logs\\elastic-agent-json.log"
    },
    "level": "error",
    "offset": "9681659",
    "origin": {
      "file": {
        "line": "130",
        "name": "process/start.go"
      }
    }
  },
  "message": "failed writing connection info to spawned application: failed to write connection information: write |1: The pipe has been ended.",
  "signal": {
    "_meta": {
      "version": "25"
    },
    "ancestors": "{\"id\":\"5slYG3gBkWRqY_Alx-Bh\",\"type\":\"event\",\"index\":\".ds-logs-elastic_agent-default-2021.03.10-000001\",\"depth\":0}",
    "depth": "1",
    "original_event": {
      "dataset": "elastic_agent"
    },
    "original_time": "2021-03-10T08:55:05.579Z",
    "parent": {
      "depth": "0",
      "id": "5slYG3gBkWRqY_Alx-Bh",
      "index": ".ds-logs-elastic_agent-default-2021.03.10-000001",
      "type": "event"
    },
    "parents": "{\"id\":\"5slYG3gBkWRqY_Alx-Bh\",\"type\":\"event\",\"index\":\".ds-logs-elastic_agent-default-2021.03.10-000001\",\"depth\":0}",
    "rule": {
      "actions": "",
      "author": "",
      "created_at": "2021-03-10T08:51:47.442Z",
      "created_by": "elastic",
      "description": "checking unindexed field",
      "enabled": "true",
      "exceptions_list": "",
      "false_positives": "",
      "filters": "",
      "from": "now-360s",
      "id": "d8aa59f0-817d-11eb-8402-556c045a42e6",
      "immutable": "false",
      "index": "apm-*-transaction*,auditbeat-*,endgame-*,filebeat-*,logs-*,packetbeat-*,winlogbeat-*",
      "interval": "1m",
      "language": "kuery",
      "license": "",
      "max_signals": "100",
      "meta": {
        "from": "5m",
        "kibana_siem_app_url": "https://a418827c149f48869e4709b94d216d13.europe-west1.gcp.cloud.es.io:9243/app/security"
      },
      "name": "Ticket Regress #91424",
      "output_index": ".siem-signals-default",
      "query": "host.name : * ",
      "references": "",
      "risk_score": "21",
      "risk_score_mapping": "",
      "rule_id": "ef48427f-6dd6-4cda-9260-8701fd317243",
      "severity": "low",
      "severity_mapping": "",
      "tags": "",
      "threat": "",
      "to": "now",
      "type": "query",
      "updated_at": "2021-03-10T09:00:05.155Z",
      "updated_by": "elastic",
      "version": "1"
    },
    "status": "open"
  }
}

7.11.2 JSON Details of Alert

{
  "_id": "5ec63e17fa2f209d0cc0f59058042cab4600362c1da319f5cc84fbc1d728b5eb",
  "_index": ".siem-signals-default-000001",
  "_score": "1",
  "_type": "_doc",
  "@timestamp": "2021-03-10T09:01:04.511Z",
  "agent": {
    "id": "d3683551-c059-dd12-c82e-19ffa7f87246",
    "type": "endpoint",
    "version": "7.12.0"
  },
  "data_stream": {
    "dataset": "endpoint.events.library",
    "namespace": "default",
    "type": "logs"
  },
  "dll": {
    "code_signature": {
      "exists": "true",
      "status": "trusted",
      "subject_name": "Microsoft Windows",
      "trusted": "true"
    },
    "Ext": {
      "code_signature": "{\"trusted\":true,\"subject_name\":\"Microsoft Windows\",\"exists\":true,\"status\":\"trusted\"}"
    },
    "hash": {
      "md5": "729a761566b60a8621a1d171baccf41c",
      "sha1": "a9852987463fdd9b05614a10a33d30b1b91f04a6",
      "sha256": "4bf259ee8bc11a51fb6ffc7c5d77b8fab9d092d6892789b92d145083607fb314"
    },
    "name": "psapi.dll",
    "path": "C:\\Windows\\System32\\psapi.dll",
    "pe": {
      "file_version": "10.0.19041.546 (WinBuild.160101.0800)",
      "imphash": "a19426362f5443c7159b76fbeafd171f",
      "original_file_name": "PSAPI"
    }
  },
  "ecs": {
    "version": "1.6.0"
  },
  "elastic": {
    "agent": {
      "id": "34e99100-816e-11eb-8402-556c045a42e6"
    }
  },
  "event": {
    "action": "load",
    "category": "library",
    "created": "2021-03-10T08:55:04.950Z",
    "dataset": "endpoint.events.library",
    "id": "M2Vj3lQihxqU6DxB+++++if5",
    "ingested": "2021-03-10T08:55:15.864Z",
    "kind": "signal",
    "module": "endpoint",
    "sequence": "258123",
    "type": "start"
  },
  "host": {
    "architecture": "x86_64",
    "hostname": "qasource",
    "id": "4143c277-074e-47a9-b37d-37f94b508705",
    "ip": "10.0.5.175,127.0.0.1,::1",
    "mac": "00:50:56:b1:af:f1",
    "name": "qasource",
    "os": {
      "Ext": {
        "variant": "Windows 10 Pro"
      },
      "family": "windows",
      "full": "Windows 10 Pro 2009 (10.0.19042.867)",
      "kernel": "2009 (10.0.19042.867)",
      "name": "Windows",
      "platform": "windows",
      "version": "2009 (10.0.19042.867)"
    }
  },
  "message": "Endpoint DLL load event",
  "process": {
    "entity_id": "ZDM2ODM1NTEtYzA1OS1kZDEyLWM4MmUtMTlmZmE3Zjg3MjQ2LTkxMzYtMTMyNTk4NDAxMDQuODE2NjYzNjAw",
    "executable": "C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-d93154\\install\\metricbeat-7.12.0-windows-x86_64\\metricbeat.exe",
    "Ext": {
      "ancestry": "ZDM2ODM1NTEtYzA1OS1kZDEyLWM4MmUtMTlmZmE3Zjg3MjQ2LTIyMDQtMTMyNTk4MzMxOTMuNTIxOTU3MDA=,ZDM2ODM1NTEtYzA1OS1kZDEyLWM4MmUtMTlmZmE3Zjg3MjQ2LTczMi0xMzI1OTgwMzQzMy4yODAyMTE1MDA=,ZDM2ODM1NTEtYzA1OS1kZDEyLWM4MmUtMTlmZmE3Zjg3MjQ2LTU5Mi0xMzI1OTgwMzQzMi42NjU5MTQ0MDA="
    },
    "name": "metricbeat.exe",
    "pid": "9136"
  },
  "signal": {
    "_meta": {
      "version": "25"
    },
    "ancestors": "{\"id\":\"WEdYG3gBlQYp9M-o7BKj\",\"type\":\"event\",\"index\":\".ds-logs-endpoint.events.library-default-2021.03.10-000001\",\"depth\":0}",
    "depth": "1",
    "original_event": {
      "action": "load",
      "category": "library",
      "created": "2021-03-10T08:55:04.950Z",
      "dataset": "endpoint.events.library",
      "id": "M2Vj3lQihxqU6DxB+++++if5",
      "ingested": "2021-03-10T08:55:15.864626897Z",
      "kind": "event",
      "module": "endpoint",
      "sequence": "258123",
      "type": "start"
    },
    "original_time": "2021-03-10T08:55:04.950Z",
    "parent": {
      "depth": "0",
      "id": "WEdYG3gBlQYp9M-o7BKj",
      "index": ".ds-logs-endpoint.events.library-default-2021.03.10-000001",
      "type": "event"
    },
    "parents": "{\"id\":\"WEdYG3gBlQYp9M-o7BKj\",\"type\":\"event\",\"index\":\".ds-logs-endpoint.events.library-default-2021.03.10-000001\",\"depth\":0}",
    "rule": {
      "actions": "",
      "author": "",
      "created_at": "2021-03-10T08:51:47.442Z",
      "created_by": "elastic",
      "description": "checking unindexed field",
      "enabled": "true",
      "exceptions_list": "",
      "false_positives": "",
      "filters": "",
      "from": "now-360s",
      "id": "d8aa59f0-817d-11eb-8402-556c045a42e6",
      "immutable": "false",
      "index": "apm-*-transaction*,auditbeat-*,endgame-*,filebeat-*,logs-*,packetbeat-*,winlogbeat-*",
      "interval": "1m",
      "language": "kuery",
      "license": "",
      "max_signals": "100",
      "meta": {
        "from": "5m",
        "kibana_siem_app_url": "https://a418827c149f48869e4709b94d216d13.europe-west1.gcp.cloud.es.io:9243/app/security"
      },
      "name": "Ticket Regress #91424",
      "output_index": ".siem-signals-default",
      "query": "host.name : * ",
      "references": "",
      "risk_score": "21",
      "risk_score_mapping": "",
      "rule_id": "ef48427f-6dd6-4cda-9260-8701fd317243",
      "severity": "low",
      "severity_mapping": "",
      "tags": "",
      "threat": "",
      "to": "now",
      "type": "query",
      "updated_at": "2021-03-10T09:00:05.155Z",
      "updated_by": "elastic",
      "version": "1"
    },
    "status": "open"
  },
  "user": {
    "domain": "NT AUTHORITY",
    "id": "S-1-5-18",
    "name": "SYSTEM"
  }
}

thanks !!

[MadameSheema]

Hi @karanbirsingh-qasource I'm not familiar with tat field but I can guide you to create an alert with a non-index field.

1. With dev tools generate the following data:

PUT /index/
PUT /index/_mapping
{
"properties":{
"@timestamp":{
"type":"date"
},
"mydestination":{
"properties":{
"ip":{
"index": "false",
"type":"keyword"
}
}
}
}
}

POST /index/_doc/
{
"@timestamp":"2021-02-22T21:00:49.337Z",
"mydestination":{
"ip": "117.242.211.13"
}
}

2. Create a a custom detection alert with the following parameters:

• Custom query: :
• Runs every: 10s
• Look-back time: 30000h

Once the alert is generated from the alert details view, you should be able to search for the mydestination field. And if you select it, should be displayed on the fimeline.

@FrankHassanabad can you please point us if this way is correct for testing this scenario? Thanks :)

[ghost]

Hi @MadameSheema

thanks for sharing the details to create the test-data for issue regress .

we have validated this issue on 7.12.0 BC4 and found that issue is still occuring . mydestination field value is comming as "-" under the alert list table . However this field value is present under alert detail fly-out.

Build Details:

Version: 7.12.0 BC4
Commit:99ac38d70e426f589bb61a034c96e602d759cfab
Build:39242

Artifact Page: https://staging.elastic.co/7.12.0-336ff10d/summary-7.12.0.html

Snap-Shoot:

@FrankHassanabad please provide your input if we are missing something for this issue regress.

thanks !!

[stephmilovic]

trying to validate and getting expected results against the current 7.12 branch. the bug fix has been in a couple of weeks now, it should be there. Can you send your mappings for the index? thanks

[ghost]

Hi @stephmilovic

thanks for looking into our observation.

we have followed the steps shared by glo in order to create sample data for detection alert having non-index field and we have re-validated the step even there is no newer build than BC4 , but the issue is still occuring on 7.12.0 BC4 at our side even we have dragged that signal to timeline to see there issue occurrence too.

Additionally below is the details of the mapping of index being used to create custom rule.

Mapping

{
  "index" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "mydestination" : {
          "properties" : {
            "ip" : {
              "type" : "keyword",
              "index" : false
            }
          }
        }
      }
    }
  }
}

Steps Followed:

1. Ran these below API hits in Dev Tool

PUT /index/
PUT /index/_mapping
{
"properties":{
"@timestamp":{
"type":"date"
},
"mydestination":{
"properties":{
"ip":{
"index": "false",
"type":"keyword"
}
}
}
}
}

POST /index/_doc/
{
"@timestamp":"2021-02-22T21:00:49.337Z",
"mydestination":{
"ip": "117.242.211.13"
}
}

2. Created custom rule using index as index pattern and * as Query .
3. Got the Alert .However mydestination field value is comming as "-" under the alert list table and in timeline section too.
   
   

Additional Observation:

• Exported Rule:
  rules_export.zip

Please let us known if we are missing something in steps followed.

thanks !!

[stephmilovic]

I reproduced it!! I was just looking at timeline, I needed the rule. Now to debug, I'll update soon. Thanks for your patience 🙏

[stephmilovic]

Ok! The issue is that we have the custom mapping on the index, but not on the .siem-signals-default-* indices where the alert is saved to. By following these steps, we should be able to reindex the field in dev_tools:

1. Get the mappings for the default siem signals index (might be different if defined in kibana.yml)

   GET .siem-signals-default-*/_mapping
   

2. Copy the response
3. Put new mapping in with properties in the copied response. Ex:

   PUT .siem-signals-default-*/_mapping
   {
     "dynamic": "false",
     "_meta": {
       "version": 13
     },
     "properties": {
       "@timestamp": {
         "type": "date"
       },
       "mydestination": {
         "properties": {
           "ip": {
             "index": "false",
             "type": "keyword"
           }
         }
       },
      ...
     }
   }
   

4. Set up an alias to reindex your data, see this blog for more info

   POST /_aliases
   {
       "actions": [
           { "add": {
               "alias": "siem_signals_default",
               "index": ".siem-signals-default-*"
           }}
       ]
   }
   

Now each new .siem-signals-default-* index generated will get reindexed with the custom fields

[ghost]

Hi @stephmilovic

Thanks for looking into our observation and providing updated steps to validate this Issue.

we have followed the steps on 7.12.0 BC4 and Found that issue is Fixed 🟢 . Now Non-Index field ( in our case it is mydestination.ip ) is showing in the Detection Alert List Table View and More details Table and Json View too.

Build Details:

Version: 7.12.0 BC4
Commit: 99ac38d70e426f589bb61a034c96e602d759cfab
Build:39242

Snap-Shoot:

Additional Details:

Custom Rule Exported Zip: rules_export.zip

Hence, we are closing this issue and adding "QA:Validated" label to it.

thanks!!

[ghost]

Bug conversion:

Created 01 new Test-Case for this Ticket under Bug Conversion task:
https://elastic.testrail.io/index.php?/cases/view/76928

[threatangler-jp]

We are seeing this same behavior in v7.16.0. Should it have been resolved in this version?