From a6a93f119745e9960d66e45450a7092fb04a953a Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Mon, 8 Mar 2021 13:53:31 -0600 Subject: [PATCH 01/28] initial stage 2 commit --- rfcs/text/0008-threat-intel.md | 129 +++++++++------------------------ 1 file changed, 34 insertions(+), 95 deletions(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index b2c779227e..5ef3d180fa 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -1,7 +1,7 @@ # 0008: Cyber Threat Intelligence Fields -- Stage: **1 (draft)** +- Stage: **2 (draft)** - Date: **2021-02-18** Elastic Security Solution will be adding the capability to ingest, process and utilize threat intelligence information for increasing detection coverage and helping analysts make quicker investigation decisions. Threat intelligence can be collected from a number of sources with a variety of structured and semi-structured data representations. This makes threat intelligence an ideal candidate for ECS mappings. Threat intelligence data will require ECS mappings to normalize it and make it usable in our security solution. This RFC is focused on identifying new field sets and values that need to be created for threat intelligence data. Existing ECS field reuse will be prioritized where possible. If new fields are required we will utilize [STIX Cyber Observable data model](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_mlbmudhl16lr) as guidance. @@ -262,109 +262,47 @@ There are many sources of threat intelligence including open source, closed sour These sources typically provide intelligence that can be downloaded through REST API or in some cases downloadable CSV's or text files. These intelligence sources will update their data repositories at varying intervals. -#### Abuse.ch Feodo Tracker -This dataset from Abuse.ch provides a list of botnet C&C servers associated with the Feodo malware family (Dridex, Emotet). -``` -# Firstseen,DstIP,DstPort,LastOnline,Malware -2020-10-29 19:16:38,181.120.29.49,80,2020-11-02,Heodo -2020-10-29 19:16:35,190.45.24.210,80,2020-11-02,Heodo -2020-10-29 19:16:32,109.242.153.9,80,2020-11-02,Heodo -2020-10-29 19:16:28,169.1.39.242,80,2020-11-02,Heodo -2020-10-29 19:14:24,201.171.244.130,80,2020-11-02,Heodo -2020-10-29 19:14:20,64.207.182.168,8080,2020-11-02,Heodo -2020-10-29 19:14:19,173.173.254.105,80,2020-11-02,Heodo -2020-10-29 19:14:16,153.204.122.254,80,2020-10-30,Heodo -2020-10-29 19:14:13,201.163.74.203,80,2020-11-02,Heodo -``` +- Abuse.ch Malware - This dataset from Abuse.ch provides a list of malware hashes. +- Abuse.ch URL - This dataset from Abuse.ch provides a list of malware URLs. +- AlienVault OTX - This dataset from AlienVault provides a list of malware hashes, URLs, and IPs. +- Anomali Limo - This dataset from Anomali provides threat information from the Limo service. + -#### Botvrij.eu - -Freely available source of indicators which includes Network indicators, File Details, Email and Registry Key - +#### Abuse.ch Malware List +This dataset from Abuse.ch provides a list of malware hashes. ``` -cc2477cf4d596a88b349257cba3ef356 # md5 - AZORult spreads as a fake ProtonVPN installer (191) -573ff02981a5c70ae6b2594b45aa7caa # md5 - AZORult spreads as a fake ProtonVPN installer (191) -c961a3e3bd646ed0732e867310333978 # md5 - AZORult spreads as a fake ProtonVPN installer (191) -2a98e06c3310309c58fb149a8dc7392c # md5 - AZORult spreads as a fake ProtonVPN installer (191) -f21c21c2fceac5118ebf088653275b4f # md5 - AZORult spreads as a fake ProtonVPN installer (191) -0ae37532a7bbce03e7686eee49441c41 # md5 - AZORult spreads as a fake ProtonVPN installer (191) -974b6559a6b45067b465050e5002214b # md5 - AZORult spreads as a fake ProtonVPN installer (191) -7966c2c546b71e800397a67f942858d0 # md5 - This Is Not a Test: APT41 Initiates Global Intrusion Campaign Using Multiple Exploits (194) -5909983db4d9023e4098e56361c96a6f # md5 - This Is Not a Test: APT41 Initiates Global Intrusion Campaign Using Multiple Exploits (194) -3e856162c36b532925c8226b4ed3481c # md5 - This Is Not a Test: APT41 Initiates Global Intrusion Campaign Using Multiple Exploits (194) -659bd19b562059f3f0cc978e15624fd9 # md5 - This Is Not a Test: APT41 Initiates Global Intrusion Campaign Using Multiple Exploits (194) - +{"md5_hash":"7871286a8f1f68a14b18ae475683f724","sha256_hash":"48a6aee18bcfe9058b35b1018832aef1c9efd8f50ac822f49abb484a5e2a4b1f","file_type":"dll","file_size":"277504","signature":null,"firstseen":"2021-01-14 06:14:05","urlhaus_download":"https://urlhaus-api.abuse.ch/v1/download/48a6aee18bcfe9058b35b1018832aef1c9efd8f50ac822f49abb484a5e2a4b1f/","virustotal":null,"imphash":"68aea345b134d576ccdef7f06db86088","ssdeep":"6144:+60EDP6uCLfGw/GpxXinM1BCo1PlumGx2mx2tXd0t115JG5:X5DpBw/KViMTB1MnEWk0115JW","tlsh":"1344D022AD13DD37E1F400FCA6A58F8561626E381F00A89777D41F8A98356F1BB2B717"} +{"md5_hash":"7b4c77dc293347b467fb860e34515163","sha256_hash":"ec59538e8de8525b1674b3b8fe0c180ac822145350bcce054ad3fc6b95b1b5a4","file_type":"dll","file_size":"277504","signature":null,"firstseen":"2021-01-14 06:11:41","urlhaus_download":"https://urlhaus-api.abuse.ch/v1/download/ec59538e8de8525b1674b3b8fe0c180ac822145350bcce054ad3fc6b95b1b5a4/","virustotal":null,"imphash":"68aea345b134d576ccdef7f06db86088","ssdeep":"6144:+60EDP6uCLfGw/GpxXinM1BCo1PlumGx2mx2tXd0t115JGY:X5DpBw/KViMTB1MnEWk0115Jr","tlsh":"4E44D022AD13DD37E1F400FCA6A58F8561626E381F00A89777D41F8A98356F1BB2B717"} +{"md5_hash":"373d34874d7bc89fd4cefa6272ee80bf","sha256_hash":"b0e914d1bbe19433cc9df64ea1ca07fe77f7b150b511b786e46e007941a62bd7","file_type":"dll","file_size":"277504","signature":null,"firstseen":"2021-01-14 06:11:22","urlhaus_download":"https://urlhaus-api.abuse.ch/v1/download/b0e914d1bbe19433cc9df64ea1ca07fe77f7b150b511b786e46e007941a62bd7/","virustotal":{"result":"25 / 66","percent":"37.88","link":"https://www.virustotal.com/gui/file/b0e914d1bbe19433cc9df64ea1ca07fe77f7b150b511b786e46e007941a62bd7/detection/f-b0e914d"},"imphash":"68aea345b134d576ccdef7f06db86088","ssdeep":"6144:+60EDP6uCLfGw/GpxXinM1BCo1PlumGx2mx2tXd0t115JGG:X5DpBw/KViMTB1MnEWk0115Jd","tlsh":"7544D022AD13DD37E1F400FCA6A58F8561626E381F00A89777D41F8A98356F1BB2B717"} ``` -#### AlienVault OTX -Rest Endpoint: `/api/v1/indicators/export` +#### Abuse.ch URL List +This dataset from Abuse.ch provides a list of botnet C&C servers associated with malware. +``` +{"id":"961548","urlhaus_reference":"https://urlhaus.abuse.ch/url/961548/","url":"http://103.72.223.103:34613/Mozi.m","url_status":"online","host":"103.72.223.103","date_added":"2021-01-14 21:19:13 UTC","threat":"malware_download","blacklists":{"spamhaus_dbl":"not listed","surbl":"not listed"},"reporter":"lrz_urlhaus","larted":"false","tags":["elf","Mozi"]} +{"id":"961546","urlhaus_reference":"https://urlhaus.abuse.ch/url/961546/","url":"http://112.30.97.184:44941/Mozi.m","url_status":"online","host":"112.30.97.184","date_added":"2021-01-14 21:19:05 UTC","threat":"malware_download","blacklists":{"spamhaus_dbl":"not listed","surbl":"not listed"},"reporter":"lrz_urlhaus","larted":"false","tags":["elf","Mozi"]} +{"id":"961547","urlhaus_reference":"https://urlhaus.abuse.ch/url/961547/","url":"http://113.110.198.53:37173/Mozi.m","url_status":"online","host":"113.110.198.53","date_added":"2021-01-14 21:19:05 UTC","threat":"malware_download","blacklists":{"spamhaus_dbl":"not listed","surbl":"not listed"},"reporter":"lrz_urlhaus","larted":"false","tags":["elf","Mozi"]} +``` -Schema +#### AlienVault OTX +This dataset from AlienVault provides a list of malware hashes, URLs, and IPs. ``` -{ - "$schema": "http://json-schema.org/draft-04/schema", - "additionalProperties": false, - "required": ["count", "next", "results", "previous"], - "properties": { - "count": {"type": "integer"}, - "next": {"type": ["string", "null"]}, - "results": { - "type": "array", - "items": { - "additionalProperties": false, - "required": ["indicator", "title", "content", "type", "id", "description"], - "properties": { - "indicator": {"type": "string"}, - "title": {"type": ["string", "null"]}, - "content": {"type": ["string", "null"]}, - "type": {"type": "string"}, - "id": {"type": "integer"}, - "description": {"type": ["string", "null"]} - } - } - }, - "previous": {"type": ["string", "null"]} - } -} +{"indicator":"86.104.194.30","description":null,"title":null,"content":"","type":"IPv4","id":1588938} +{"indicator":"90421f8531f963d81cf54245b72cde80","description":"MD5 of a5725af4391d21a232dc6d4ad33d7d915bd190bdac9b1826b73f364dc5c1aa65","title":"Win32:Hoblig-B","content":"","type":"FileHash-MD5","id":9751110} +{"indicator":"ip.anysrc.net","description":null,"title":null,"content":"","type":"hostname","id":16782717} ``` -Example +#### Anomali Limo +This dataset from Anomali provides threat information from the Limo service. ``` -{ - "count": 3, - "next": null, - "results": [ - { - "indicator": "rustybrooks.com", - "description": null, - "title": null, - "content": "", - "type": "domain", - "id": 1 - }, - { - "indicator": "roll20.com", - "description": null, - "title": null, - "content": "", - "type": "domain", - "id": 3 - }, - { - "indicator": "redacted.ch", - "description": null, - "title": null, - "content": "", - "type": "domain", - "id": 6 - } - ], - "previous": null -} +{"created":"2020-01-22T02:58:57.431Z","description":"TS ID: 55241332361; iType: mal_url; State: active; Org: Cloudflare; Source: CyberCrime","id":"indicator--44c85d4f-45ca-4977-b693-c810bbfb7a28","labels":["malicious-activity","threatstream-severity-medium","threatstream-confidence-76"],"modified":"2020-01-22T02:58:57.431Z","name":"mal_url: http://chol.cc/Work6/PvqDq929BSx_A_D_M1n_a.php","object_marking_refs":["marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da"],"pattern":"[url:value = 'http://chol.cc/Work6/PvqDq929BSx_A_D_M1n_a.php']","type":"indicator","valid_from":"2020-01-22T02:58:57.431Z"} +{"created":"2020-01-22T02:58:57.503Z","description":"TS ID: 55241332307; iType: mal_url; State: active; Org: ServerMania; Source: CyberCrime","id":"indicator--f9fe5c81-6869-4247-af81-62b7c8aba209","labels":["malicious-activity","threatstream-severity-medium","threatstream-confidence-68"],"modified":"2020-01-22T02:58:57.503Z","name":"mal_url: http://worldatdoor.in/lewis/Panel/five/PvqDq929BSx_A_D_M1n_a.php","object_marking_refs":["marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da"],"pattern":"[url:value = 'http://worldatdoor.in/lewis/Panel/five/PvqDq929BSx_A_D_M1n_a.php']","type":"indicator","valid_from":"2020-01-22T02:58:57.503Z"} +{"created":"2020-01-22T02:58:57.57Z","description":"TS ID: 55241332302; iType: mal_url; State: active; Org: SPRINTHOST.RU - shared/premium hosting, VDS, dedic; Source: CyberCrime","id":"indicator--b0e14122-9005-4776-99fc-00872476c6d1","labels":["malicious-activity","threatstream-severity-medium","threatstream-confidence-71"],"modified":"2020-01-22T02:58:57.57Z","name":"mal_url: http://f0387770.xsph.ru/login","object_marking_refs":["marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da"],"pattern":"[url:value = 'http://f0387770.xsph.ru/login']","type":"indicator","valid_from":"2020-01-22T02:58:57.57Z"} ``` + @@ -378,7 +316,7 @@ Stage 2: Identifies scope of impact of changes. Are breaking changes required? S * ECS project (e.g. docs, tooling) The goal here is to research and understand the impact of these changes on users in the community and development teams across Elastic. 2-5 sentences each. --> - * Ingestion mechanism: Primary ingestion mechanisms will be Filebeat modules and Ingest Packages. There will be no impact on ingestion mechanisms. + * Ingestion mechanism: Primary ingestion mechanisms will be Filebeat modules and Ingest Packages. There will be no impact on ingestion mechanisms. [Filebeat module](https://www.elastic.co/guide/en/beats/filebeat/7.12/exported-fields-threatintel.html) is scheduled to be released in `7.12`. * Usage mechanism: The primary use of the proposed ECS fields and values is through Elastic Security solution. In 7.10 we released Indicator match rule to support the use of the proposed new fields and values. ## Concerns @@ -431,10 +369,10 @@ The following are the people that consulted on the contents of this RFC. * [STIX Cyber Observable data model](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_mlbmudhl16lr) Some examples of open source intelligence are: - * [Abuse.ch Feodo Tracker](https://feodotracker.abuse.ch/downloads/ipblocklist.csv) - see below for sample data - * [Botvrij](https://botvrij.eu/data/) - * [Phish Tank](https://www.phishtank.com/) + * [Abuse.ch Malware Tracker](https://feodotracker.abuse.ch/) + * [Abuse.ch URL Tracker](https://urlhaus.abuse.ch/) * [AlienVault OTX](https://otx.alienvault.com/api) + * [Anomali Limo](https://www.anomali.com/resources/limo) Some examples of commercial intelligence include: * [Anomali ThreatStream](https://www.anomali.com/products/threatstream) @@ -451,6 +389,7 @@ Some examples of commercial intelligence include: * Stage 1: https://github.com/elastic/ecs/pull/1037 * Stage 1 correction: https://github.com/elastic/ecs/pull/1100 * Stage 1 (originally stage 2 prior to removal of RFC stage 4): https://github.com/elastic/ecs/pull/1127 +* Stage 2: -- Stage: **2 (draft)** +- Stage: **2 (candidate)** - Date: **2021-02-18** Elastic Security Solution will be adding the capability to ingest, process and utilize threat intelligence information for increasing detection coverage and helping analysts make quicker investigation decisions. Threat intelligence can be collected from a number of sources with a variety of structured and semi-structured data representations. This makes threat intelligence an ideal candidate for ECS mappings. Threat intelligence data will require ECS mappings to normalize it and make it usable in our security solution. This RFC is focused on identifying new field sets and values that need to be created for threat intelligence data. Existing ECS field reuse will be prioritized where possible. If new fields are required we will utilize [STIX Cyber Observable data model](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_mlbmudhl16lr) as guidance. From 6edd79fb822eec250f812fe267ce16009a7aebf6 Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Mon, 15 Mar 2021 14:48:04 -0500 Subject: [PATCH 04/28] changed indicator.description to keyword --- rfcs/text/0008-threat-intel.md | 2 +- rfcs/text/0008/threat.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index c6539ec2de..014208a5e7 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -29,7 +29,7 @@ threat.indicator.first_seen | date | 2020-12-01 | The date and time when intelli threat.indicator.last_seen | date | 2020-12-02| The date and time when intelligence source last reported sighting this indicator. threat.indicator.sightings | long | 20 | Number of times this indicator was observed conducting threat activity threat.indicator.type | keyword | ipv4-addr, domain-name, email-addr | Type of indicator as represented by Cyber Observable in STIX 2.0 -threat.indicator.description | wildcard | 201.10.10.90 was seen delivering Angler EK | Describes the type of action conducted by the threat +threat.indicator.description | keyword | 201.10.10.90 was seen delivering Angler EK | Describes the type of action conducted by the threat threat.indicator.dataset | keyword | theatintel | Identifies the name of specific dataset from the intelligence source. threat.indicator.module | keyword | threatintel.{abusemalware,abuseurl,misp,otx,limo} | Identifies the name of specific module where the data is coming from. threat.indicator.provider | keyword | Abuse.ch | Identifies the name of intelligence provider. diff --git a/rfcs/text/0008/threat.yml b/rfcs/text/0008/threat.yml index 5c55961a07..f606f9dac2 100644 --- a/rfcs/text/0008/threat.yml +++ b/rfcs/text/0008/threat.yml @@ -59,7 +59,7 @@ - name: indicator.description level: extended - type: wildcard + type: keyword short: Indicator description description: > Describes the type of action conducted by the threat. From 5117b8e8daa7df49e34a10b8565042657a281cf8 Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Mon, 15 Mar 2021 14:49:01 -0500 Subject: [PATCH 05/28] typo for t.i.dataset --- rfcs/text/0008-threat-intel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index 014208a5e7..021ddb96d9 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -30,7 +30,7 @@ threat.indicator.last_seen | date | 2020-12-02| The date and time when intellige threat.indicator.sightings | long | 20 | Number of times this indicator was observed conducting threat activity threat.indicator.type | keyword | ipv4-addr, domain-name, email-addr | Type of indicator as represented by Cyber Observable in STIX 2.0 threat.indicator.description | keyword | 201.10.10.90 was seen delivering Angler EK | Describes the type of action conducted by the threat -threat.indicator.dataset | keyword | theatintel | Identifies the name of specific dataset from the intelligence source. +threat.indicator.dataset | keyword | threatintel | Identifies the name of specific dataset from the intelligence source. threat.indicator.module | keyword | threatintel.{abusemalware,abuseurl,misp,otx,limo} | Identifies the name of specific module where the data is coming from. threat.indicator.provider | keyword | Abuse.ch | Identifies the name of intelligence provider. threat.indicator.confidence | keyword | High, 10, Confirmed by other sources, Certain, Almost Certain / Nearly Certain | Identifies the confidence rating assigned by the provider using STIX confidence scales (N/H/M/L, 0-10, Admirality, WEP, or DNI). From 1f4c9dc7272a29f1ca4e83ed88848b68c117f2f6 Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Mon, 15 Mar 2021 14:56:04 -0500 Subject: [PATCH 06/28] updated tlp examples to match --- rfcs/text/0008-threat-intel.md | 6 +++--- rfcs/text/0008/threat.yml | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index 021ddb96d9..fbc8ae6e42 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -30,15 +30,15 @@ threat.indicator.last_seen | date | 2020-12-02| The date and time when intellige threat.indicator.sightings | long | 20 | Number of times this indicator was observed conducting threat activity threat.indicator.type | keyword | ipv4-addr, domain-name, email-addr | Type of indicator as represented by Cyber Observable in STIX 2.0 threat.indicator.description | keyword | 201.10.10.90 was seen delivering Angler EK | Describes the type of action conducted by the threat -threat.indicator.dataset | keyword | threatintel | Identifies the name of specific dataset from the intelligence source. -threat.indicator.module | keyword | threatintel.{abusemalware,abuseurl,misp,otx,limo} | Identifies the name of specific module where the data is coming from. +threat.indicator.dataset | keyword | threatintel.{abusemalware,abuseurl,misp,otx,limo} | Identifies the name of specific dataset from the intelligence source. +threat.indicator.module | keyword | threatintel | Identifies the name of specific module where the data is coming from. threat.indicator.provider | keyword | Abuse.ch | Identifies the name of intelligence provider. threat.indicator.confidence | keyword | High, 10, Confirmed by other sources, Certain, Almost Certain / Nearly Certain | Identifies the confidence rating assigned by the provider using STIX confidence scales (N/H/M/L, 0-10, Admirality, WEP, or DNI). threat.indicator.ip | ip | 1.2.3.4 | Identifies a threat indicator as an IP address (irrespective of direction). threat.indicator.domain | keyword | evil.com | Identifies a threat indicator as a domain (irrespective of direction). threat.indicator.port | long | 443 | Identifies a threat indicator as a port number (irrespective of direction). threat.indicator.email.address | keyword | phish@evil.com | Identifies a threat indicator as an email address (irrespective of direction). -threat.marking.tlp | keyword | RED | Data markings represent restrictions, permissions, and other guidance for how data can be used and shared. Examples could be TLP (White, Green, Amber, Red). +threat.marking.tlp | keyword | RED | Data markings represent restrictions, permissions, and other guidance for how data can be used and shared. Examples could be TLP (WHITE, GREEN, AMBER, RED). threat.indicator.scanner_stats | long | 4 | Count of Anti virus/EDR that successfully detected malicious file or URL. Sources like VirusTotal, Reversing Labs often provide these statistics. threat.indicator.matched.atomic | keyword | 2f5207f2add28b46267dc99bc5382480 | Identifies the atomic indicator that matched a local environment endpoint or network event. threat.indicator.matched.field | keyword | threat.indicator.ip | Identifies the field of the atomic indicator that matched a local environment endpoint or network event. diff --git a/rfcs/text/0008/threat.yml b/rfcs/text/0008/threat.yml index f606f9dac2..0c02915a2b 100644 --- a/rfcs/text/0008/threat.yml +++ b/rfcs/text/0008/threat.yml @@ -161,10 +161,10 @@ Traffic Light Protocol sharing markings. Expected values are: - * White - * Green - * Amber - * Red + * WHITE + * GREEN + * AMBER + * RED example: White From c43f9e0f60ca5ffd18a18cec69bdec27412e780f Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Mon, 15 Mar 2021 14:58:54 -0500 Subject: [PATCH 07/28] updated people --- rfcs/text/0008-threat-intel.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index fbc8ae6e42..07234e9e3d 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -358,6 +358,9 @@ Stage 4: Identify at least one real-world, production-ready implementation that The following are the people that consulted on the contents of this RFC. * @shimonmodi | author +* @dcode | author +* @peasead | author +* @dcode | subject matter expert * @peasead | subject matter expert * @MikePaquette | subject matter expert * @devonakerr | sponsor From 540e64e84fbaad0a116049a72129a92aa57137ba Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Mon, 15 Mar 2021 15:00:01 -0500 Subject: [PATCH 08/28] changed .type to have 1 example --- rfcs/text/0008-threat-intel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index 07234e9e3d..1e187397d6 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -28,7 +28,7 @@ Field | Type | Example | Description threat.indicator.first_seen | date | 2020-12-01 | The date and time when intelligence source first reported sighting this indicator threat.indicator.last_seen | date | 2020-12-02| The date and time when intelligence source last reported sighting this indicator. threat.indicator.sightings | long | 20 | Number of times this indicator was observed conducting threat activity -threat.indicator.type | keyword | ipv4-addr, domain-name, email-addr | Type of indicator as represented by Cyber Observable in STIX 2.0 +threat.indicator.type | keyword | ipv4-addr | Type of indicator as represented by Cyber Observable in STIX 2.0 threat.indicator.description | keyword | 201.10.10.90 was seen delivering Angler EK | Describes the type of action conducted by the threat threat.indicator.dataset | keyword | threatintel.{abusemalware,abuseurl,misp,otx,limo} | Identifies the name of specific dataset from the intelligence source. threat.indicator.module | keyword | threatintel | Identifies the name of specific module where the data is coming from. From b48f6e3216739ac48b9a5ffaeb317bc20c8f5bed Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Thu, 25 Mar 2021 11:47:42 -0500 Subject: [PATCH 09/28] Update rfcs/text/0008/threat.yml Co-authored-by: Eric Beahan --- rfcs/text/0008/threat.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/text/0008/threat.yml b/rfcs/text/0008/threat.yml index 0c02915a2b..199c3abec8 100644 --- a/rfcs/text/0008/threat.yml +++ b/rfcs/text/0008/threat.yml @@ -160,7 +160,7 @@ description: > Traffic Light Protocol sharing markings. - Expected values are: + Recommended values are: * WHITE * GREEN * AMBER From fb057b087c6a9556fbea9a8449cd79792f2dabc8 Mon Sep 17 00:00:00 2001 From: Ryland Herrick Date: Mon, 12 Apr 2021 16:49:42 -0500 Subject: [PATCH 10/28] Add event fieldset under threat.indicator fieldset This is used to preserve the event fields of the original indicator event in the case of said indicator enriching another event. --- rfcs/text/0008/event.yml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 rfcs/text/0008/event.yml diff --git a/rfcs/text/0008/event.yml b/rfcs/text/0008/event.yml new file mode 100644 index 0000000000..7f52d30a6e --- /dev/null +++ b/rfcs/text/0008/event.yml @@ -0,0 +1,5 @@ +--- +- name: event + reusable: + expected: + - threat.indicator From a9d547115412e8867d78ead51d46be1a2b7f6b5a Mon Sep 17 00:00:00 2001 From: Ryland Herrick Date: Fri, 23 Apr 2021 17:07:43 -0500 Subject: [PATCH 11/28] Remove threat enrichment proposal/documentation This is going to become a separate RFC that proposes this use case under a slightly different schema: a nested list of objects conforming to the indicator fieldset. --- rfcs/text/0008-threat-intel.md | 113 +-------------------------------- 1 file changed, 2 insertions(+), 111 deletions(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index fba0c5c234..3583e85a41 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -89,9 +89,9 @@ Stage 1: Describe at a high-level how these field changes will be used in practi The additions described above will be used to enable cyber threat intelligence capabilities in Elastic Security solution. A new rule type Indicator match will be introduced in 7.10 and the proposed ECS updates will enable a new category of detection alerts that match incoming log and event data against threat intelligence sources. Additionally in the future we will also develop enrichment flows that add context from threat intelligence to alerts and events to assist analysts in their investigative workflows. -There are two primary uses for these fields. +While there are two primary uses for these fields, this RFC deals primarily with the first: ingestion/storage of threat intelligence. -1. **Storing threat intelligence as an event document in threat index(s).** +**Storing threat intelligence as an event document in threat index(s).** Threat intelligence data will be collected from multiple sources stored in threat indices. The ECS fields proposed here will be used to structure the documents collected from various sources. @@ -143,115 +143,6 @@ There are two primary uses for these fields. } ``` -2. **Adding threat intelligence match/enrichment to another document which could be in a source event index or signals index.** - - The Indicator Match Rule will be used to generate signals when a match occurs between a source event and threat intelligence document. The ECS fields proposed here will be used to add the enrichment and threat intel context in the signal document. - -**Example** -```json5 -{ - "process": { - "name": "svchost.exe", - "pid": 1644, - "entity_id": "MDgyOWFiYTYtMzRkYi1kZTM2LTFkNDItMzBlYWM3NDVlOTgwLTE2NDQtMTMyNDk3MTA2OTcuNDc1OTExNTAw", - "executable": "C:\\Windows\\System32\\svchost.exe" - }, - "message": "Endpoint file event", - "@timestamp": "2020-11-17T19:07:46.0956672Z", - "file": { - "path": "C:\\Windows\\Prefetch\\SVCHOST.EXE-AE7DB802.pf", - "extension": "pf", - "name": "SVCHOST.EXE-AE7DB802.pf", - "hash": { - "sha256": "0c415dd718e3b3728707d579cf8214f54c2942e964975a5f925e0b82fea644b4" - } - }, - "threat": { - "indicator": [ - { - // Each enrichment is added as a nested object under `threat.indicator.*` - // Copy all the object indicators under `indicator.*`, providing full context - "file": { - "hash": { - "sha256": "0c415dd718e3b3728707d579cf8214f54c2942e964975a5f925e0b82fea644b4", - "md5": "1eee2bf3f56d8abed72da2bc523e7431" - }, - "size": 656896, - "name": "invoice.doc" - }, - /* `matched` will provide context about which of the indicators above matched on this - particular enrichment. If multiple matches for this indicator object, this could - be a list */ - "matched": "sha256", - "marking": { - "tlp": "WHITE" - }, - "first_seen": "2020-10-01", - "last_seen": "2020-11-01", - "sightings": 4, - "type": ["sha256", "md5", "file_name", "file_size"], - "description": "file last associated with delivering Angler EK", - - // Copy event.* data from source threatintel document - "provider": "Abuse.ch", - "dataset": "threatintel.abusemalware", - "module": "threatintel" - } - ] - }, - // Tag the enriched document to indicate the threat enrichment matched - "tags": [ - "threat-match" - ], - // This should already exist from the original ingest pipeline of the document - "related": { - "hash": [ - "0c415dd718e3b3728707d579cf8214f54c2942e964975a5f925e0b82fea644b4" - ] - } -} -``` - -### Proposed enrichment pipeline mechanics pseudocode - -1. Original document completes its standard pipeline for the given source (i.e. filebeat module pipeline) -2. Original document is sent to "threat lookup" pipeline -3. For each indicator type, we perform the following (a file sha256 for example): - - if exists "file.hash.sha256": - - enrich processor: - "policy_name": "file-sha256-policy", - "field" : "file.hash.sha256", - "target_field": "threat_match", - "max_matches": "1" - - policy file-sha256-policy: - "match": { - "indices": "threat-*", - "match_field": "file.hash.sha256", - "enrich_fields": ["event", "file", "indicator"] - } - - rename: - field: "threat_match.file" - target: "threat_match.indicator.file" - - rename: - field: "threat_match.event.provider" - target: "threat_match.indicator.provider" - - rename: - field: "threat_match.event.dataset" - target: "threat_match.indicator.dataset" - - rename: - field: "threat_match.event.module" - target: "threat_match.indicator.module" - - set: - field: "threat_match.indicator.matched" - value: "sha256" - - append: - field: "threat.indicator" - value: "{{ threat_match.indicator }}" - - remove: - field: "threat_match" - -**NOTE**: There may be some optimization on which enrichments we attempt based upon the event categorization fields. For instance, we know that data that presents the netflow model or "interface" doesn't contain a sha256 hash. Since those categorization fields are lists, if data presented as both netflow and file (for whatever reason), then we'd check both network-related lookups and file-related lookups - ## Source data - * Ingestion mechanism: Primary ingestion mechanisms will be Filebeat modules and Ingest Packages. There will be no impact on ingestion mechanisms. [Filebeat module](https://www.elastic.co/guide/en/beats/filebeat/7.12/exported-fields-threatintel.html) is scheduled to be released in `7.12`. + * Ingestion mechanism: Primary ingestion mechanisms will be Filebeat modules and Ingest Packages. There will be no impact on ingestion mechanisms. [Filebeat module](https://www.elastic.co/guide/en/beats/filebeat/7.12/exported-fields-threatintel.html) was released in `7.12`. * Usage mechanism: The primary use of the proposed ECS fields and values is through Elastic Security solution. In 7.10 we released Indicator match rule to support the use of the proposed new fields and values. ## Concerns From 8a161f4474446c49866551ff796a66b78a602d64 Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Mon, 24 May 2021 13:43:06 -0500 Subject: [PATCH 17/28] Update rfcs/text/0008-threat-intel.md Co-authored-by: Eric Beahan --- rfcs/text/0008-threat-intel.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index a1469b30f3..5b96d2151b 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -248,6 +248,7 @@ The following are the people that consulted on the contents of this RFC. * @shimonmodi | author * @dcode | author * @peasead | author +* @rylnd | author * @dcode | subject matter expert * @peasead | subject matter expert * @MikePaquette | subject matter expert From 5c826ac00c3ad0304a011c1c0f8e9723c249dcd9 Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Mon, 24 May 2021 13:43:19 -0500 Subject: [PATCH 18/28] Update rfcs/text/0008-threat-intel.md Co-authored-by: Eric Beahan --- rfcs/text/0008-threat-intel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index 5b96d2151b..2213aa7ba2 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -84,7 +84,7 @@ Stage 3: Add or update all remaining field definitions. The list should now be e Stage 1: Describe at a high-level how these field changes will be used in practice. Real world examples are encouraged. The goal here is to understand how people would leverage these fields to gain insights or solve problems. ~1-3 paragraphs. --> -The additions described above will be used to enable cyber threat intelligence capabilities in Elastic Security solution. A new rule type Indicator match will be introduced in 7.10 and the proposed ECS updates will enable a new category of detection alerts that match incoming log and event data against threat intelligence sources. Additionally in the future we will also develop enrichment flows that add context from threat intelligence to alerts and events to assist analysts in their investigative workflows. +The additions described above will be used to enable cyber threat intelligence capabilities in Elastic Security solution. A new rule type Indicator match was introduced in 7.10 and the proposed ECS updates will enable a new category of detection alerts that match incoming log and event data against threat intelligence sources. Additionally in the future we will also develop enrichment flows that add context from threat intelligence to alerts and events to assist analysts in their investigative workflows. While there are two primary uses for these fields, this RFC deals primarily with the first: ingestion/storage of threat intelligence. From b24d6d107c2d6ab829073f885913d8d9a299e2e5 Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Fri, 28 May 2021 11:20:29 -0500 Subject: [PATCH 19/28] updated example documents --- rfcs/text/0008-threat-intel.md | 140 +++++++++++++++++++++++++-------- rfcs/text/0008/threat.yml | 13 ++- 2 files changed, 120 insertions(+), 33 deletions(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index 2213aa7ba2..017332d4ce 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -33,6 +33,7 @@ threat.indicator.description | keyword | 201.10.10.90 was seen delivering Angler threat.indicator.dataset | keyword | threatintel.{abusemalware,abuseurl,misp,otx,limo} | Identifies the name of specific dataset from the intelligence source. threat.indicator.module | keyword | threatintel | Identifies the name of specific module where the data is coming from. threat.indicator.provider | keyword | Abuse.ch | Identifies the name of intelligence provider. +threat.indicator.reference | keyword | https://urlhaus.abuse.ch/url/1292596 | Provides a reference for the indicator for additional information. threat.indicator.confidence | keyword | High, 10, Confirmed by other sources, Certain, Almost Certain / Nearly Certain | Identifies the confidence rating assigned by the provider using STIX confidence scales (N/H/M/L, 0-10, Admirality, WEP, or DNI). threat.indicator.ip | ip | 1.2.3.4 | Identifies a threat indicator as an IP address (irrespective of direction). threat.indicator.domain | keyword | evil.com | Identifies a threat indicator as a domain (irrespective of direction). @@ -92,53 +93,128 @@ While there are two primary uses for these fields, this RFC deals primarily with Threat intelligence data will be collected from multiple sources stored in threat indices. The ECS fields proposed here will be used to structure the documents collected from various sources. -**Example** +**Examples** + +Network Example ```json5 { + // Metadata about the indicator event "@timestamp": "2019-08-10T11:09:23.000Z", - "event.kind": "enrichment", - "event.category": "threat", - "event.type": "indicator", - "event.provider": "Abuse.ch", - "event.reference": "https://feodotracker.abuse.ch", - "event.dataset": "threatintel.abusemalware", - "event.module": "threatintel", - - // The top-level file object here allows expressing multiple indicators for a single file object - "file.hash.sha256": "0c415dd718e3b3728707d579cf8214f54c2942e964975a5f925e0b82fea644b4", - "file.hash.md5": "1eee2bf3f56d8abed72da2bc523e7431", - "file.size": 656896, - "file.name": "invoice.doc", - - // The indicator prefix here gives context of the indicators - "indicator.marking.tlp": "WHITE", - "indicator.time_first_seen": "2020-10-01", - "indicator.time_last_seen": "2020-11-01", - "indicator.sightings": "4", - /* It's possible to have multiple related indicators in a given document, - e.g. sha256, sha1, ssdeep, etc. If that's the case this should be an array - of types (i.e. [sha1, sha256, ssdeep]) */ - "indicator.type": ["sha256", "md5", "file_name", "file_size"], - "indicator.description": "file last associated with delivering Angler EK", - - // Filebeats and other fields, not part of ECS proposal - "fileset.name": "abusemalware", - "input.type": "log", - "log.offset": 0, + "event": { + "kind": "enrichment", + "category": "threat", + "type": "indicator", + "reference": "https://urlhaus.abuse.ch", + "severity": 7, + "risk_score": 10, + "original": "2020-10-29 19:16:38" + }, + + // Metadata about the indicator data + "threat.indicator": { + "first_seen": "2020-10-01", + "last_seen": "2020-11-01", + "sightings": "10", + "type": [ + "ipv4-addr", + "port", + "domain-name", + "email-addr" + ], + "description": "Email address, domain, port, and IP address observed using an Angler EK campaign.", + "dataset": "threatintel.abuseurl", + "module": "threatintel", + "provider": "Abuse.ch", + "reference": "https://urlhaus.abuse.ch/url/1292596/", + "confidence": "High", + "ip": "1.2.3.4", + "domain": "malicious.evil", + "port": 443, + "email.address": "phish@malicious.evil", + "marking.tlp": "WHITE", + "scanner_stats": 4 + }, // Any indicators should also be copied to relevant related.* field "related": { "hash": [ "1eee2bf3f56d8abed72da2bc523e7431", "0c415dd718e3b3728707d579cf8214f54c2942e964975a5f925e0b82fea644b4" + ], + "hosts": [ + "nefarious.evil" + ], + "ip": [ + "1.2.3.4" ] }, + + // Tags for context "tags": [ "threatintel", "forwarded" - ], + ] +} + +File Example +```json5 +{ + // Metadata about the indicator event + "@timestamp": "2019-08-10T11:09:23.000Z", + "event": { + "kind": "enrichment", + "category": "threat", + "type": "indicator", + "reference": "https://bazaar.abuse.ch", + "severity": 7, + "risk_score": 10, + "original": "2020-10-29 19:16:38" + }, + + // Metadata about the indicator data + "threat.indicator": { + "first_seen": "2020-10-01", + "last_seen": "2020-11-01", + "sightings": "10", + "type": [ + "file" + ], + "description": "Implant used during an Angler EK campaign.", + "dataset": "threatintel.malwarebazaar", + "module": "threatintel", + "provider": "Abuse.ch", + "reference": "https://bazaar.abuse.ch/sample/f3ec9a2f2766c6bcf8c2894a9927c227649249ac146aabfe8d26b259be7d7055", + "confidence": "High", + "file": { + "hash.sha256": "0c415dd718e3b3728707d579cf8214f54c2942e964975a5f925e0b82fea644b4", + "hash.md5": "1eee2bf3f56d8abed72da2bc523e7431", + "size": 656896, + "name": "invoice.doc" + }, + "marking.tlp": "WHITE", + "scanner_stats": 4 + }, + + // Any indicators should also be copied to relevant related.* field + "related": { + "hash": [ + "1eee2bf3f56d8abed72da2bc523e7431", + "0c415dd718e3b3728707d579cf8214f54c2942e964975a5f925e0b82fea644b4" + ], + "hosts": [ + "nefarious.evil" + ], + "ip": [ + "1.2.3.4" + ] + }, + + // Tags for context + "tags": [ + "threatintel", + "forwarded" + ] } -``` ## Source data diff --git a/rfcs/text/0008/threat.yml b/rfcs/text/0008/threat.yml index 88f63cdb53..76c8cbd5be 100644 --- a/rfcs/text/0008/threat.yml +++ b/rfcs/text/0008/threat.yml @@ -37,7 +37,7 @@ description: > Type of indicator as represented by Cyber Observable in STIX 2.0. - Expected values + Recommended values * autonomous-system * artifact * directory @@ -48,6 +48,7 @@ * ipv6-addr * mac-addr * mutex + * port * process * software * url @@ -83,6 +84,16 @@ example: VirusTotal + + - name: indicator.reference + level: extended + type: keyword + short: Indicator reference + description: > + Provides a reference for the indicator for additional information. + + example: https://urlhaus.abuse.ch/url/1234567 + - name: indicator.confidence level: extended type: keyword From 526219bc6c95ab8a9c762c754946b2a518b25221 Mon Sep 17 00:00:00 2001 From: Eric Beahan Date: Fri, 28 May 2021 13:52:49 -0500 Subject: [PATCH 20/28] fix example formatting --- rfcs/text/0008-threat-intel.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index 017332d4ce..a339c27220 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -155,6 +155,7 @@ Network Example "forwarded" ] } +``` File Example ```json5 @@ -215,6 +216,7 @@ File Example "forwarded" ] } +``` ## Source data From 314a427113bf81a4669abcc199152a2020b0ba93 Mon Sep 17 00:00:00 2001 From: Eric Beahan Date: Fri, 28 May 2021 15:14:23 -0500 Subject: [PATCH 21/28] another formatting fix --- rfcs/text/0008-threat-intel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index a339c27220..3dfe7b62c0 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -91,7 +91,7 @@ While there are two primary uses for these fields, this RFC deals primarily with **Storing threat intelligence as an event document in threat index(s).** - Threat intelligence data will be collected from multiple sources stored in threat indices. The ECS fields proposed here will be used to structure the documents collected from various sources. +Threat intelligence data will be collected from multiple sources stored in threat indices. The ECS fields proposed here will be used to structure the documents collected from various sources. **Examples** From 41a84865c74fe2278b9e721d7b305cffafd829a1 Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Thu, 3 Jun 2021 14:52:15 -0500 Subject: [PATCH 22/28] moved proposed fields to existing event and url fieldsets --- rfcs/text/0008-threat-intel.md | 8 +++--- rfcs/text/0008/threat.yml | 49 +--------------------------------- 2 files changed, 4 insertions(+), 53 deletions(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index 3dfe7b62c0..a900c411fa 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -30,13 +30,8 @@ threat.indicator.last_seen | date | 2020-12-02| The date and time when intellige threat.indicator.sightings | long | 20 | Number of times this indicator was observed conducting threat activity threat.indicator.type | keyword | ipv4-addr | Type of indicator as represented by Cyber Observable in STIX 2.0 threat.indicator.description | keyword | 201.10.10.90 was seen delivering Angler EK | Describes the type of action conducted by the threat -threat.indicator.dataset | keyword | threatintel.{abusemalware,abuseurl,misp,otx,limo} | Identifies the name of specific dataset from the intelligence source. -threat.indicator.module | keyword | threatintel | Identifies the name of specific module where the data is coming from. -threat.indicator.provider | keyword | Abuse.ch | Identifies the name of intelligence provider. -threat.indicator.reference | keyword | https://urlhaus.abuse.ch/url/1292596 | Provides a reference for the indicator for additional information. threat.indicator.confidence | keyword | High, 10, Confirmed by other sources, Certain, Almost Certain / Nearly Certain | Identifies the confidence rating assigned by the provider using STIX confidence scales (N/H/M/L, 0-10, Admirality, WEP, or DNI). threat.indicator.ip | ip | 1.2.3.4 | Identifies a threat indicator as an IP address (irrespective of direction). -threat.indicator.domain | keyword | evil.com | Identifies a threat indicator as a domain (irrespective of direction). threat.indicator.port | long | 443 | Identifies a threat indicator as a port number (irrespective of direction). threat.indicator.email.address | keyword | phish@evil.com | Identifies a threat indicator as an email address (irrespective of direction). threat.marking.tlp | keyword | RED | Data markings represent restrictions, permissions, and other guidance for how data can be used and shared. Examples could be TLP (WHITE, GREEN, AMBER, RED). @@ -57,6 +52,9 @@ event.reference | keyword | https://feodotracker.abuse.ch/ | URL to the intellig event.severity | long | 7 | severity provided by threat intelligence source event.risk_score | float | 10 | risk score provided by threat intelligence source event.original | keyword | 2020-10-29 19:16:38,181.120.29.49,80,2020-11-02,Heodo | raw intelligence event +event.dataset | keyword | threatintel.{abusemalware,abuseurl,misp,otx,limo} | Identifies the name of specific dataset from the intelligence source +event.module | keyword | threatintel | Identifies the name of specific module where the data is coming from. +event.provider | keyword | Abuse.ch | Identifies the name of intelligence provider. ### Using existing ECS Fields to store indicator information diff --git a/rfcs/text/0008/threat.yml b/rfcs/text/0008/threat.yml index 76c8cbd5be..0e5dd6411f 100644 --- a/rfcs/text/0008/threat.yml +++ b/rfcs/text/0008/threat.yml @@ -41,9 +41,7 @@ * autonomous-system * artifact * directory - * domain-name - * email-addr - * file + * domain-na * file * ipv4-addr * ipv6-addr * mac-addr @@ -76,24 +74,6 @@ example: 4 - - name: indicator.provider - level: extended - type: keyword - description: > - Identifies the name of the intelligence provider. - - example: VirusTotal - - - - name: indicator.reference - level: extended - type: keyword - short: Indicator reference - description: > - Provides a reference for the indicator for additional information. - - example: https://urlhaus.abuse.ch/url/1234567 - - name: indicator.confidence level: extended type: keyword @@ -110,24 +90,6 @@ example: High - - name: indicator.module - level: extended - type: keyword - short: Indicator module - description: > - Identifies the name of specific module this data is coming from. - - example: threatintel - - - name: indicator.dataset - level: extended - type: keyword - short: Indicator dataset - description: > - Identifies the name of specific dataset from the intelligence source. - - example: threatintel.abusemalware - - name: indicator.ip level: extended type: ip @@ -137,15 +99,6 @@ example: 1.2.3.4 - - name: indicator.domain - level: extended - type: keyword - short: Indicator domain name - description: > - Identifies a threat indicator as a domain (irrespective of direction). - - example: example.com - - name: indicator.port level: extended type: long From e007771e9afb368597828b2e2f368c9cfd7695d7 Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Tue, 8 Jun 2021 14:37:54 -0500 Subject: [PATCH 23/28] Update threat.yml fixed a formatting issue for indicatory.type --- rfcs/text/0008/threat.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rfcs/text/0008/threat.yml b/rfcs/text/0008/threat.yml index 0e5dd6411f..a25083f238 100644 --- a/rfcs/text/0008/threat.yml +++ b/rfcs/text/0008/threat.yml @@ -41,7 +41,9 @@ * autonomous-system * artifact * directory - * domain-na * file + * domain-name + * email-addr + * file * ipv4-addr * ipv6-addr * mac-addr From f8ed4c27664d7a15b18e240b72223f0802fa6de8 Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Tue, 15 Jun 2021 14:15:13 -0500 Subject: [PATCH 24/28] added modified_at field --- rfcs/text/0008-threat-intel.md | 15 +++++++++------ rfcs/text/0008/threat.yml | 9 +++++++++ 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index a900c411fa..e3f8fe42c4 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -25,8 +25,9 @@ Stage 1: Describe at a high level how this change affects fields. Which fieldset Field | Type | Example | Description --- | --- | --- | --- -threat.indicator.first_seen | date | 2020-12-01 | The date and time when intelligence source first reported sighting this indicator -threat.indicator.last_seen | date | 2020-12-02| The date and time when intelligence source last reported sighting this indicator. +threat.indicator.first_seen | date | 2020-11-05T17:25:47.000Z | The date and time when intelligence source first reported sighting this indicator +threat.indicator.last_seen | date | 2020-11-05T17:25:47.000Z | The date and time when intelligence source last reported sighting this indicator. +threat.indicator.modified_at | date | 2020-11-05T17:25:47.000Z | The date and time when intelligence source last modified information for this indicator. threat.indicator.sightings | long | 20 | Number of times this indicator was observed conducting threat activity threat.indicator.type | keyword | ipv4-addr | Type of indicator as represented by Cyber Observable in STIX 2.0 threat.indicator.description | keyword | 201.10.10.90 was seen delivering Angler EK | Describes the type of action conducted by the threat @@ -110,8 +111,9 @@ Network Example // Metadata about the indicator data "threat.indicator": { - "first_seen": "2020-10-01", - "last_seen": "2020-11-01", + "first_seen": "2020-11-05T17:25:47.000Z", + "last_seen": "2020-11-05T17:25:47.000Z", + "modified_at": "2020-11-05T17:25:47.000Z" "sightings": "10", "type": [ "ipv4-addr", @@ -172,8 +174,9 @@ File Example // Metadata about the indicator data "threat.indicator": { - "first_seen": "2020-10-01", - "last_seen": "2020-11-01", + "first_seen": "2020-11-05T17:25:47.000Z", + "last_seen": "2020-11-05T17:25:47.000Z", + "modified_at": "2020-11-05T17:25:47.000Z" "sightings": "10", "type": [ "file" diff --git a/rfcs/text/0008/threat.yml b/rfcs/text/0008/threat.yml index 0e5dd6411f..3cf97fe0dc 100644 --- a/rfcs/text/0008/threat.yml +++ b/rfcs/text/0008/threat.yml @@ -21,6 +21,15 @@ example: "2020-11-05T17:25:47.000Z" + - name: indicator.updated_at + level: extended + type: date + short: Date/time indicator was last updated. + description: > + The date and time when intelligence source last modified information for this indicator. + + example: "2020-11-05T17:25:47.000Z" + - name: indicator.sightings level: extended type: long From f7dbf750a61ca2688dd7f39b435a35d39d27d0f8 Mon Sep 17 00:00:00 2001 From: Andrew Pease <7442091+peasead@users.noreply.github.com> Date: Tue, 15 Jun 2021 15:58:38 -0500 Subject: [PATCH 25/28] typo --- rfcs/text/0008/threat.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/text/0008/threat.yml b/rfcs/text/0008/threat.yml index fb4b4a2724..26935c6a5f 100644 --- a/rfcs/text/0008/threat.yml +++ b/rfcs/text/0008/threat.yml @@ -21,7 +21,7 @@ example: "2020-11-05T17:25:47.000Z" - - name: indicator.updated_at + - name: indicator.modified_at level: extended type: date short: Date/time indicator was last updated. From 31616fa2fa8b4f6b692da1cf73e4981f10df396b Mon Sep 17 00:00:00 2001 From: Adrian Serrano Date: Wed, 23 Jun 2021 10:52:07 +0200 Subject: [PATCH 26/28] Correct expected indicator.type value for X509 Certificates The documentation for the `indicator.type` field lists `x-509-certificate` as an expected value. However, the correct STIX 2.0 Cyber Observable type name for X509 Certificates is `x509-certificate`. --- rfcs/text/0008/threat.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/text/0008/threat.yml b/rfcs/text/0008/threat.yml index 26935c6a5f..edcc60da98 100644 --- a/rfcs/text/0008/threat.yml +++ b/rfcs/text/0008/threat.yml @@ -63,7 +63,7 @@ * url * user-account * windows-registry-key - * x-509-certificate + * x509-certificate example: ipv4-addr From c94e5f109ce6d758f981aa4a3f85193206e05df4 Mon Sep 17 00:00:00 2001 From: Eric Beahan Date: Wed, 23 Jun 2021 14:17:37 -0500 Subject: [PATCH 27/28] missing colon --- rfcs/text/0008/threat.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/text/0008/threat.yml b/rfcs/text/0008/threat.yml index edcc60da98..327cedbf14 100644 --- a/rfcs/text/0008/threat.yml +++ b/rfcs/text/0008/threat.yml @@ -46,7 +46,7 @@ description: > Type of indicator as represented by Cyber Observable in STIX 2.0. - Recommended values + Recommended values: * autonomous-system * artifact * directory From aa27247d29702037c96a775c29afa9cadd2dd14d Mon Sep 17 00:00:00 2001 From: Eric Beahan Date: Wed, 23 Jun 2021 14:35:57 -0500 Subject: [PATCH 28/28] set advance date --- rfcs/text/0008-threat-intel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/text/0008-threat-intel.md b/rfcs/text/0008-threat-intel.md index e3f8fe42c4..d8be1e6498 100644 --- a/rfcs/text/0008-threat-intel.md +++ b/rfcs/text/0008-threat-intel.md @@ -2,7 +2,7 @@ - Stage: **2 (candidate)** -- Date: **2021-02-18** +- Date: **2021-06-23** Elastic Security Solution will be adding the capability to ingest, process and utilize threat intelligence information for increasing detection coverage and helping analysts make quicker investigation decisions. Threat intelligence can be collected from a number of sources with a variety of structured and semi-structured data representations. This makes threat intelligence an ideal candidate for ECS mappings. Threat intelligence data will require ECS mappings to normalize it and make it usable in our security solution. This RFC is focused on identifying new field sets and values that need to be created for threat intelligence data. Existing ECS field reuse will be prioritized where possible. If new fields are required we will utilize [STIX Cyber Observable data model](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_mlbmudhl16lr) as guidance.