ECS 8.0.1

Tooling and Artifact Changes

Bugfixes

• Pin markupsafe==2.0.1 to resolve ImportError exception. #1804

ECS 8.0.0

What's new in ECS 8.0

We're pleased to announce ECS 8.0.

Thank you to all the ECS contributors who help support the broader Elastic community.

Versioning: 1.x -> 8.0

ECS versioning now aligns with the Elastic platform beginning with 8.0.

ECS didn't follow the same release cadence as the Elastic platform when first introduced. Over time this approach added complexity for our users. For example, users might find themselves asking, "which Elastic version maps to ECS 1.6.0?". By aligning, it's clear what version of ECS maps to which Elastic platform version.

Power in simplicity. 😃

Removed fields

The following fields are removed in ECS 8.0:

Field	Migrate to*	Reference
log.original	event.original	RFC 0017
process.ppid	process.parent.pid	RFC 0022
host.user.* reuse	user.* reuses	user.* field set usage

*Field aliases can help transition existing searches or visualizations depending on these removed fields.

New field data types

ECS 1.x introduced wildcard and match_only_text as beta field types. As of ECS 8.0, these data types are now GA.

The field types selected for ECS provide the best default experience for most users. However, some users may see interoperable data types better fitting for their use cases, and they can read more about options here.

Tooling changes

Elasticsearch generated artifacts

In 1.x, the project maintained sample index templates for two versions of Elasticsearch (6.x, 7.x). In 8.0, ECS now produces two sample template types: composable and legacy.

In composable, each ECS field set has a component template. An example component template, template.json, references each field set component template. These artifacts work with the new index templates introduced in Elasticsearch 7.8.

The legacy template will continue working with the legacy index template API.

Removed features

• Removed the already deprecated --oss flag
• Removed Go code generator to simplify the project's tooling and CI/CD pipeline.

Changelog

Schema Changes

Breaking changes

• Remove host.user.* field reuse. #1439
• Remove deprecation notice on http.request.method. #1443
• Migrate log.origin.file.line from integer to long. #1533
• Remove log.original field. #1580
• Remove process.ppid field. #1596

Added

• Added faas.* field set as beta. #1628, #1755

Improvements

• Wildcard type field migration GA. #1582
• match_only_text type field migration GA. #1584
• Threat indicator fields GA from RFC 0008. #1586

Tooling and Artifact Changes

Breaking Changes

• Removing deprecated --oss from generator #1404
• Removing use-cases directory #1405
• Remove Go code generator. #1567
• Remove template generation for ES6. #1680
• Update folder structure for generated ES artifacts. #1700, #1762
• Updated support for overridable composable settings template. #1737

Improvements

• Align input options for --include and --subset arguments #1519
• Remove remaining Go deps after removing Go code generator. #1585
• Add explicit default_field: true for Beats artifacts. #1633
• Reorganize docs directory structure. #1679
• Added support for analyzer definitions for text fields. #1737

Bugfixes

• Fixed the default_field flag for root fields in Beats generator. #1711

ECS 1.12.2

Tooling and Artifact Changes

Bugfixes

• Add object as fallback for flattened type. #1653

ECS 1.12.1

Schema Changes

Bugfixes

• Updating x509 order to correct nesting. ##1621

ECS 1.12.0

The following RFCs have advanced as a part of this release:

Stage 3 (GA)

• RFC 0018 - extend threat.* field set
• RFC 0001 - wildcard field migration
• RFC 0023 - migrate text to match_only_text type

Stage 2 (beta)

• RFC 0002 - service.environment field

Stage 1 (experimental)

• RFC 0010 - email.* field set
• RFC 0025 - container metric fields

There's also been a couple of new field additions in 1.12: file.fork_name, service.address, process.end, code_signature.digest_algorithm and code_signature.timestamp.

Lastly, a couple tooling and documentation improvements. There now exists support for multi-field type fallback to better support ES 6 types as well as the new match_only_text type. And finally, we updated examples within user to better clarify things.

Changelog

Schema Changes

Bugfixes

• Updating hash order to correct nesting. #1603
• Removing incorrect hash reuses. #1604
• Updating pe order to correct nesting. #1605
• Removing incorrect pe reuses. #1606
• Correcting enrichments to an array type. #1608

Added

• Added file.fork_name field. #1288
• Added service.address field. #1537
• Added service.environment as a beta field. #1541
• Added process.end field. #1544
• Added container metric fields into experimental schema. #1546
• Add code_signature.digest_algorithm and code_signature.timestamp fields. #1557
• Add email.* field set in the experimental fields. #1569

Improvements

• Beta migration on some keyword fields to wildcard. #1517
• Promote threat.software.* and threat.group.* fields to GA. #1540
• Update user.name and user.id examples for clarity. #1566
• Beta migration of text and .text multi-fields to match_only_text. #1532, #1571

Tooling and Artifact Changes

Added

• Support ES 6.x type fallback for match_only_text field types. #1528

Bugfixes

• Prevent failure if no files need to be deleted find | xargs rm. #1588

Improvements

• Document field type family interoperability in FAQ. #1591

ECS 1.11.0

The following RFCs have advanced as part of this release:

Stage 3 (GA)

• RFC 0012 - orchestrator field set

Stage 2 (beta)

• RFC 0008 - Threat indicator fields
• RFC 0015 - elf file fields
• RFC 0018 - Extend the threat.* field set with threat.software.* and threat.group.* fields
• RFC 0021 - Threat enrichment

Stage 1 (experimental)

• RFC 0016 - Target process fields

The event.agent_id_status field is also new in 1.11 to reflect the status of the agent.id verification performed by a receiving system or data pipeline.

Lastly, many tooling and documentation improvements, including the --exclude flag. The --exclude flag adds the ability to remove individual fields from the schema. More detail is available in the usage doc.

Changelog

Schema Changes

Added

• elf.* field set added as beta. #1410
• Remove beta from orchestrator field set. #1417
• Extend threat.* field set beta. #1438
• Added event.agent_id_status field. #1454
• process.target and process.target.parent added to experimental schema. #1467
• Threat indicator fields progress to beta stage. #1471, #1504
• threat.enrichments beta fields. #1478, #1504

Improvements

• Fix ecs GitHub repo link source branch #1393
• Add --exclude flag to Generator to support field removal testing #1411
• Explicitly include user identifiers in relater.user description. #1420
• Improve descriptions for cloud.region and cloud.availability fields. #1452
• Clarify event.kind descriptions for alert and signal. #1548

Deprecated

• Note deprecation of the host.user.* field reuse. #1422
• Note deprecation of log.original superseded by event.original #1469

Tooling and Artifact Changes

Bugfixes

• Remove ignore_above when index: false and doc_values: false. #1483
• Ensure doc_values is carried into Beats artifacts. #1488

Added

• Support match_only_text data type in Go code generator. #1418
• Support for multi-level, self-nestings. #1459
• beta attribute now supported on categorization allowed values. #1511

Improvements

• Swap Location and Field Set columns in Field Reuse table for better readability. #1472, #1476
• Use a bullet points to list field reuses. #1473
• Improve wording in Threat schema #1505

ECS 1.10.0

A handful of new additions from the ECS RFC process are included in this release:

• The host metrics RFC has advanced to Finished status with host metrics fields becoming GA.
• The orchestrator fieldset RFC has advanced to Stage 3, and the fieldset has been released for beta.
• The data_stream fields moved to Stage 2, and are released for beta.
• We are extending the existing `threat.* fields, which are released as experimental.

In addition to RFC proposed changes, ECS 1.10.0 also adds some documentation updates, including the ability to add a short_override to field reuses for a custom description.

Finally, there is now support for flattened and nested types in the Go code generator script.

Changelog

Schema Changes

Added

• Add data_stream fieldset. #1307
• Add orchestrator fieldset as beta fields. #1326
• Extend threat.* experimental fields with proposed changes from RFC 0018. #1344, #1351
• Allow custom descriptions for self-nesting reuses via short_override #1366

Improvements

• Updated descriptions to use Elastic Security #1305
• Host metrics fields from RFC 0005 are now GA. #1319
• Adjustments to the field set "usage" docs #1345
• Adjustments to the sidebar naming convention for usage and examples docs #1354
• Update user.* field reuse descriptions. #1382

Tooling and Artifact Changes

Bugfixes

• Correcting fieldset name capitalization for generated ES template #1323

Improvements

• Support nested types in go code generator. #1254, #1350
• Go code generator now supports the flattened data type. #1302
• Adjustments to use terminology that doesn't have negative connotation. #1315

ECS 1.9.0

Several additions introduced from the ECS RFC process are included in this release:

• The multiple users proposal has advanced to Finished status with user.changes.*, user.effective.*, and user.target.* field reuses becoming GA.
• Host metrics fields are now beta.
• The threat.indicator fields, elf.* fields, pe.* extensions, and data_stream.* fieldset are now in the experimental ECS schema.

A new section has been added to the ECS event categorization documentation. Real-world example events are categorized to demonstrate using the event categorization fields to group and identify similar events from multiple data sources.

In addition to RFC proposed changes, ECS 1.9.0 also adds:

• http.request.id
• cloud.service.name
• hash.ssdeep
• code_signature.team_id and code_signature.signing_id
• Additional fields to the geo.* fieldset: geo.timezone, geo.postal_code, geo.continent_code

Finally, *.mac field descriptions now suggest normalizing MAC address values to the RFC7042 format.

Changelog

Schema Changes

Added

• Added hash.ssdeep. #1169
• Added cloud.service.name. #1204
• Added http.request.id. #1208
• data_stream.* fieldset introduced in experimental schema and artifacts. #1215
• Added geo.timezone, geo.postal_code, and geo.continent_code. #1229
• Added beta host metrics fields. #1248
• Added code_signature.team_id, code_signature.signing_id. #1249
• Extended pe fields added to experimental schema. #1256
• Add elf fieldset to experimental schema. #1261
• Add threat.indicator fields to experimental schema. #1268

Improvements

• Include formatting guidance and examples for MAC address fields. #456
• New section in ECS detailing event categorization fields usage. #1242
• user.changes.*, user.effective.*, and user.target.* field reuses are GA. #1271

Tooling and Artifact Changes

Improvements

• Update Python dependencies #1310, #1318
• Adjustments to use terminology that doesn't have negative connotation. #1315

ECS 1.8.0

In this release, two ECS RFCs are advancing. The multiple users in an event RFC proposed field reuses now appear in the ECS documentation as beta. The host metrics fields are also advancing and are available in the experimental schema and artifacts.

Accompanying the multiple user changes, the user.* fieldset adds ECS' first usage doc. The user usage page contains guidance on categorization, user ids, field reuse, and mapping examples.

The event categorization fields, with the initial set of allowed values, were introduced as beta in ECS 1.4.0. Over the past several ECS released, we've iterated and further fleshed out these fields and values. We're excited to announce that the event categorization fields are now generally available!

In addition to the event categorizations fields becoming GA, two additional event.category allowed values have also been introduced: registry and session.

A new field, os.type, is intended to ease filtering for Windows, Unix, Linux, and macOS events.

Finally, a component template and composable templates (per fieldset) have been added as generated artifacts. The legacy index templates for Elasticsearch 6.x and 7.x are still being maintained. More details covered here.

Changelog

Schema Changes

Bugfixes

• Clean up event.reference description. #1181
• Go code generator fails if scaled_float type is used. #1250

Added

• Added event.category "registry". #1040
• Added event.category "session". #1049
• Added usage documentation for user fields. #1066
• Added user fields at user.effective.*, user.target.* and user.changes.*. #1066
• Added os.type. #1111

Improvements

• Event categorization fields GA. #1067
• Note [ and ] bracket characters may enclose a literal IPv6 address when populating url.domain. #1131
• Reinforce the exclusion of the leading dot from url.extension. #1151

Deprecated

• Deprecated host.user.* fields for removal at the next major. #1066

Tooling and Artifact Changes

Bugfixes

• tracing fields should be at root of Beats fields.ecs.yml artifacts. #1164

Added

• Added the path key when type is alias, to support the alias field type. #877
• Added support for scaled_float's mandatory parameter scaling_factor. #1042
• Added ability for --oss flag to fall back constant_keyword to keyword. #1046
• Added support in the generated Go source go for wildcard, version, and constant_keyword data types. #1050
• Added support for marking fields, field sets, or field reuse as beta in the documentation. #1051
• Added support for constant_keyword's optional parameter value. #1112
• Added component templates for ECS field sets. #1156, #1186, #1191
• Added functionality for merging custom and core multi-fields. #982

Improvements

• Make all fields linkable directly. #1148
• Added a notice highlighting that the tracing fields are not nested under the
  namespace tracing. #1162
• ES 6.x template data types will fallback to supported types. #1171, #1176, #1186
• Add a documentation page discussing the experimental artifacts. #1189

ECS 1.7.0

Experimental Changes

A few months ago, we introduced the RFC process. This process is meant to fully vet big additions or changes to ECS. A key aspect of this process is that proposals advance in stages. Each stage represents the vetting and maturity of the proposal.

We won’t go over the process in detail here, but one of its key aspects is that accepted “stage 2” proposals appear in “experimental” ECS artifacts. They don’t yet appear officially in ECS documentation. Proposals that reach “stage 3” are the ones that will officially appear in ECS documentation.

ECS 1.7 is the first release that includes RFCs that have reached stage 2 / experimental changes. A new directory has therefore been added, where all the usual generated artifacts are published including the experimental changes. This is at experimental/generated.

This release includes experimental changes from two RFCs reaching stage 2:

• Replace the keyword type on many existing ECS fields with the new wildcard type.
• Adding more places where user fields can be nested, in order to capture privilege escalations & demotions as well as IAM. These experimental nestings are user.effective.*, user.target.*, and user.changes.*.

“Normal” Changes

Contrary to the new experimental changes described above, the following changes are reflected in the documentation.

Two new fields are introduced: http.[request|response].mime_type/ and threat.technique.subtechnique.

Both the network.direction and event.category fields add support for additional allowed values.

The ECS generator script adds two new arguments, --oss and --strict. See usage for more details and examples.

Lastly, we have changed the index pattern of the sample Elasticsearch template from ecs-* to try-ecs-* to avoid conflicting with Logstash' template when run in ECS compatibility mode.

Changelog

Schema Changes

Bugfixes

• The protocol allowed value under event.type should not have the expected_event_types defined. #964
• Clarify the definition of file.extension (no dots). #1016

Added

• Added Mime Type fields to HTTP request and response. #944
• Added network directions ingress and egress. #945
• Added threat.technique.subtechnique to capture MITRE ATT&CK® subtechniques. #951
• Added configuration as an allowed event.category. #963
• Added a new directory with experimental artifacts, which includes all changes
  from RFCs that have reached stage 2. #993, #1053, #1115, #1117, #1118

Improvements

• Expanded field set definitions for source.* and destination.*. #967
• Provided better guidance for mapping network events. #969
• Added the field .subdomain under client, destination, server, source
  and url, to match its presence at dns.question.subdomain. #981
• Clarified ambiguity in guidance on how to use x509 fields for connections with
  only one certificate. #1114

Tooling and Artifact Changes

Breaking changes

• Changed the index pattern of the sample Elasticsearch template from ecs-* to
  try-ecs-* to avoid conflicting with Logstash' ecs-logstash-*. #1048

Bugfixes

• Addressed issue where foreign reuses weren't using the user-supplied as value for their destination. #960
• Experimental artifacts failed to install due to event.original index setting. #1053

Added

• Introduced --strict flag to perform stricter schema validation when running the generator script. #937
• Added check under --strict that ensures composite types in example fields are quoted. #966
• Added ignore_above and normalizer support for keyword multi-fields. #971
• Added --oss flag for users who want to generate ECS templates for use on OSS clusters. #991

Improvements

• Field details Jinja2 template components have been consolidated into one template #897
• Add [discrete] marker before each section header in field details. #989
• --ref now loads experimental/schemas based on git ref in addition to schemas. #1063