Merge branch 'feature-ecs-1.8' into fortinet-ecs-1.8

CHANGELOG.asciidoc

@@ -3,6 +3,233 @@
 :issue: https://github.com/elastic/beats/issues/
 :pull: https://github.com/elastic/beats/pull/
 
+[[release-notes-7.11.0]]
+=== Beats version 7.11.0
+https://github.com/elastic/beats/compare/v7.10.2...v7.11.0[View commits]
+
+==== Breaking changes
+
+*Affecting all Beats*
+
+- Allow embedding of CAs, Certificate of private keys for anything that support TLS in ouputs and inputs. {pull}21179[21179]
+- Update to ECS 1.7.0. {pull}22571[22571]
+- Add support for SCRAM-SHA-512 and SCRAM-SHA-256 in Kafka output. {pull}12867[12867]
+
+*Auditbeat*
+
+- Use ECS 1.7 ingress/egress network directions instead of inbound/outbound for system/socket. {pull}22991[22991]
+- Use ingress/egress instead of inbound/outbound for ECS 1.7 in auditd module. {pull}23000[23000]
+
+*Filebeat*
+
+- Add fileset to ingest Kibana's ECS audit logs. {pull}22696[22696]
+- Remove `suricata.eve.timestamp` alias field. {issue}10535[10535] {pull}22095[22095]
+- Rename bad ECS field name tracing.trace.id to trace.id in aws elb fileset. {pull}22571[22571]
+- Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. {pull}22975[22975]
+- Rename `network.direction` values in crowdstrike/falcon to `ingress`/`egress`. {pull}23041[23041]
+
+*Heartbeat*
+- Adds negative body match. {pull}20728[20728]
+
+*Metricbeat*
+
+- Change cloud.provider from googlecloud to gcp. {pull}21775[21775]
+- Rename googlecloud module to gcp module. {pull}22246[22246]
+- Use ingress/egress instead of inbound/outbound for system/socket metricset. {pull}22992[22992]
+- Change types of numeric metrics from Kubelet summary api to double so as to cover big numbers. {pull}23335[23335]
+
+*Packetbeat*
+
+- Update how Packetbeat classifies network directionality to bring it in line with ECS 1.7 {pull}22996[22996]
+
+*Winlogbeat*
+
+- Use ECS 1.7 ingress/egress instead of inbound/outbound network.direction in sysmon. {pull}22997[22997]
+
+==== Bugfixes
+
+*Affecting all Beats*
+
+- Fix memory leak and events duplication in docker autodiscover and add_docker_metadata. {pull}21851[21851]
+- Fix duplicated pod events in kubernetes autodiscover for pods with init or ephemeral containers. {pull}22438[22438]
+- Fix FileVersion contained in Windows exe files. {pull}22581[22581]
+- Log debug message if the Kibana dashboard can not be imported from the archive because of the invalid archive directory structure {issue}12211[12211], {pull}13387[13387]
+- Periodic metrics in logs will now report `libbeat.output.events.active` and `beat.memstats.rss` as gauges (rather than counters). {pull}22877[22877]
+- Use PROGRAMDATA environment variable instead of C:\ProgramData for windows install service {pull}22874[22874]
+- Fix reporting of cgroup metrics when running under Docker {pull}22879[22879]
+- Fix typo in config docs {pull}23185[23185]
+- Fix panic due to unhandled DeletedFinalStateUnknown in k8s OnDelete {pull}23419[23419]
+- Fix error loop with runaway CPU use when the Kafka output encounters some connection errors {pull}23484[23484]
+
+*Auditbeat*
+
+- file_integrity: stop monitoring excluded paths {issue}21278[21278] {pull}21282[21282]
+- Note incompatibility of system/socket on ARM. {pull}23381[23381]
+
+*Filebeat*
+
+- Fix Zeek dashboard reference to `zeek.ssl.server.name` field. {pull}21696[21696]
+- Fix network.direction logic in zeek connection fileset. {pull}22967[22967]
+- Fix aws s3 overview dashboard. {pull}23045[23045]
+- Fix bad `network.direction` values in Fortinet/firewall fileset. {pull}23072[23072]
+- Fix Cisco ASA/FTD module's parsing of WebVPN log message 716002. {pull}22966[22966]
+- Add support for organization and custom prefix in AWS/CloudTrail fileset. {issue}23109[23109] {pull}23126[23126]
+- Simplify regex for organization custom prefix in AWS/CloudTrail fileset. {issue}23203[23203] {pull}23204[23204]
+- Fix syslog header parsing in infoblox module. {issue}23272[23272] {pull}23273[23273]
+- Fix concurrent modification exception in Suricata ingest node pipeline. {pull}23534[23534]
+- Fix handling of ModifiedProperties field in Office 365. {pull}23777[23777]
+
+*Heartbeat*
+
+- Fixed missing `tls` fields when connecting to https via proxy. {issue}15797[15797] {pull}22190[22190]
+
+*Metricbeat*
+
+- Change Session ID type from int to string {pull}22359[22359]
+- Fix filesystem types on Windows in filesystem metricset. {pull}22531[22531]
+- Fix failiures caused by custom beat names with more than 15 characters {pull}22550[22550]
+- Update NATS dashboards to leverage connection and route metricsets {pull}22646[22646]
+- Fix rate metrics in Kafka broker metricset by using last minute rate instead of mean rate. {pull}22733[22733]
+- Update config in `windows.yml` file. {issue}23027[23027]{pull}23327[23327]
+- Fix metric grouping for windows/perfmon module {issue}23489[23489] {pull}23505[23505]
+
+*Packetbeat*
+
+- Fix SIP parser logic related to line length check. {pull}23411[23411]
+
+
+*Winlogbeat*
+
+- Protect against accessing an undefined variable in Security module. {pull}22937[22937]
+- Add source.ip validation for event ID 4778 in the Security module. {issue}19627[19627]
+
+==== Added
+
+*Affecting all Beats*
+
+- Add istiod metricset. {pull}21519[21519]
+- Add support for OpenStack SSL metadata APIs in `add_cloud_metadata`. {pull}21590[21590]
+- Add cloud.account.id for GCP into add_cloud_metadata processor. {pull}21776[21776]
+- Add proxy metricset for istio module. {pull}21751[21751]
+- Add kubernetes.node.hostname metadata of Kubernetes node. {pull}22189[22189]
+- Enable always add_resource_metadata for Pods and Services of kubernetes autodiscovery. {pull}22189[22189]
+- Add add_resource_metadata option setting (always enabled) for add_kubernetes_metadata setting. {pull}22189[22189]
+- Add support for ephemeral containers in kubernetes autodiscover and `add_kubernetes_metadata`. {pull}22389[22389] {pull}22439[22439]
+- Added support for wildcard fields and keyword fallback in beats setup commands. {pull}22521[22521]
+- Fix polling node when it is not ready and monitor by hostname {pull}22666[22666]
+- Add `expand_keys` option to `decode_json_fields` processor and `json` input, to recusively de-dot and expand json keys into hierarchical object structures {pull}22849[22849]
+- Update k8s client and release k8s leader lock gracefully {pull}22919[22919]
+- Improve event normalization performance {pull}22974[22974]
+- Add tini as init system in docker images {pull}22137[22137]
+- Added "detect_mime_type" processor for detecting mime types {pull}22940[22940]
+- Added "add_network_direction" processor for determining perimeter-based network direction. {pull}23076[23076]
+- Added new `rate_limit` processor for enforcing rate limits on event throughput. {pull}22883[22883]
+- Allow node/namespace metadata to be disabled on kubernetes metagen and ensure add_kubernetes_metadata honors host {pull}23012[23012]
+- Improve equals check. {pull}22778[22778]
+
+*Auditbeat*
+
+- Add several improvements for auditd module for improved ECS field mapping {pull}22647[22647]
+- Add ECS 1.7 `configuration` categorization in certain events in auditd module. {pull}23000[23000]
+
+*Filebeat*
+
+
+- Adding support for Oracle Database Audit Logs {pull}21991[21991]
+- Add max_number_of_messages config into s3 input. {pull}21993[21993]
+- Add SSL option to checkpoint module {pull}19560[19560]
+- Added support for MySQL Enterprise audit logs. {pull}22273[22273]
+- Rename googlecloud module to gcp module. {pull}22214[22214]
+- Rename awscloudwatch input to aws-cloudwatch. {pull}22228[22228]
+- Rename google-pubsub input to gcp-pubsub. {pull}22213[22213]
+- Copy tag names from MISP data into events. {pull}21664[21664]
+- Added TLS JA3 fingerprint, certificate not_before/not_after, certificate SHA1 hash, and certificate subject fields to Zeek SSL dataset. {pull}21696[21696]
+- Add platform logs in the azure filebeat module. {pull}22371[22371]
+- Added `event.ingested` field to data from the Netflow module. {pull}22412[22412]
+- Improve panw ECS url fields mapping. {pull}22481[22481]
+- Improve Nats filebeat dashboard. {pull}22726[22726]
+- Add support for UNIX datagram sockets in `unix` input. {issues}18632[18632] {pull}22699[22699]
+- Add `http.request.mime_type` for Elasticsearch audit log fileset. {pull}22975[22975]
+- Add new httpjson input features and mark old config ones for deprecation {pull}22320[22320]
+- Add configuration option to set external and internal networks for panw panos fileset {pull}22998[22998]
+- Add `subbdomain` fields for rsa2elk modules. {pull}23035[23035]
+- Add subdomain enrichment for suricata/eve fileset. {pull}23011[23011]
+- Add subdomain enrichment for zeek/dns fileset. {pull}23011[23011]
+- Add `event.category` "configuration" to auditd module events. {pull}23010[23010]
+- Add `event.category` "configuration" to gsuite module events. {pull}23010[23010]
+- Add `event.category` "configuration" to o365 module events. {pull}23010[23010]
+- Add `event.category` "configuration" to zoom module events. {pull}23010[23010]
+- Add `network.direction` to auditd/log fileset. {pull}23041[23041]
+- Add logic for external network.direction in sophos xg fileset {pull}22973[22973]
+- Preserve AWS CloudTrail eventCategory in aws.cloudtrail.event_category. {issue}22776[22776] {pull}22805[22805]
+- Add top_level_domain enrichment for suricata/eve fileset. {pull}23046[23046]
+- Add top_level_domain enrichment for zeek/dns fileset. {pull}23046[23046]
+- Add `observer.egress.zone` and `observer.ingress.zone` for cisco/asa and cisco/ftd filesets. {pull}23068[23068]
+- Allow cisco/asa and cisco/ftd filesets to override network directionality based off of zones. {pull}23068[23068]
+- Allow cef and checkpoint modules to override network directionality based off of zones {pull}23066[23066]
+- Add `network.direction` to netflow/log fileset. {pull}23052[23052]
+- Add the ability to override `network.direction` based on interfaces in Fortinet/firewall fileset. {pull}23072[23072]
+- Add `network.direction` override by specifying `internal_networks` in gcp module. {pull}23081[23081]
+- Migrate microsoft/defender_atp to httpjson v2 config {pull}23017[23017]
+- Migrate microsoft/m365_defender to httpjson v2 config {pull}23018[23018]
+- Migrate okta to httpjson v2 config {pull}23059[23059]
+- Add support for Snyk Vulnerability and Audit API. {pull}22677[22677]
+- Misp improvements: Migration to httpjson v2 config, pagination and deduplication ID {pull}23070[23070]
+- Add Google Workspace module and mark Gsuite module as deprecated {pull}22950[22950]
+- Mark m365 defender, defender atp, okta and google workspace modules as GA {pull}23113[23113]
+- Added `alternative_host` option to google pubsub input {pull}23215[23215]
+
+*Heartbeat*
+
+- Add mime type detection for http responses. {pull}22976[22976]
+
+*Metricbeat*
+
+- Move s3_daily_storage and s3_request metricsets to use cloudwatch input. {pull}21703[21703]
+- Duplicate system.process.cmdline field with process.command_line ECS field name. {pull}22325[22325]
+- Add awsfargate module task_stats metricset to monitor AWS ECS Fargate. {pull}22034[22034]
+- Add connection and route metricsets for nats metricbeat module to collect metrics per connection/route. {pull}22445[22445]
+- Add unit file states to system/service {pull}22557[22557]
+- `kibana` module: `stats` metricset no-longer collects usage-related data. {pull}22732[22732]
+- Add more TCP states to Metricbeat system socket_summary. {pull}14347[14347]
+- Add io.ops in fields exported by system.diskio. {pull}22066[22066]
+- Adjust the Apache status fields in the fleet mode. {pull}22821[22821]
+- Add AWS Fargate overview dashboard. {pull}22941[22941]
+- Add process.state, process.cpu.pct, process.cpu.start_time and process.memory.pct. {pull}22845[22845]
+- Move IIS module to GA and map fields. {issue}22609[22609] {pull}23024[23024]
+- Apache: convert status.total_kbytes to status.total_bytes in fleet mode. {pull}23022[23022]
+- Release MSSQL as GA {pull}23146[23146]
+
+*Packetbeat*
+
+- Add support for overriding the published index on a per-protocol/flow basis. {pull}22134[22134]
+- Change build process for x-pack distribution {pull}21979[21979]
+- Tuned the internal queue size to reduce the chances of events being dropped. {pull}22650[22650]
+- Add support for "http.request.mime_type" and "http.response.mime_type". {pull}22940[22940]
+
+*Winlogbeat*
+
+- Add file.pe and process.pe fields to ProcessCreate & LoadImage events in Sysmon module. {issue}17335[17335] {pull}22217[22217]
+- Add dns.question.subdomain fields for sysmon DNS events. {pull}22999[22999]
+- Add additional event categorization for security and sysmon modules. {pull}22988[22988]
+- Add dns.question.top_level_domain fields for sysmon DNS events. {pull}23046[23046]
+
+*Elastic Log Driver*
+
+- Add new winlogbeat security dashboard {pull}18775[18775]
+
+==== Deprecated
+
+*Filebeat*
+
+- The experimental modules for Citrix Netscaler and Symantec Endpoint Protection have been removed.
+  As we continue to expand our coverage of common security data sources, we may consider supporting
+  Citrix Netscaler and Symantec Endpoint Protection in a future release. {issue}23129[23129] {pull}23130[23130]
+
+==== Known Issue
+
+
+
 [[release-notes-7.10.2]]
 === Beats version 7.10.2
 https://github.com/elastic/beats/compare/v7.10.1\...v7.10.2[View commits]