Log raw events and errors containing events to a separate file #37475

belimawr · 2023-12-20T15:45:38Z

Proposed commit message

This commit introduces a new logger that can be configured through logging.sensitive that can be used to log any message that contains the whole event or could contain any sensitive data

At the moment it is used by multiple outputs to log indexing errors containing the whole event and errors returned by Elasticsearch that can potentially contain the whole event.

Open questions

Expected behaviour in a container environment?

When Beats is running in a Container environment we set the log output to stdout (see the example in our documentation, however the "sensitive logger" will always log to a file in the default log file.

Decision:
In all cases add a new field log.type: sensitive for the sensitive logger and allow it to also log to stderr as well as disabling the logger.

The docker command, note the -e CLI flag:

docker run -d \
  --name=filebeat \
  --user=root \
  --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  --volume="registry:/usr/share/filebeat/data:rw" \
  docker.elastic.co/beats/filebeat:8.11.4 filebeat -e --strict.perms=false \
  -E output.elasticsearch.hosts=["elasticsearch:9200"]

When running the command above, the events log file is created inside the container in /usr/share/filebeat/logs/

To have the sensitive logger logging to stderr, add the following CLI flag to Filebeat: -E logging.sensitive.to_stderr=true -E logging.sensitive.to_files=false

Generated log file example:

root@76af12033efd: /usr/share/filebeatroot@76af12033efd:/usr/share/filebeat# ls -la /usr/share/filebeat/logs/
total 16
drwxrwxr-x 1 root root 4096 Jan 17 08:05 .
drwxr-xr-x 1 root root 4096 Jan 17 07:39 ..
-rw------- 1 root root 2757 Jan 17 08:05 filebeat-sensitive-data-20240117.ndjson
root@76af12033efd: /usr/share/filebeatroot@76af12033efd:/usr/share/filebeat#

Expected behaviour when running under Elastic-Agent?

The same applies when running under Elastic-Agent: the Elastic-Agent runs Beats passing the the CLI flag -E logging.sensitive.to_stderr=true -E logging.sensitive.to_files=false, collects the stderr and stdout from the Beat, wraps every line in a JSON containing some metadata and logs it to the correct log file. The concept of a sensitive log file will be added to the Elastic-Agent.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Update go.mod after [logp] Add typed loggers allowing log entries to go to different outputs elastic-agent-libs#171 is merged
Create PR on Elastic-Agent to handle the sensitive logs
Decide what to do when running in a container environment
Decide what to do for Elastic-Agent
~~Test if the Elastic-Agent diagnostics can collect the new log file(s)~~

How to test this PR locally

Standalone Filebeat

Start Filebeat with the following configuration

filebeat.inputs:
  - type: filestream
    id: filestream-input-id
    enabled: true
    parsers:
      - ndjson:
          target: ""
          overwrite_keys: true
          expand_keys: true
          add_error_key: true
          ignore_decoding_error: false
    paths:
      - /tmp/flog.log

output:
  elasticsearch:
    hosts:
      - localhost:9200
    protocol: https
    username: elastic
    password: changeme
    allow_older_versions: true
    ssl.verification_mode: none

logging:
  level: debug
  events:
    files:
      name: filebeat-events-data # that's the default, change it if you want another name.

Create the log file /tmp/flog.log with the following content:

{"message":"foo bar","int":10,"string":"str"}
{"message":"another message","int":20,"string":"str2"}
{"message":"index failure","int":"not a number","string":10}
{"message":"second index failure","int":"not a number","string":10}
A broken JSON

Raw events should be logged to a different log file, in the same folder as the normal logs, the filename matches the glob filebeat-events-data*.ndjson.

By default the logs go in a logs folder that's created in the same folder you're running Filebeat from, here are the files created when running this branch:

% ls -la logs 
-rw------- 1 tiago tiago  65K Jan 16 16:24 filebeat-20240116.ndjson
-rw------- 1 tiago tiago 2.8K Jan 16 16:22 filebeat-sensitive-data-20240116.ndjson

If you need to run the test again, either add more data to /tmp/flog.log or remove the data folder Filebeat created at star up, this will make Filebeat re-ingest the file.

Under Elastic-Agent

Package the Elastic-Agent from Use a different log file for sensitive data elastic-agent#4129
Replace the Filebeat binary by the binary built from this branch/PR
Create /tmp/flog.log with a few lines, the data is not important
Start the Elastic-Agent with the following configuration (adjust if needed)

outputs:
  default:
    type: elasticsearch
    hosts:
        - http://localhost:9200
    username: elastic
    password: changeme
    preset: balanced

inputs:
  - type: filestream
    id: your-input-id
    streams:
      - id: your-filestream-stream-id
        data_stream:
          dataset: generic
        paths:
          - /tmp/flog.log

agent.monitoring:
  enabled: false
  logs: false
  metrics: false
  pprof.enabled: false
  use_output: default
  http:
      enabled: false

agent.logging.to_stderr: true
agent.logging.metrics.enabled: false

To create ingest failures the easiest way is to close the write index from the datastream, to do that go to Kibana -> Dev Tools

To get the backing index for a datastream:

GET /_data_stream/logs-generic-default

This will return something like:

{
  "data_streams": [
    {
      "name": "logs-generic-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-logs-generic-default-2024.01.22-000001",
          "index_uuid": "0pq-XIYfSjuUQhTxlJKJjQ",
          "prefer_ilm": true,
          "ilm_policy": "logs",
          "managed_by": "Index Lifecycle Management"
        }
      ]
    }
  ]
}

Take note of the index_name .ds-logs-generic-default-2024.01.22-000001.
Close this index:

POST .ds-logs-generic-default-2024.01.22-000001/_close

Add more data to the file /tmp/flog.log
You should see in the logs something like this:

{
  "log.level": "warn",
  "@timestamp": "2024-01-24T11:22:43.762+0100",
  "message": "Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2024, time.January, 24, 11, 22, 32, 671147360, time.Local), Meta:{\"input_id\":\"your-input-id\",\"raw_index\":\"logs-generic-default\",\"stream_id\":\"your-filestream-stream-id\"}, Fields:{\"agent\":{\"ephemeral_id\":\"7ec26fa3-aa6d-4dc9-b49c-4ed359243af7\",\"id\":\"b3f86956-66e9-4b3e-a477-8f21f2778d2b\",\"name\":\"millennium-falcon\",\"type\":\"filebeat\",\"version\":\"8.13.0\"},\"data_stream\":{\"dataset\":\"generic\",\"namespace\":\"default\",\"type\":\"logs\"},\"ecs\":{\"version\":\"8.0.0\"},\"elastic_agent\":{\"id\":\"b3f86956-66e9-4b3e-a477-8f21f2778d2b\",\"snapshot\":true,\"version\":\"8.13.0\"},\"event\":{\"dataset\":\"generic\"},\"host\":{\"architecture\":\"x86_64\",\"containerized\":false,\"hostname\":\"millennium-falcon\",\"id\":\"851f339d77174301b29e417ecb2ec6a8\",\"ip\":[\"42.42.42.42\",\"42.42.42.42\",\"8652:d3d0:3e34:49e8:f7f0:42dd:75e8:7807\",\"42.42.42.42\",\"8652:d3d0:3e34:49e8:f7f0:42dd:75e8:7807\",\"8652:d3d0:3e34:49e8:f7f0:42dd:75e8:7807\",\"42.42.42.42\",\"8652:d3d0:3e34:49e8:f7f0:42dd:75e8:7807\",\"42.42.42.42\",\"mac\":[\"57-A3-02-AD-65-79\"],\"name\":\"millennium-falcon\",\"os\":{\"build\":\"rolling\",\"family\":\"arch\",\"kernel\":\"6.7.0-arch3-1\",\"name\":\"Arch Linux\",\"platform\":\"arch\",\"type\":\"linux\",\"version\":\"\"}},\"input\":{\"type\":\"filestream\"},\"log\":{\"file\":{\"device_id\":\"34\",\"inode\":\"155800\",\"path\":\"/tmp/flog.log\"},\"offset\":446},\"message\":\"123.124.200.160 - - [24/Jan/2024:11:12:39 +0100] \\\"DELETE /unleash/infrastructures HTTP/1.1\\\" 400 16825\"}, Private:(*input_logfile.updateOp)(0xc001477c50), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"index_closed_exception\",\"reason\":\"closed\",\"index_uuid\":\"0pq-XIYfSjuUQhTxlJKJjQ\",\"index\":\".ds-logs-generic-default-2024.01.22-000001\"}, dropping event!",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-default",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-default"
  },
  "log.logger": "elasticsearch",
  "log.origin": {
    "file.line": 461,
    "file.name": "elasticsearch/client.go",
    "function": "github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).bulkCollectPublishFails"
  },
  "log.type": "sensitive",
  "ecs.version": "1.6.0"
}

~~## Related issues~~
~~## Use cases~~
~~## Screenshots~~
~~## Logs~~

mergify · 2023-12-20T15:46:12Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

belimawr · 2023-12-20T15:48:45Z

CI is failing because elastic/elastic-agent-libs#169 has not been merged yet.

elasticmachine · 2023-12-20T15:57:40Z

❕ Build Aborted

Expand to view the summary

Build stats

Start Time: 2023-12-20T15:45:54.571+0000
Duration: 12 min 40 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3
Skipped	0
Total	3

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-20T18:01:25Z

💔 Build Failed

Expand to view the summary

Build stats

Start Time: 2023-12-20T15:52:51.576+0000
Duration: 128 min 20 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28464
Skipped	1994
Total	30458

Steps errors

Expand to view the steps failures

`libbeat-unitTest - mage build unitTest`

Took 12 min 53 sec . View more details here
Description: mage build unitTest

`libbeat-unitTest - mage build unitTest`

Took 9 min 20 sec . View more details here
Description: mage build unitTest

`libbeat-unitTest - mage build unitTest`

Took 9 min 20 sec . View more details here
Description: mage build unitTest

`libbeat-goIntegTest - mage goIntegTest`

Took 15 min 31 sec . View more details here
Description: mage goIntegTest

`libbeat-goIntegTest - mage goIntegTest`

Took 11 min 51 sec . View more details here
Description: mage goIntegTest

`libbeat-goIntegTest - mage goIntegTest`

Took 11 min 53 sec . View more details here
Description: mage goIntegTest

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

Took 2 min 49 sec . View more details here
Description: mage build unitTest

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

Took 1 min 50 sec . View more details here
Description: mage build unitTest

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

Took 1 min 50 sec . View more details here
Description: mage build unitTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-21T10:15:36Z

💔 Build Failed

Expand to view the summary

Build stats

Start Time: 2023-12-21T08:05:10.271+0000
Duration: 130 min 12 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28596
Skipped	1997
Total	30593

Steps errors

Expand to view the steps failures

`libbeat-unitTest - mage build unitTest`

Took 12 min 50 sec . View more details here
Description: mage build unitTest

`libbeat-unitTest - mage build unitTest`

Took 9 min 20 sec . View more details here
Description: mage build unitTest

`libbeat-unitTest - mage build unitTest`

Took 9 min 18 sec . View more details here
Description: mage build unitTest

`libbeat-goIntegTest - mage goIntegTest`

Took 17 min 51 sec . View more details here
Description: mage goIntegTest

`libbeat-goIntegTest - mage goIntegTest`

Took 11 min 2 sec . View more details here
Description: mage goIntegTest

`libbeat-goIntegTest - mage goIntegTest`

Took 11 min 4 sec . View more details here
Description: mage goIntegTest

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

Took 2 min 50 sec . View more details here
Description: mage build unitTest

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

Took 1 min 0 sec . View more details here
Description: mage build unitTest

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

Took 1 min 0 sec . View more details here
Description: mage build unitTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-21T13:06:19Z

💔 Build Failed

Expand to view the summary

Build stats

Start Time: 2023-12-21T10:57:09.356+0000
Duration: 129 min 56 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28729
Skipped	2015
Total	30744

Steps errors

Expand to view the steps failures

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

Took 2 min 36 sec . View more details here
Description: mage build unitTest

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

Took 1 min 52 sec . View more details here
Description: mage build unitTest

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

Took 1 min 53 sec . View more details here
Description: mage build unitTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-21T19:47:21Z

💚 Build Succeeded

Expand to view the summary

Build stats

Start Time: 2023-12-21T16:52:41.894+0000
Duration: 174 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28732
Skipped	2015
Total	30747

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-22T19:22:06Z

❕ Build Aborted

Expand to view the summary

Build stats

Start Time: 2023-12-22T17:12:53.432+0000
Duration: 129 min 57 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28113
Skipped	2010
Total	30123

Steps errors

Expand to view the steps failures

`filebeat-goIntegTest - mage goIntegTest`

Took 5 min 57 sec . View more details here
Description: mage goIntegTest

`metricbeat-pythonIntegTest - mage pythonIntegTest`

Took 31 min 27 sec . View more details here
Description: mage pythonIntegTest

`x-pack/filebeat-windows-2022-windows-2022 - mage build unitTest`

Took 7 min 21 sec . View more details here
Description: mage build unitTest

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-22T20:21:39Z

💚 Build Succeeded

Expand to view the summary

Build stats

Duration: 172 min 52 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2024-01-07T19:59:29Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b log-raw-events-to-file upstream/log-raw-events-to-file
git merge upstream/main
git push upstream log-raw-events-to-file

elasticmachine · 2024-01-15T18:40:20Z

💚 Build Succeeded

Expand to view the summary

Build stats

Duration: 171 min 59 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-16T15:29:51Z

💔 Build Failed

Expand to view the summary

Build stats

Duration: 9 min 12 sec

Pipeline error

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-16T21:23:10Z

💔 Build Failed

Expand to view the summary

Build stats

Start Time: 2024-01-16T19:13:04.374+0000
Duration: 130 min 53 sec

Test stats 🧪

Test	Results
Failed	0
Passed	31481
Skipped	1882
Total	33363

Steps errors

Expand to view the steps failures

`x-pack/filebeat-goIntegTest - mage goIntegTest`

Took 6 min 15 sec . View more details here
Description: mage goIntegTest

`x-pack/filebeat-goIntegTest - mage goIntegTest`

Took 3 min 47 sec . View more details here
Description: mage goIntegTest

`x-pack/filebeat-goIntegTest - mage goIntegTest`

Took 3 min 45 sec . View more details here
Description: mage goIntegTest

`x-pack/filebeat-windows-2022-windows-2022 - mage build unitTest`

Took 8 min 36 sec . View more details here
Description: mage build unitTest

`x-pack/filebeat-windows-2022-windows-2022 - mage build unitTest`

Took 3 min 40 sec . View more details here
Description: mage build unitTest

`x-pack/filebeat-windows-2022-windows-2022 - mage build unitTest`

Took 3 min 31 sec . View more details here
Description: mage build unitTest

`x-pack/filebeat-windows-2016-windows-2016 - mage build unitTest`

Took 7 min 26 sec . View more details here
Description: mage build unitTest

`x-pack/filebeat-windows-2016-windows-2016 - mage build unitTest`

Took 3 min 38 sec . View more details here
Description: mage build unitTest

`x-pack/filebeat-windows-2016-windows-2016 - mage build unitTest`

Took 3 min 36 sec . View more details here
Description: mage build unitTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-17T09:49:41Z

💔 Build Failed

Expand to view the summary

Build stats

Duration: 132 min 5 sec

Pipeline error

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

belimawr · 2024-01-17T10:43:24Z

go.mod

@@ -419,3 +419,5 @@ replace (

 // Exclude this version because the version has an invalid checksum.
 exclude github.com/docker/distribution v2.8.0+incompatible
+
+replace github.com/elastic/elastic-agent-libs => github.com/belimawr/elastic-agent-libs v0.2.9-0.20240116105334-25f61a14ad41


I'll update this once elastic/elastic-agent-libs#171 is merged.

Update documentation and notice file

This commit fixes the lint issues and adds small improvements.

There was some dependency conflict with Python protobuf 3.19.4 and 3.19.5 when running `make update`. To solve the conflict I updated all usages to 3.19.5.

belimawr · 2024-01-26T10:34:41Z

Rebased onto main to solve conflicts

elasticmachine · 2024-01-26T13:50:55Z

💚 Build Succeeded

Expand to view the summary

Build stats

Start Time: 2024-01-26T10:34:55.627+0000
Duration: 196 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28197
Skipped	1865
Total	30062

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2024-02-05T14:30:54Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b log-raw-events-to-file upstream/log-raw-events-to-file
git merge upstream/main
git push upstream log-raw-events-to-file

mergify · 2024-03-04T10:13:13Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

pierrehilbert · 2024-03-07T15:32:12Z

Setting back to draft for now.

belimawr · 2024-04-12T14:51:36Z

Closing as the behaviour has changed and it was easier to start from a new branch. The new PR with the same feature: #38767

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 20, 2023

belimawr self-assigned this Dec 20, 2023

belimawr added the Team:Elastic-Agent Label for the Agent team label Dec 20, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 20, 2023

belimawr mentioned this pull request Dec 20, 2023

[logp] Add typed loggers allowing log entries to go to different outputs elastic/elastic-agent-libs#171

Merged

5 tasks

belimawr force-pushed the log-raw-events-to-file branch from db6216e to 3185361 Compare December 20, 2023 15:52

belimawr force-pushed the log-raw-events-to-file branch from 7efeff1 to b39a910 Compare December 22, 2023 17:29

belimawr force-pushed the log-raw-events-to-file branch from b39a910 to 0a2e9a2 Compare January 15, 2024 15:48

belimawr added skip-ci Skip the build in the CI but linting backport-skip Skip notification from the automated backport with mergify labels Jan 16, 2024

belimawr removed the skip-ci Skip the build in the CI but linting label Jan 16, 2024

belimawr force-pushed the log-raw-events-to-file branch from dcde0e5 to 8cbb4b4 Compare January 17, 2024 07:37

belimawr commented Jan 17, 2024

View reviewed changes

belimawr marked this pull request as ready for review January 17, 2024 11:00

belimawr requested review from a team as code owners January 17, 2024 11:00

belimawr added 16 commits January 26, 2024 11:29

Update documentation and notice

9df5b79

Update documentation and notice file

Add integration test

8f1277d

Add changelog

3dd701d

update documentation

5fda997

fix tests

a558c68

Fix lint issues and small improvements

28eeaf5

This commit fixes the lint issues and adds small improvements.

Fix bug when merging logging configurations

28aecd0

Add diagnostics hook to collect events log files

4a829d9

Improve documentation

6bd6ff1

Update defaults for events logger

3d6447e

add logger.type to events logger

e892d1e

rename eventsLogger to sensitiveLogger

8c29eb3

remove diagnostics hook

8b61197

Update log fields and replace events-data by sensitive-data

e97f588

update protobuf to 3.19.5

f573aca

There was some dependency conflict with Python protobuf 3.19.4 and 3.19.5 when running `make update`. To solve the conflict I updated all usages to 3.19.5.

update default sensitive log file name on configuration files

3fc66fc

belimawr force-pushed the log-raw-events-to-file branch from 24118b3 to 3fc66fc Compare January 26, 2024 10:34

belimawr requested review from a team as code owners January 26, 2024 10:34

pierrehilbert marked this pull request as draft March 7, 2024 15:32

belimawr mentioned this pull request Apr 9, 2024

Log raw events to a separate log file elastic/elastic-agent#4549

Merged

17 tasks

belimawr closed this Apr 12, 2024

Log raw events and errors containing events to a separate file #37475

Log raw events and errors containing events to a separate file #37475

Conversation

belimawr commented Dec 20, 2023 • edited Loading

Proposed commit message

Open questions

Expected behaviour in a container environment?

Expected behaviour when running under Elastic-Agent?

Checklist

Author's Checklist

How to test this PR locally

Standalone Filebeat

Under Elastic-Agent

mergify bot commented Dec 20, 2023

belimawr commented Dec 20, 2023

elasticmachine commented Dec 20, 2023

❕ Build Aborted

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Dec 20, 2023

💔 Build Failed

Build stats

Test stats 🧪

Steps errors

libbeat-unitTest - mage build unitTest

libbeat-unitTest - mage build unitTest

libbeat-unitTest - mage build unitTest

libbeat-goIntegTest - mage goIntegTest

libbeat-goIntegTest - mage goIntegTest

libbeat-goIntegTest - mage goIntegTest

x-pack/dockerlogbeat-unitTest - mage build unitTest

x-pack/dockerlogbeat-unitTest - mage build unitTest

x-pack/dockerlogbeat-unitTest - mage build unitTest

Error signal

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Dec 21, 2023

💔 Build Failed

Build stats

Test stats 🧪

Steps errors

libbeat-unitTest - mage build unitTest

libbeat-unitTest - mage build unitTest

libbeat-unitTest - mage build unitTest

libbeat-goIntegTest - mage goIntegTest

libbeat-goIntegTest - mage goIntegTest

libbeat-goIntegTest - mage goIntegTest

x-pack/dockerlogbeat-unitTest - mage build unitTest

x-pack/dockerlogbeat-unitTest - mage build unitTest

x-pack/dockerlogbeat-unitTest - mage build unitTest

Error signal

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Dec 21, 2023

💔 Build Failed

Build stats

Test stats 🧪

Steps errors

x-pack/dockerlogbeat-unitTest - mage build unitTest

x-pack/dockerlogbeat-unitTest - mage build unitTest

x-pack/dockerlogbeat-unitTest - mage build unitTest

Error signal

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Dec 21, 2023

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Dec 22, 2023

❕ Build Aborted

Build stats

Test stats 🧪

Steps errors

filebeat-goIntegTest - mage goIntegTest

metricbeat-pythonIntegTest - mage pythonIntegTest

x-pack/filebeat-windows-2022-windows-2022 - mage build unitTest

🤖 GitHub comments

belimawr commented Dec 20, 2023 •

edited

Loading

`libbeat-unitTest - mage build unitTest`

`libbeat-unitTest - mage build unitTest`

`libbeat-unitTest - mage build unitTest`

`libbeat-goIntegTest - mage goIntegTest`

`libbeat-goIntegTest - mage goIntegTest`

`libbeat-goIntegTest - mage goIntegTest`

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

`Error signal`

`libbeat-unitTest - mage build unitTest`

`libbeat-unitTest - mage build unitTest`

`libbeat-unitTest - mage build unitTest`

`libbeat-goIntegTest - mage goIntegTest`

`libbeat-goIntegTest - mage goIntegTest`

`libbeat-goIntegTest - mage goIntegTest`

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

`Error signal`

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

`x-pack/dockerlogbeat-unitTest - mage build unitTest`

`Error signal`

`filebeat-goIntegTest - mage goIntegTest`

`metricbeat-pythonIntegTest - mage pythonIntegTest`

`x-pack/filebeat-windows-2022-windows-2022 - mage build unitTest`

`x-pack/filebeat-goIntegTest - mage goIntegTest`

`x-pack/filebeat-goIntegTest - mage goIntegTest`

`x-pack/filebeat-goIntegTest - mage goIntegTest`

`x-pack/filebeat-windows-2022-windows-2022 - mage build unitTest`

`x-pack/filebeat-windows-2022-windows-2022 - mage build unitTest`

`x-pack/filebeat-windows-2022-windows-2022 - mage build unitTest`

`x-pack/filebeat-windows-2016-windows-2016 - mage build unitTest`

`x-pack/filebeat-windows-2016-windows-2016 - mage build unitTest`

`x-pack/filebeat-windows-2016-windows-2016 - mage build unitTest`

`Error signal`