-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
flatcar 2605.9.0+ with systemd 246 journal logs produced is not readable by older versions #328
Comments
We ran into this issue as well. In our case, we use a fluent-bit daemonset to ship the journal from all kubernetes hosts into splunk. In moving to Flatcar 2605.10.0 we stopped receiving journal logs. We fixed this by making two changes, both of which were necessary:
Update: FYI if you need to update an existing host, you will need to erase the journald log directory before the compression settings will take effect. We'll be opening a ticket against fluent-bit as well here shortly and will link back here. Thanks @mindw for tipping us off to this! |
Fluent Bit issue: fluent/fluent-bit#2998 |
@jcrowthe tried this looks not working, i tried also restarting the server still no logs collected by fluent-bit |
any update on this issue ? |
Hi @surajnarwade, following what it has been said into the issue it seems there are two options:
Seems like we don't have much activity on fluent-bit PR - let us explore the other option ! |
Hey folks, As has been mentioned above, keyed hashes can be deactivated by setting Keyed hashes are a security / integrity feature employed by journald to mitigate hash collision attacks on the journal. Switching off this feature may expose hosts to journal integrity / manipulation issues - which can be critical if you use the journal for e.g. security auditing. Switching off keyed hashes however does seem to be the only available stop-gap solution to keep fluentd log collection / aggregation remaining functional - at least until upstream picks up the PR and updates their log parser. Considering Flatcar's mission statement to be a fully open source, minimal-footprint, secure by default and always up-to-date Linux distribution for running containers at scale I do not feel however that disabling keyed hashes by default (e.g. using the Bottlerocket patch) would be a good service to the wider Flatcar user base. |
Upstream merged the change to support the newer format, I think we can close this as there is nothing to be done anymore (assuming that an upstream release comes with it goes out). |
Description
Impact
This affects all current dockerized systemd log collection methods. Specifically the reference k8s one.
Environment and steps to reproduce
We have a custom setup - but the below should work. The links below should provide a lot better reference.
a. setup a k8s cluster using flatcar https://typhoon.psdn.io/flatcar-linux/aws/
b. install the fluentd elastic addond https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch
a. [ requested the start of a new pod or container ]
b. [ container image downloaded ]
Expected behavior
systemd logs are collected
Additional information
bottlerocket-os/bottlerocket#1200
one reference workaround:
bottlerocket-os/bottlerocket#1206
Even if this ticket isn't handled - at least it hopefully save some people precious time.
The text was updated successfully, but these errors were encountered: