Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kola/kubeadm/tmpl: forward docker logs to journald #228

Merged
merged 2 commits into from
Sep 7, 2021

Conversation

tormath1
Copy link
Contributor

@tormath1 tormath1 commented Sep 7, 2021

this can help debugging failing tests.
See also: https://docs.docker.com/config/containers/logging/journald/

Signed-off-by: Mathieu Tortuyaux mtortuyaux@microsoft.com

For ARM64, tests are failing and only indications we have are from kubelet:

Sep  7 09:22:43.120921 kubelet[1354]: E0907 09:22:43.120852    1354 pod_workers.go:747] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"ebpf-mount\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=ebpf-mount pod=cilium-g84j6_kube-system(aa57d085-b710-4447-b704-550e59266eee)\"" pod="kube-system/cilium-g84j6" podUID=aa57d085-b710-4447-b704-550e59266eee

How to use

  • run a kubeadm.* test
  • check logs into _kola_temp/.../journal.txt:
Sep  7 15:41:10.780613 kernel: IPVS: [rr] scheduler registered.
Sep  7 15:41:10.786475 kernel: IPVS: [wrr] scheduler registered.
Sep  7 15:41:10.793491 kernel: IPVS: [sh] scheduler registered.
Sep  7 15:41:10.822230 6d438354cb89[1103]: I0907 13:41:10.821885       1 node.go:172] Successfully retrieved node IP: 10.0.0.3
Sep  7 15:41:10.822776 6d438354cb89[1103]: I0907 13:41:10.821982       1 server_others.go:140] Detected node IP 10.0.0.3
Sep  7 15:41:10.822930 6d438354cb89[1103]: W0907 13:41:10.822038       1 server_others.go:598] Unknown proxy mode "", assuming iptables proxy
Sep  7 15:41:10.876805 6d438354cb89[1103]: I0907 13:41:10.876354       1 server_others.go:206] kube-proxy running in dual-stack mode, IPv4-primary
Sep  7 15:41:10.878064 6d438354cb89[1103]: I0907 13:41:10.876530       1 server_others.go:212] Using iptables Proxier.
Sep  7 15:41:10.878299 6d438354cb89[1103]: I0907 13:41:10.876593       1 server_others.go:219] creating dualStackProxier for iptables.
...

docker containers logs are now redirected to journald.

@tormath1 tormath1 self-assigned this Sep 7, 2021
@tormath1 tormath1 requested a review from a team September 7, 2021 13:49
@tormath1 tormath1 marked this pull request as ready for review September 7, 2021 13:49
mode: 0644
contents:
remote:
url: data:text/plain;base64,eyJsb2ctZHJpdmVyIjogImpvdXJuYWxkIn0K`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think inline: | should work, too.
Another alternative is data:,… with URL encoding.
The base64 encoding is not readable for humans^^

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I wanted to avoid URL encoding - let's give a try to inline: |.

Comment on lines 104 to 105
remote:
url: data:text/plain;base64,eyJsb2ctZHJpdmVyIjogImpvdXJuYWxkIn0K`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
remote:
url: data:text/plain;base64,eyJsb2ctZHJpdmVyIjogImpvdXJuYWxkIn0K`
inline: |
{"log-driver": "journald"}

@tormath1
Copy link
Contributor Author

tormath1 commented Sep 7, 2021

@pothos comment has been addressed - too bad it's not today we'll start to write config fully in base64 :trollface:

this can help debugging failing tests.
See also: https://docs.docker.com/config/containers/logging/journald/

Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
@tormath1 tormath1 force-pushed the tormath1/docker-logs-container branch from 03aed12 to d14cf13 Compare September 7, 2021 14:31
@tormath1 tormath1 merged commit 0750baa into flatcar-master Sep 7, 2021
@tormath1 tormath1 deleted the tormath1/docker-logs-container branch September 7, 2021 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants