Binary data in teleport log

[igtsekov]

Description

What happened:
From time to time I am experiencing a very strange issue. Binary data is saved into the teleport logs. It looks like this:

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

and the log is getting HUGE. After restart, the problem is fixed.
What you expected to happen:
The logs to be normal.

Reproduction Steps

1. start teleport process
2. wait for some logs to accumulate
3. truncate the log file (echo > <path/to/log/file>)
4. wait for new logs to be added to the file
5. see the beginning of the file (less <path/to/log/file>)

Server Details

• Teleport version (run teleport version): v7.1.3
• Server OS (e.g. from /etc/os-release): Ubuntu 16.04.7 LTS
• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): AWS
• Additional details: no

Client Details

• Tsh version (tsh version): v7.1.3
• Computer OS (e.g. Linux, macOS, Windows): macOS
• Browser version (for UI-related issues): 94.0.1 (64-bit)
• Installed via (e.g. apt, yum, brew, website download): website download
• Additional details: no

Debug Logs

Please include or attach debug logs, when appropriate. Obfuscate sensitive information!

• Start Teleport with --debug flag (teleport --debug) 🤷
• Run tsh with --debug flag (tsh --debug) 🤷
  I do not have any additional information.

[igtsekov]

After some digging around we found out that when the size of the log file is changed for some reason, for example, rotate it (we are using logrorate) when the next log message is added to the file at the beginning of the file is filled with binary data.

[igtsekov]

fix: #9468

[zmb3]

This has been resolved.