Cherry-pick #21911 to 7.10: [Filebeat] Prevent 'log' sending duplicate messages due to renaming #23745
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elastic#21911) Input:Log: Reset TTL of registry state when a file is renamed. In some rare cases the registry state is marked for removal (TTL is set to 0) while the file is only renamed. Log detects the renaming of the file and updates the name of the file. As the file still exists it should also update the TTL of the renamed file. (cherry picked from commit bb50d32)
jsoriano
added
[zube]: In Review
backport
Team:Services
|
Pinging @elastic/integrations-services (Team:Services) |
botelastic
bot
added
needs_team
|
Is the error related to this change? |
I don't think so. This test is skipped as flaky in 7.11/7.x/master. backporting the skip to 7.10 too: #23754 |
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
kvch
approved these changes
Feb 1, 2021
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
elastic#21911) (elastic#23745) Input:Log: Reset TTL of registry state when a file is renamed. In some rare cases the registry state is marked for removal (TTL is set to 0) while the file is only renamed. Log detects the renaming of the file and updates the name of the file. As the file still exists it should also update the TTL of the renamed file. (cherry picked from commit 7d6c6fe) Co-authored-by: William Deurwaarder <william.git@xs4all.nl>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #21911 to 7.10 branch. Original message:
Input:Log: Reset TTL of registry state when a file is renamed. In some
rare cases the registry state is marked for removal (TTL is set to 0)
while the file is only renamed. Log detects the renaming of the file
and updates the name of the file. As the file still exists it should
also update the TTL of the renamed file.
filebeat.yml.txt
I tried to add label 'bug' but have no idea how to do that.
What does this PR do?
This PR prevents in some rare cases that messages are resent by filebeat:input:log. The scenarios are written in the 'why is it important'-section. It copies the TTL of the registry state of a renamed file from newState to oldState as is done with the name.
Why is it important?
In some rare cases a log-file is marked as removed (TTL is set 0) while in fact the file is renamed.
This issue has been noticed in two similar yet different scenarios:
Issue 1
In case renaming of the log-file happens after scanning but before removal-test, and the original file is not recreated yet, the registry state is marked for removal (TTL is set to 0) as the harvester is finished and the file pointed to by the source of the registry state (the original file) does not exist.
Issue 2
In this case the source of the registry state still points to the original file (updating the registry state has been postponed). In case the close-event of the harvester (per example inactive or timeout) happens after scanning but before removal-test, the registry state is marked for removal (TTL is set to 0) as the harvester is finished, and the file pointed by source of the registry state (original file) does not match the registry state itself (as the content points to the already renamed file).
In both cases will the renaming of the file be detected in the next 'run' and the source of the registry state will be updated. Unfortunately is TTL (= 0) not updated. This results in the deletion of the registry state. In the 'run' after that the renamed file will be detected again by filebeat and seen as a new file (no registry state exists) and all messages will be resent.
This can be fixed by (re)setting the TTL of the renamed file (next to the name of the source). Also from a technical point-of-view this makes sense as the file does exists as it is just found in the scan.
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
The easiest way to test this is adjusting the code and adding a sleep (per example 4 seconds) between scan and the removal-test (Run-method of input.go) as that increases the chance that this will happen tremendously. Let filebeat scan a directory where files are constantly created and rotated. In that way I can reproduce this in 100% of the cases. I have attached my configuration file used for this test.
Use cases
In some rare cases prevent filebeat:input:log to resend logs while files are renamed (rotated).