[Filebeat] Prevent 'log' sending duplicate messages due to renaming

[williamd67]

Input:Log: Reset TTL of registry state when a file is renamed. In some
rare cases the registry state is marked for removal (TTL is set to 0)
while the file is only renamed. Log detects the renaming of the file
and updates the name of the file. As the file still exists it should
also update the TTL of the renamed file.

filebeat.yml.txt

I tried to add label 'bug' but have no idea how to do that.

What does this PR do?

This PR prevents in some rare cases that messages are resent by filebeat:input:log. The scenarios are written in the 'why is it important'-section. It copies the TTL of the registry state of a renamed file from newState to oldState as is done with the name.

Why is it important?

In some rare cases a log-file is marked as removed (TTL is set 0) while in fact the file is renamed.

This issue has been noticed in two similar yet different scenarios:

1. harvester is inactive and file is renamed; which means that registry state is marked as 'Finished'
2. harvester was and has been active since the renaming the file; which means that source of registry state still points to the original file name

Issue 1
In case renaming of the log-file happens after scanning but before removal-test, and the original file is not recreated yet, the registry state is marked for removal (TTL is set to 0) as the harvester is finished and the file pointed to by the source of the registry state (the original file) does not exist.

Issue 2
In this case the source of the registry state still points to the original file (updating the registry state has been postponed). In case the close-event of the harvester (per example inactive or timeout) happens after scanning but before removal-test, the registry state is marked for removal (TTL is set to 0) as the harvester is finished, and the file pointed by source of the registry state (original file) does not match the registry state itself (as the content points to the already renamed file).

In both cases will the renaming of the file be detected in the next 'run' and the source of the registry state will be updated. Unfortunately is TTL (= 0) not updated. This results in the deletion of the registry state. In the 'run' after that the renamed file will be detected again by filebeat and seen as a new file (no registry state exists) and all messages will be resent.

This can be fixed by (re)setting the TTL of the renamed file (next to the name of the source). Also from a technical point-of-view this makes sense as the file does exists as it is just found in the scan.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

The easiest way to test this is adjusting the code and adding a sleep (per example 4 seconds) between scan and the removal-test (Run-method of input.go) as that increases the chance that this will happen tremendously. Let filebeat scan a directory where files are constantly created and rotated. In that way I can reproduce this in 100% of the cases. I have attached my configuration file used for this test.

Use cases

In some rare cases prevent filebeat:input:log to resend logs while files are renamed (rotated).

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[cla-checker-service]

💚 CLA has been signed

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: [kvch commented: jenkins run tests]

• Start Time: 2020-10-21T10:28:40.991+0000

• Duration: 63 min 17 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4474
Skipped	561
Total	5035

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[kvch]

jenkins run tests