[Auditbeat/FIM/fsnotify]: remove time window where a child file operation of a directory can be lost #39133

pkoutsovasilis · 2024-04-22T15:23:36Z

Proposed commit message

This PR fixes lost events for file integrity module of Auditbeat. Specifically as captured here there is tiny time window where if a child file get's created right after the internal dir contents snapshot of filepath.Walk and before we add the path to the fsnotify watcher, this file is now off the radar.

Mutliple of this tiny windows have been present during some buildkite CI runs (#39110 (comment), #39109 (comment), #39108 (comment))

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

N/A

How to test this PR locally

cd auditbeat && for i in {1..15}; do mage build unitTest; done

Related issues

Closes [Auditbeat/fim/fsnotify]: tiny time window that loses file events #39125

Use cases

N/A

Screenshots

N/A

Logs

N/A

…eration in of a directory can be lost

elasticmachine · 2024-04-22T15:28:07Z

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

elasticmachine · 2024-04-22T16:22:31Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 89 min 14 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…silis/fix_fsnotify_recursive

auditbeat/module/file_integrity/monitor/recursive.go

…silis/fix_fsnotify_recursive

pkoutsovasilis · 2024-04-23T20:23:34Z

run docs-build

Tacklebox

LGTM

Tacklebox · 2024-04-25T07:55:33Z

auditbeat/module/file_integrity/monitor/monitor_test.go

@@ -202,7 +207,7 @@ func TestRecursiveSubdirPermissions(t *testing.T) {
 	for {
 		// No event is received
 		ev, err := readTimeout(t, watcher)
-		if err == errReadTimeout {
+		if errors.Is(err, errReadTimeout) {


I need to spend more time learning go-isms.

…tion of a directory can be lost (#39133) * fix(auditbeat/fim/fsnotify): remove time window where a child file operation in of a directory can be lost * fix(auditbeat/fim/fsnotify): remove redundant error wrapping (cherry picked from commit 074f221) # Conflicts: # auditbeat/module/file_integrity/monitor/recursive.go

…tion of a directory can be lost (#39133) * fix(auditbeat/fim/fsnotify): remove time window where a child file operation in of a directory can be lost * fix(auditbeat/fim/fsnotify): remove redundant error wrapping (cherry picked from commit 074f221)

…where a child file operation of a directory can be lost (#39231) * [Auditbeat/FIM/fsnotify]: remove time window where a child file operation of a directory can be lost (#39133) * fix(auditbeat/fim/fsnotify): remove time window where a child file operation in of a directory can be lost * fix(auditbeat/fim/fsnotify): remove redundant error wrapping (cherry picked from commit 074f221) # Conflicts: # auditbeat/module/file_integrity/monitor/recursive.go * fix: resolve conflicts --------- Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

…tion of a directory can be lost (#39133) (#39233) * fix(auditbeat/fim/fsnotify): remove time window where a child file operation in of a directory can be lost * fix(auditbeat/fim/fsnotify): remove redundant error wrapping (cherry picked from commit 074f221) Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

…tion of a directory can be lost (#39133) (#39232) * fix(auditbeat/fim/fsnotify): remove time window where a child file operation in of a directory can be lost * fix(auditbeat/fim/fsnotify): remove redundant error wrapping (cherry picked from commit 074f221) Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

pkoutsovasilis added backport-7.17 Automated backport to the 7.17 branch with mergify bugfix Team:Security-Linux Platform Linux Platform Team in Security Solution backport-v8.13.0 Automated backport with mergify labels Apr 22, 2024

pkoutsovasilis self-assigned this Apr 22, 2024

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 22, 2024

pkoutsovasilis force-pushed the pkoutsovasilis/fix_fsnotify_recursive branch from c7759a6 to b56f1d6 Compare April 22, 2024 15:24

fix(auditbeat/fim/fsnotify): remove time window where a child file op…

ed523b2

…eration in of a directory can be lost

pkoutsovasilis force-pushed the pkoutsovasilis/fix_fsnotify_recursive branch from b56f1d6 to ed523b2 Compare April 22, 2024 15:25

elastic deleted a comment from mergify bot Apr 22, 2024

pkoutsovasilis marked this pull request as ready for review April 22, 2024 15:28

pkoutsovasilis requested a review from a team as a code owner April 22, 2024 15:28

pkoutsovasilis added the backport-v8.14.0 Automated backport with mergify label Apr 22, 2024

Merge remote-tracking branch 'refs/remotes/beats/main' into pkoutsova…

f0836bd

…silis/fix_fsnotify_recursive

intxgo reviewed Apr 23, 2024

View reviewed changes

auditbeat/module/file_integrity/monitor/recursive.go Outdated Show resolved Hide resolved

pkoutsovasilis added 2 commits April 23, 2024 11:06

fix(auditbeat/fim/fsnotify): remove redundant error wrapping

15b0ae3

Merge remote-tracking branch 'refs/remotes/beats/main' into pkoutsova…

4f59b5b

…silis/fix_fsnotify_recursive

Merge branch 'main' into pkoutsovasilis/fix_fsnotify_recursive

c863acb

Tacklebox approved these changes Apr 25, 2024

View reviewed changes

nfritts approved these changes Apr 25, 2024

View reviewed changes

pkoutsovasilis merged commit 074f221 into elastic:main Apr 26, 2024
39 checks passed

mergify bot mentioned this pull request Apr 26, 2024

[7.17](backport #39133) [Auditbeat/FIM/fsnotify]: remove time window where a child file operation of a directory can be lost #39231

Merged

6 tasks

mergify bot mentioned this pull request Apr 26, 2024

[8.13](backport #39133) [Auditbeat/FIM/fsnotify]: remove time window where a child file operation of a directory can be lost #39232

Merged

6 tasks

mergify bot mentioned this pull request Apr 26, 2024

[8.14](backport #39133) [Auditbeat/FIM/fsnotify]: remove time window where a child file operation of a directory can be lost #39233

Merged

6 tasks

dliappis mentioned this pull request Apr 26, 2024

[Auditbeat/fim/fsnotify]: tiny time window that loses file events #39125

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Auditbeat/FIM/fsnotify]: remove time window where a child file operation of a directory can be lost #39133

[Auditbeat/FIM/fsnotify]: remove time window where a child file operation of a directory can be lost #39133

pkoutsovasilis commented Apr 22, 2024 •

edited

Loading

elasticmachine commented Apr 22, 2024

elasticmachine commented Apr 22, 2024 •

edited

Loading

Build stats

pkoutsovasilis commented Apr 23, 2024

Tacklebox left a comment

Tacklebox Apr 25, 2024

[Auditbeat/FIM/fsnotify]: remove time window where a child file operation of a directory can be lost #39133

[Auditbeat/FIM/fsnotify]: remove time window where a child file operation of a directory can be lost #39133

Conversation

pkoutsovasilis commented Apr 22, 2024 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Apr 22, 2024

elasticmachine commented Apr 22, 2024 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

pkoutsovasilis commented Apr 23, 2024

Tacklebox left a comment

Choose a reason for hiding this comment

Tacklebox Apr 25, 2024

Choose a reason for hiding this comment

pkoutsovasilis commented Apr 22, 2024 •

edited

Loading

elasticmachine commented Apr 22, 2024 •

edited

Loading