Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix(l0_flush): drops permit before fsync, potential cause for OOMs (#…
…8327) ## Problem Slack thread: https://neondb.slack.com/archives/C033RQ5SPDH/p1720511577862519 We're seeing OOMs in staging on a pageserver that has l0_flush.mode=Direct enabled. There's a strong correlation between jumps in `maxrss_kb` and `pageserver_timeline_ephemeral_bytes`, so, it's quite likely that l0_flush.mode=Direct is the culprit. Notably, the expected max memory usage on that staging server by the l0_flush.mode=Direct is ~2GiB but we're seeing as much as 24GiB max RSS before the OOM kill. One hypothesis is that we're dropping the semaphore permit before all the dirtied pages have been flushed to disk. (The flushing to disk likely happens in the fsync inside the `.finish()` call, because we're using ext4 in data=ordered mode). ## Summary of changes Hold the permit until after we're done with `.finish()`.
- Loading branch information
9bb16c8
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
3129 tests run: 3002 passed, 0 failed, 127 skipped (full report)
Flaky tests (2)
Postgres 14
test_basebackup_with_high_slru_count[github-actions-selfhosted-sequential-10-13-30]
: releasetest_basebackup_with_high_slru_count[github-actions-selfhosted-vectored-10-13-30]
: releaseCode coverage* (full report)
functions
:32.6% (6941 of 21292 functions)
lines
:50.0% (54547 of 109094 lines)
* collected from Rust tests only
9bb16c8 at 2024-07-09T23:05:29.914Z :recycle: