L0 flush: opt-in mechanism to bypass PageCache reads and writes

[problame]

part of #7418

Motivation

(reproducing #7418)

When we do an InMemoryLayer::write_to_disk, there is a tremendous amount of random read I/O, as deltas from the ephemeral file (written in LSN order) are written out to the delta layer in key order.

In benchmarks (#7409) we can see that this delta layer writing phase is substantially more expensive than the initial ingest of data, and that within the delta layer write a significant amount of the CPU time is spent traversing the page cache.

High-Level Changes

Add a new mode for L0 flush that works as follows:

• Read the full ephemeral file into memory -- layers are much smaller than total memory, so this is afforable
• Do all the random reads directly from this in memory buffer instead of using blob IO/page cache/disk reads.
• Add a semaphore to limit how many timelines may concurrently do this (limit peak memory).
• Make the semaphore configurable via PS config.

Implementation Details

The new BlobReaderRef::Slice is a temporary hack until we can ditch blob_io for InMemoryLayer => Plan for this is laid out in #8183

Correctness

The correctness of this change is quite obvious to me: we do what we did before (blob_io) but read from memory instead of going to disk.

The highest bug potential is in doing owned-buffers IO. I refactored the API a bit in preliminary PR #8186 to make it less error-prone, but still, careful review is requested.

Performance

I manually measured single-client ingest performance from pgbench -i ....

Full report: https://neondatabase.notion.site/2024-06-28-benchmarking-l0-flush-performance-e98cff3807f94cb38f2054d8c818fe84?pvs=4

tl;dr:

• no speed improvements during ingest, but
• significantly lower pressure on PS PageCache (eviction rate drops to 1/3)
  • (that's why I'm working on this)
• noticable but modestly lower CPU time

This is good enough for merging this PR because the changes require opt-in.

We'll do more testing in staging & pre-prod.

Stability / Monitoring

memory consumption: there's no hard limit on max InMemoryLayer size (aka "checkpoint distance") , hence there's no hard limit on the memory allocation we do for flushing. In practice, we a) log a warning when we flush oversized layers, so we'd know which tenant is to blame and b) if we were to put a hard limit in place, we would have to decide what to do if there is an InMemoryLayer that exceeds the limit.
It seems like a better option to guarantee a max size for frozen layer, dependent on checkpoint_distance. Then limit concurrency based on that.

metrics: we do have the flush_time_histo, but that includes the wait time for the semaphore. We could add a separate metric for the time spent after acquiring the semaphore, so one can infer the wait time. Seems unnecessary at this point, though.

[github-actions]

3000 tests run: 2885 passed, 0 failed, 115 skipped (full report)

---

Code coverage* (full report)

• functions: 32.7% (6922 of 21173 functions)
• lines: 50.0% (54264 of 108471 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
c5bc214 at 2024-07-02T11:42:57.331Z :recycle:}

[problame]

the remote_storage changes will hopefully land before this PR in separate PR #8193

[problame]

Did some manual perf testing. Updated PR description, report here: https://www.notion.so/neondatabase/2024-06-28-benchmarking-l0-flush-performance-e98cff3807f94cb38f2054d8c818fe84?pvs=4

[problame]

Stability / Monitoring

TODO

Existing page cache metrics are sufficient to quantify the impact on PageCache.

Do we want metrics on semaphore wait queue length or are higher-level metrics sufficient?

[jcsp]

Do we want metrics on semaphore wait queue length or are higher-level metrics sufficient?

If it's easy, then a queue depth stat is a nice thing to have in our back pocket. Not mandatory though.

[problame]

@jcsp I addressed your review comments, see latest pushes.

Also, given that I plumbed through the l0_flush::L0FlushGlobalState, maybe we want to move the GlobalResourceUnits there, so it no longer is a global lazy static? (Obviously in a follow-up PR)

neon/pageserver/src/tenant/storage_layer/inmemory_layer.rs

Lines 116 to 121 in 511d664

	// Per-timeline RAII struct for its contribution to [`GlobalResources`]
	struct GlobalResourceUnits {
	// How many dirty bytes have I added to the global dirty_bytes: this guard object is responsible
	// for decrementing the global counter by this many bytes when dropped.
	dirty_bytes: u64,
	}