feat(pageserver): support retain_lsn in bottommost gc-compaction

[skyzh]

Problem

part of #8002

Summary of changes

The main thing in this pull request is the new generate_key_retention function. It decides which deltas to retain and generate images for a given key based on its history + retain_lsn + horizon.

On that, we generate a flat single level of delta layers over all deltas included in the compaction. In the future, we can decide whether to split them over the LSN axis as described in the RFC.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3126 tests run: 3005 passed, 0 failed, 121 skipped (full report)

---

Code coverage* (full report)

• functions: 32.6% (7019 of 21509 functions)
• lines: 50.2% (55718 of 111048 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
24fb4d5 at 2024-07-23T19:24:25.817Z :recycle:}

[problame]

Did a first review pass.

[skyzh]

added tests for the core generate_key_retention function and ready for review :)

[skyzh]

added timeline e2e unit tests

[problame]

So, apart from the added tests, changes since my last review are that we now use lowest_retain_lsn instead of gc_cutoff. What was wrong there?

Also, are you sure that always passing lowest_retain_lsn is correct?
Suppose you have the case of a lot of delta for a given key range. So much delta that we need to create multiple deltas with same key range but split up LSN ranges. The current code, afaict, will create multiple deltas with LSN_range.start == lowest_retain_lsn. No?

Possibly this is a pre-existing issue though?

[skyzh]

So, apart from the added tests, changes since my last review are that we now use lowest_retain_lsn instead of gc_cutoff

The lowest image layer is created over lowest_retain_lsn instead of gc_cutoff. gc_cutoff can be considered as a retain_lsn.

The current code, afaict, will create multiple deltas with LSN_range.start == lowest_retain_lsn.

Yes, for now, the bottom-most compaction always produce delta layers of the LSN range from lowest_retain_lsn to the GC horizon. We could further split it over the LSN as an optimization in the future.

[problame]

Yes, for now, the bottom-most compaction always produce delta layers of the LSN range from lowest_retain_lsn to the GC horizon. We could further split it over the LSN as an optimization in the future.

Are we not respecting the target file size yet?

[skyzh]

Are we not respecting the target file size yet?

Yes, not for now, this is tracked in the epic.