-
Notifications
You must be signed in to change notification settings - Fork 396
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pageserver: use PITR GC cutoffs as authoritative #8365
Conversation
3079 tests run: 2964 passed, 0 failed, 115 skipped (full report)Flaky tests (3)Postgres 14Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
5b487b1 at 2024-07-15T13:06:22.884Z :recycle: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the function comment need updating?
Not this PR's fault, but the terminology with the time- and size-based cutoffs are really confusing. "pitr horizon", "cutoff", "retention period" all sound like synonyms to me. Some thoughts:
- How about renaming renaming the
cutoff_horizon
andpitr
variables towal_size_cutoff
andtime_cutoff
or something similar? - Does this function (
find_gc_cutoffs') really need to return both cutoff values? When perfoming GC, you really only need one Lsn cutoff.
Timeline::gc_timeline` keeps statistics of which cutoff value is used per layer, but I don't think that's a very interesting statistic.
Yes to both, let's do this in a separate PR so that we have separate PRs for the functional change and the refactor. Edit: actually, the distinction between time & space cutoffs is useful for observability, so that we can have metrics that distinguish "how much are retaining because the user asked for it (PITR)'" from "how much are we physically retaining (which may include space-based retention that the user didn't ask for)". |
## Problem Pageserver GC uses a size-based condition (GC "horizon" in addition to time-based "PITR"). Eventually we plan to retire the size-based condition: #6374 Currently, we always apply the more conservative of the two, meaning that tenants always retain at least 64MB of history (default horizon), even after a very long time has passed. This is particularly acute in cases where someone has dropped tables/databases, and then leaves a database idle: the horizon can prevent GCing very large quantities of historical data (we already account for this in synthetic size by ignoring gc horizon). We're not entirely removing GC horizon right now because we don't want to 100% rely on standby_horizon for robustness of physical replication, but we can tweak our logic to avoid retaining that 64MB LSN length indefinitely. ## Summary of changes - Rework `Timeline::find_gc_cutoffs`, with new logic: - If there is no PITR set, then use `DEFAULT_PITR_INTERVAL` (1 week) to calculate a time threshold. Retain either the horizon or up to that thresholds, whichever requires less data. - When there is a PITR set, and we have unambiguously resolved the timestamp to an LSN, then ignore the GC horizon entirely. For typical PITRs (1 day, 1 week), this will still easily retain enough data to avoid stressing read only replicas. The key property we end up with, whether a PITR is set or not, is that after enough time has passed, our GC cutoff on an idle timeline will catch up with the last_record_lsn. Using `DEFAULT_PITR_INTERVAL` is a bit of an arbitrary hack, but this feels like it isn't really worth the noise of exposing in TenantConfig. We could just make it a different named constant though. The end-end state will be that there is no gc_horizon at all, and that tenants with pitr_interval=0 would truly retain no history, so this constant would go away.
Problem
Pageserver GC uses a size-based condition (GC "horizon" in addition to time-based "PITR").
Eventually we plan to retire the size-based condition: #6374
Currently, we always apply the more conservative of the two, meaning that tenants always retain at least 64MB of history (default horizon), even after a very long time has passed. This is particularly acute in cases where someone has dropped tables/databases, and then leaves a database idle: the horizon can prevent GCing very large quantities of historical data (we already account for this in synthetic size by ignoring gc horizon).
We're not entirely removing GC horizon right now because we don't want to 100% rely on standby_horizon for robustness of physical replication, but we can tweak our logic to avoid retaining that 64MB LSN length indefinitely.
Summary of changes
Timeline::find_gc_cutoffs
, with new logic:DEFAULT_PITR_INTERVAL
(1 week) to calculate a time threshold. Retain either the horizon or up to that thresholds, whichever requires less data.The key property we end up with, whether a PITR is set or not, is that after enough time has passed, our GC cutoff on an idle timeline will catch up with the last_record_lsn.
Using
DEFAULT_PITR_INTERVAL
is a bit of an arbitrary hack, but this feels like it isn't really worth the noise of exposing in TenantConfig. We could just make it a different named constant though. The end-end state will be that there is no gc_horizon at all, and that tenants with pitr_interval=0 would truly retain no history, so this constant would go away.Checklist before requesting a review
Checklist before merging