Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver: drop out of secondary download if iteration time has passed #8182

Closed
jcsp opened this issue Jun 27, 2024 · 0 comments · Fixed by #8198
Closed

pageserver: drop out of secondary download if iteration time has passed #8182

jcsp opened this issue Jun 27, 2024 · 0 comments · Fixed by #8198
Assignees
Labels
a/tech_debt Area: related to tech debt c/storage/pageserver Component: storage: pageserver t/on_call_followup

Comments

@jcsp
Copy link
Contributor

jcsp commented Jun 27, 2024

For large tenants, a call to download_timeline can run for a long time, and during that time the actual resident set can change (i.e. the heatmap used at the start of the download is now outdated).

If a tenant's download has taken longer than the configured download period, then we should drop out and let ourselves be rescheduled. This will cost slightly more heatmap downloads, but it's worth it to avoid downloading layers we don't need.

An acute example of the behavior we want to avoid is during shard splits: a new tenant's resident set is initially huge (contains ancestor layers), and soon after becomes much smaller, but if we started secondary downloads then they'll continue trying to download all those ancestor layers even after the attached location has dropped them.

@jcsp jcsp added c/storage/pageserver Component: storage: pageserver a/tech_debt Area: related to tech debt t/on_call_followup labels Jun 27, 2024
@jcsp jcsp self-assigned this Jun 28, 2024
@jcsp jcsp closed this as completed in babbe12 Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a/tech_debt Area: related to tech debt c/storage/pageserver Component: storage: pageserver t/on_call_followup
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant