pageserver: drop out of secondary download if iteration time has passed #8182
Labels
a/tech_debt
Area: related to tech debt
c/storage/pageserver
Component: storage: pageserver
t/on_call_followup
For large tenants, a call to
download_timeline
can run for a long time, and during that time the actual resident set can change (i.e. the heatmap used at the start of the download is now outdated).If a tenant's download has taken longer than the configured download period, then we should drop out and let ourselves be rescheduled. This will cost slightly more heatmap downloads, but it's worth it to avoid downloading layers we don't need.
An acute example of the behavior we want to avoid is during shard splits: a new tenant's resident set is initially huge (contains ancestor layers), and soon after becomes much smaller, but if we started secondary downloads then they'll continue trying to download all those ancestor layers even after the attached location has dropped them.
The text was updated successfully, but these errors were encountered: