pageserver: drop out of secondary download if iteration time has passed

[jcsp]

Problem

Very long running downloads can be wasteful, because the heatmap they're using is outdated after a few minutes.

Closes: #8182

Summary of changes

• Impose a deadline on timeline downloads, using the same period as we use for scheduling, and returning an UpdateError::Restart when it is reached. This restart will involve waiting for a scheduling interval, but that's a good thing: it helps let other tenants proceed.
• Refactor download_timeline so that the part where we update the state for local layers is done even if we fall out of the layer download loop with an error: this is important, especially for big tenants, because only layers in the SecondaryDetail state will be considered for eviction.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2965 tests run: 2849 passed, 0 failed, 116 skipped (full report)

---

Flaky tests (3)

Postgres 16

• test_pg_regress[None]: debug

Postgres 15

• test_lr_with_slow_safekeeper: release
• test_ondemand_wal_download_in_replication_slot_funcs: debug

Code coverage* (full report)

• functions: 32.7% (6907 of 21118 functions)
• lines: 50.1% (54141 of 108100 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
424f6f5 at 2024-06-28T17:13:07.104Z :recycle:}