pageserver: refine shutdown handling in secondary download

[jcsp]

Problem

Some code paths during secondary mode download are returning Ok() rather than UpdateError::Cancelled. This is functionally okay, but it means that the end of TenantDownloader::download has a sanity check that the progress is 100% on success, and prints a "Correcting drift..." warning if not. This warning can be emitted in a test, e.g. https://neon-github-public-dev.s3.amazonaws.com/reports/pr-8049/9503642976/index.html#/testresult/fff1624ba6adae9e.

Summary of changes

• In secondary download cancellation paths, use Err(UpdateError::Cancelled) rather than Ok(), so that we drop out of the download function and do not reach the progress sanity check.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3216 tests run: 3074 passed, 0 failed, 142 skipped (full report)

---

Flaky tests (3)

Postgres 16

• test_sharding_split_compaction[compact-shard-ancestors-persistent]: debug

Postgres 15

• test_ondemand_wal_download_in_replication_slot_funcs: debug

Postgres 14

• test_pageserver_restarts_under_worload: release

Code coverage* (full report)

• functions: 31.5% (6635 of 21060 functions)
• lines: 48.6% (51631 of 106254 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
6026890 at 2024-06-13T19:42:02.764Z :recycle:}