fix(scrubber): more robust metadata consistency checks

[yliang412]

Part of #8128.

Problem

Scrubber uses scan_metadata command to flag metadata inconsistencies. To trust it at scale, we need to make sure the errors we emit is a reflection of real scenario. One check performed in the scrubber is to see whether layers listed in the latest index_part.json is present in object listing. Currently, the scrubber does not robustly handle the case where objects are uploaded/deleted during the scan. Below is some traces on possible race conditions that produces false positives.

Case 1: Layer insertion

RemoteTimelineClient	Scrubber	S3 Ground Truth After Step
		indexed_layers: [A], layers_present: [A]
	read tenant_objects [A]
insert B
upload index to include B		indexed_layers: [A, B], layers_present: [A, B]
	download index [A, B]
	check B is in tenant_objects → false, error

In the above example, B is actually present on S3. To resolve this, we can issue a HeadObject to the missing layers and see if they are actually present.

Case 2: Layer deletion (impossible)

RemoteTimelineClient	Scrubber	S3 Ground Truth After Step
		indexed_layers: [A, B], layers_present: [A, B]
	download index, indexed_layers = [A, B]
delete A from index		indexed_layers: [A], layers_present: [A, B]
delete A		indexed_layers: [B], layers_present: [B]
	read B, tenant_objects = [B], indexed_layers = [A, B]
	check if A is present in tenant objects, nope, → error (this is a false positive)

This case is made impossible by always read the tenant objects listing first before downloading the index.

Case 3: Fast L0 replacement

RemoteTimelineClient	Scrubber	S3 Ground Truth After Step
		indexed_layers: [], layers_present: []
	read tenant_objects []
insert L0
upload index to include L0		indexed_layers: [L0], layers_present: [L0]
	read index [L0]
	check L0 is in tenant_objects → false
upload L1; upload index, delete L0		indexed_layers: [L1], layers_present: [L1]
	head_object(L0) → fail

This is very unlikely but theoretically possible. Instead of rescanning the index, the implementation will emit the error with a caveat indicating the missing layer is L0.

Summary of changes

Condition for success: An object in the index is (1) in the object listing we acquire from S3 or (2) found in a HeadObject request (new object).

• Add in the HeadObject requests for the layers missing from the object listing.
• Keep the order of first getting the object listing and then downloading the layers.
• Update check to only consider shards with highest shard count.
• Skip analyzing a timeline if deleted_at tombstone is marked in index_part.json.
• Add new test to see if scrubber actually detect the metadata inconsistency.

Misc

• A timeline with no ancestor should always have some layers.
• Removed experimental histograms

Caveat

• Ancestor layer is not cleaned until pageserver: tenant deletions after shard splits may leave some layers behind #8308 is implemented. If ancestor layers reference non-existing layers in the index, the scrubber will emit false positives.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3126 tests run: 3005 passed, 0 failed, 121 skipped (full report)

---

Flaky tests (1)

Postgres 15

• test_wal_restore: release

Code coverage* (full report)

• functions: 32.6% (7003 of 21459 functions)
• lines: 50.0% (55263 of 110539 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
a6cfed9 at 2024-07-22T13:48:58.375Z :recycle:}

[skyzh]

did a quick review, have you tried running this with staging pageserver? I assume this process is read-only and it should be safe to run it.

[yliang412]

Tested on staging, logged warnings and errors as expected.

[jcsp]

This looks good, couple of minor comments.