storage controller: fix heatmaps getting disabled during shard split

[jcsp]

Problem

At the start of do_tenant_shard_split, we drop any secondary location for the parent shards. The reconciler uses presence of secondary locations as a condition for enabling heatmaps.

On the pageserver, child shards inherit their configuration from parents, but the storage controller assumes the child's ObservedState is the same as the parent's config from the prepare phase. The result is that some child shards end up with inaccurate ObservedState, and until something next migrates or restarts, those tenant shards aren't uploading heatmaps, so their secondary locations are downloading everything that was resident at the moment of the split (including ancestor layers which are often cleaned up shortly after the split).

Closes: #8189

Summary of changes

• Use PlacementPolicy to control enablement of heatmap upload, rather than the literal presence of secondaries in IntentState: this way we avoid switching them off during shard split
• test: during tenant split test, assert that the child shards have heatmap uploads enabled.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2940 tests run: 2823 passed, 0 failed, 117 skipped (full report)

---

Flaky tests (2)

Postgres 16

• test_subscriber_restart: release

Postgres 15

• test_timeline_size_quota_on_startup: release

Code coverage* (full report)

• functions: 32.7% (6910 of 21115 functions)
• lines: 50.1% (54154 of 108067 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
13d7f11 at 2024-06-28T17:30:46.651Z :recycle:}

[VladLazar]

So the issue was that the observed state was HA aware, but the intent wasn't. This made the tenant shard dirty() and triggered a post shard split reconcile which unset the HA tenant config stuff.

---

Looks good - nice find!