pageserver: proactive post-split compaction to drop unneeded data on child shards

[jcsp]

This is an optimization mentioned in the sharding RFC, to reduce the physical size of child shards after a split, by self-compacting historic layers to remove data that doesn't belong on this shard.

Background

There are broadly three cases:

• Happy path: we're splitting a small tenant where there isn't much data yet. Doesn't really matter if ancestor shard's layers remain around for a while.
• Okay path: we're splitting a large tenant, but after a while compaction on the child shards enables GC of layers from the parent shard.
• Expensive path: we're splitting a large tenant, and there are child timelines that prevent GC of old layers. The child shards will all keep layers from the parent shard around indefinitely.

Solution

Implement a special compaction pass that re-writes existing layer files, using the same shard filtering that we already do in create_image_layers. (Aside: this capability might also be useful in future when we want to do format migrations to re-write existing layers)

There are a few open questions on the details:

• How to handle re-writing a layer with the same name as an existing one (a re-written layer would have a different path in remote storage, but the same name in local storage).
• Whether to bother re-writing layers in general, or just to output re-written image layers at key LSNs (e.g. branch points).