feat(pageserver): add aux_file_v2 feature flag

[skyzh]

Problem

Added aux_file_v2 to the TimelineMetadata structure. The structure has not been touched for several months so I'm skeptical whether we can really add it there...

The flag cannot be reverted to false once being set. Once the tenant config has try_enable_aux_file_v2 set to true, the write path of AUX file will read this flag and set the timeline to have aux_file_v2 enabled. This flag will be persisted into the index part, so even if the flag got lost in the tenant config, the page server will still proceed with the v2 storage.

Summary of changes

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2754 tests run: 2633 passed, 1 failed, 120 skipped (full report)

---

Failures on Postgres 16

• test_ts_of_lsn_api: release

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_ts_of_lsn_api[release-pg16]"

Flaky tests (1)

Postgres 15

• test_vm_bit_clear_on_heap_lock: release

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
37e25ab at 2024-04-24T18:30:47.141Z :recycle:}

[skyzh]

need to fix unit tests, e2e works.

[arpad-m]

As said on slack, I think for the rollout we can just add it as a per-tenant config, no need to have it in timeline metadata. That would simplify this I think.

[skyzh]

@arpad-m The timeline metadata is here to prevent the flag from being accidentally reverted, and also handle branching/history correctly. If the control plane / storage controller can guarantee that the flag won't be flipped, then it's fine to set it as a tenant config. (I can simply rename the current tenant config try_enable_aux_file_v2 to aux_file_v2 and drop all timeline metadata code if we decide to do so.)

[skyzh]

I feel the only thing we are uncertain is whether we need to persist the aux_file_v2 state. Maybe I can submit a pull request with only tenant config change to unblock myself, and go back to this discussion later?

[skyzh]

Opened #7505 instead. We can discuss the metadata change later.