rfcs: add RFC for timeline archival

[jcsp]

A design for a cheap low-resource state for idle timelines:

• Epic: Pageserver Timeline Archival #8088

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3117 tests run: 3002 passed, 0 failed, 115 skipped (full report)

---

Flaky tests (2)

Postgres 16

• test_subscriber_restart: release

Postgres 14

• test_lsn_lease_size[False]: debug

Code coverage* (full report)

• functions: 32.6% (6938 of 21280 functions)
• lines: 50.0% (54545 of 109057 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
f7338e2 at 2024-07-09T18:16:41.194Z :recycle:}

[VladLazar]

Agree with the overal approach

---

Got some clarification requests, suggestions, and a concern about integrity of tenant manifest updates.

[VladLazar]

very nice

[jcsp]

What is the "visible set" though? That's what I was trying to get at.

Added a sentence to make that clearer -- it's a concept I want to add soon and use a few different places (see also: https://neondb.slack.com/archives/C06Q6MQ2QSC/p1719825156554879)

[arpad-m]

this is a well written RFC, it is thought through and I think a good plan forward. Thinking of it, the feature is bigger than I thought, it touches so many components of the pageserver, but I think there is a clear path to getting it implemented piece by piece.

I'd like to see a description of the deletion flow for a timeline, now that it is mentioned in the manifest. Should we first delete the timeline folder and then remove it from the manifest, or should it be the other way around?

And how does that interact with timelines that are either in the process of being offloaded or in the process of being archived, i.e. if we first delete in the manifest and then crash, will it look after a restart like a non-offloaded but archived timeline?