docs: rolling storage controller restarts RFC

[VladLazar]

Problem

Storage controller upgrades (restarts, more generally) can cause multi-second availability gaps.
While the storage controller does not sit on the main data path, it's generally not acceptable
to block management requests for extended periods of time (e.g. #8034).

Summary of changes

This RFC describes the issues around the current storage controller restart procedure
and describes an implementation which reduces downtime to a few milliseconds on the happy path.

Related #7797

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

No tests were run or test report is not available

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
91b4533 at 2024-07-24T13:29:33.042Z :recycle:}

[jcsp]

Thinking of extra safety measures: we might in future like to carry an HTTP header on controller requests to pageservers, which would change for new leaders, so that pageservers could refuse requests from stale leaders. Might be worth embedding some counter in the leader table for that purpose.