storage controller: add node deletion API

[jcsp]

Problem

In anticipation of later adding a really nice drain+delete API, I initially only added an intentionally basic /drop API that is just about usable for deleting nodes in a pinch, but requires some ugly storage controller restarts to persuade it to restart secondaries.

Summary of changes

I started making a few tiny fixes, and ended up writing the delete API...

• Quality of life nit: ordering of node + tenant listings in storcon_cli
• Papercut: Fix the attach_hook using the wrong operation type for reporting slow locks
• Make Service::spawn tolerate generation_pageserver columns that point to nonexistent node IDs. I started out thinking of this as a general resilience thing, but when implementing the delete API I realized it was actually a legitimate end state after the delete API is called (as that API doesn't wait for all reconciles to succeed).
• Add a DELETE API for nodes, which does not gracefully drain, but does reschedule everything. This becomes safe to use when the system is in any state, but will incur availability gaps for any tenants that weren't already live-migrated away. If tenants have already been drained, this becomes a totally clean + safe way to decom a node.
• Add a test and a storcon_cli wrapper for it

This is meant to be a robust initial API that lets us remove nodes without doing ugly things like restarting the storage controller -- it's not quite a totally graceful node-draining routine yet. There's more work in #8333 to get to our end-end state.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3054 tests run: 2939 passed, 0 failed, 115 skipped (full report)

---

Flaky tests (2)

Postgres 16

• test_isolation[None]: debug
• test_tenant_creation_fails: debug

Code coverage* (full report)

• functions: 32.6% (6971 of 21351 functions)
• lines: 50.0% (54789 of 109536 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
20b69e7 at 2024-07-11T12:54:08.057Z :recycle:}

[VladLazar]

Have you considered implementing this as a background operation where the caller has to poll for the absence of the node? It would look a lot like the drain code and I think it would be easier on the operator (i.e us 😄 ).

[jcsp]

Have you considered implementing this as a background operation where the caller has to poll for the absence of the node? It would look a lot like the drain code and I think it would be easier on the operator (i.e us 😄 ).

Yes, we should add something like that. This command is meant as a stepping stone that gives us a clean way to remove a node, without necessarily being totally graceful about draining it (yet)

[jcsp]

I created #8333 to track the remaining work to make node removal really slick.