pageserver/controller: enable tenant deletion without attachment

[jcsp]

Problem

As described in #7952, the controller's attempt to reconcile a tenant before finally deleting it can get hung up waiting for the compute notification hook to accept updates.

The fact that we try and reconcile a tenant at all during deletion is part of a more general design issue (#5080), where deletion was implemented as an operation on attached tenant, requiring the tenant to be attached in order to delete it, which is not in principle necessary.

Closes: #7952

Summary of changes

• In the pageserver deletion API, only do the traditional deletion path if the tenant is attached. If it's secondary, then tear down the secondary location, and then do a remote delete. If it's not attached at all, just do the remote delete.
• In the storage controller, instead of ensuring a tenant is attached before deletion, do a best-effort detach of the tenant, and then call into some arbitrary pageserver to issue a deletion of remote content.

The pageserver retains its existing delete behavior when invoked on attached locations. We can remove this later when all users of the API are updated to either do a detach-before-delete. This will enable removing the "weird" code paths during startup that sometimes load a tenant and then immediately delete it, and removing the deletion markers on tenants.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3186 tests run: 3048 passed, 0 failed, 138 skipped (full report)

---

Flaky tests (2)

Postgres 15

• test_storage_controller_smoke: debug
• test_vm_bit_clear_on_heap_lock: debug

Code coverage* (full report)

• functions: 31.5% (6599 of 20927 functions)
• lines: 48.5% (51059 of 105267 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
ffee566 at 2024-06-05T20:30:44.791Z :recycle:}

[koivunej]

This seems good to me. I am thinking of if there could be situations where we'd still try to attach the tenant for deletion once the "raw deletion" introduced here has been started, but no, because this is only for sharded tenants.

[problame]

LGTM and looks like what we discussed many months ago (in Vienna?) wrt doing the deletions through the PS, even if we could issue them from the controller just as well.

Few remarks / concerns on edge cases and asks for follow-ups, please resolve and/or create issues for yourself as needed.