pageserver: return ACCEPTED when deletion already in flight

[jcsp]

Problem

test_sharding_smoke recently got an added section that checks deletion of a sharded tenant. The storage controller does a retry loop for deletion, waiting for a 404 response. When deletion is a bit slow (debug builds), the retry of deletion was getting a 500 response -- this caused the test to become flaky (example failure: https://neon-github-public-dev.s3.amazonaws.com/reports/release-proxy/8659801445/index.html#testresult/b4cbf5b58190f60e/retries)

There was a false comment in the code:

         match tenant.current_state() {
             TenantState::Broken { .. } | TenantState::Stopping { .. } => {
-                // If a tenant is broken or stopping, DeleteTenantFlow can
-                // handle it: broken tenants proceed to delete, stopping tenants
-                // are checked for deletion already in progress.

If the tenant is stopping, DeleteTenantFlow does not in fact handle it, but returns a 500-yielding errror.

Summary of changes

Before calling into DeleteTenantFlow, if the tenant is in stopping|broken state then return 202 if a deletion is in progress. This makes the API friendlier for retries.

The historic AlreadyInProgress (409) response still exists for if we enter DeleteTenantFlow and unexpectedly see the tenant stopping. That should go away when we implement #5080 . For the moment, callers that handle 409s should continue to do so.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2748 tests run: 2630 passed, 0 failed, 118 skipped (full report)

---

Code coverage* (full report)

• functions: 28.0% (6430 of 22963 functions)
• lines: 46.6% (45026 of 96567 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
c779e5b at 2024-04-15T16:11:49.383Z :recycle:}

[problame]

The test_tenant_delete_concurrent description still says

    Validate that concurrent delete requests to the same tenant behave correctly:
    exactly one should succeed.

Should be updated to

...  succeed without actually entering the deletion code

[jcsp]

The test_tenant_delete_concurrent description still says...

Updated in c779e5b