control_plane: revise compute_hook locking (don't serialise all calls)

[jcsp]

Problem

• Previously, an async mutex was held for the duration of ComputeHook::notify. This served multiple purposes:
  • Ensure updates to a given tenant are sent in the proper order
  • Prevent concurrent calls into neon_local endpoint updates in test environments (neon_local is not safe to call concurrently)
  • Protect the inner ComputeHook::state hashmap that is used to calculate when to send notifications.

This worked, but had the major downside that while we're waiting for a compute hook request to the control plane to succeed, we can't notify about any other tenants. Notifications block progress of live migrations, so this is a problem.

Summary of changes

• Protect ComputeHook::state with a sync lock instead of an async lock
• Use a separate async lock ( ComputeHook::neon_local_lock ) for preventing concurrent calls into neon_local, and only take this in the neon_local code path.
• Add per-tenant async locks in ShardedComputeHookTenant, and use these to ensure that only one remote notification can be sent at once per tenant. If several shards update concurrently, their updates will be coalesced.
• Add an explicit semaphore that limits concurrency of calls into the cloud control plane.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2802 tests run: 2678 passed, 0 failed, 124 skipped (full report)

---

Flaky tests (3)

Postgres 14

• test_timeline_init_break_before_checkpoint: debug
• test_empty_branch_remote_storage_upload: debug
• test_branched_from_many_empty_parents_size: debug

Code coverage* (full report)

• functions: 28.0% (6406 of 22857 functions)
• lines: 46.9% (45098 of 96230 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
cf1b476 at 2024-04-06T20:07:07.354Z :recycle:}

[jcsp]

This didn't reduce down as neatly as I had hoped, but I think the logic is sound: we try to acquire a ComputeHookTenant's async lock before sending an update, and if we can't acquire it we drop out, wait for the lock, and try again. The request we send is always derived from latest state of the ComputeHookTenant, so there is no longer any sensitivity to which order notify() calls complete in.

I considered using an async lock for ComputeHook::state to make the control flow a bit less tortured, but on balance I prefer to use a sync lock so that the compiler enforces that we do not hold it while waiting for IO.

[problame]

I think this code has no locking bugs, but, it's all quite brittle / hard to follow.

My only strong request for chagnes is the #7088 (comment)

The naming changes would be pretty nice, but, if you don't have time for them, I can do them in a follow-up.

[jcsp]

Comments on names/structs are all quite reasonable, amended in cf1b476