fix(compute_ctl): race condition in configurator

[petuhovskiy]

There was a tricky race condition in compute_ctl, that sometimes makes configurator skip updates. It makes a deadlock because:

• control-plane cannot configure compute, because it's in ConfigurationPending state
• compute_ctl doesn't do any reconfiguration because configurator_main_loop missed notification for it

Full sequence that reproduces the issue:

1. start_compute finishes works and changes status self.set_status(ComputeStatus::Running);
2. configurator received update about Running state and dropped the mutex lock in the iteration
3. /configure request was triggered at the same time as step 1, and got the mutex lock
4. same /configure request set the spec and updated the state to ConfigurationPending, also sent a notification
5. next iteration in configurator got the mutex lock, but missed the notification

There are more details in this slack thread: https://neondb.slack.com/archives/C03438W3FLZ/p1727281028478689?thread_ts=1727261220.483799&cid=C03438W3FLZ

patch author: @ololobus

[github-actions]

5013 tests run: 4849 passed, 0 failed, 164 skipped (full report)

---

Flaky tests (5)

Postgres 17

• test_pageserver_compaction_smoke: release-arm64
• test_subscriber_restart: release-x86-64, debug-x86-64

Postgres 16

• test_subscriber_restart: release-x86-64

Postgres 15

• test_subscriber_restart: release-x86-64

Code coverage* (full report)

• functions: 32.0% (7494 of 23396 functions)
• lines: 50.0% (60473 of 120865 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
dbed75f at 2024-09-26T11:55:51.290Z :recycle:}

[ololobus]

Had a comment already written locally :) I think it's worth of elaborating a bit