pageserver: circuit breaker on compaction

[jcsp]

Problem

We already back off on compaction retries, but the impact of a failing compaction can be so great that backing off up to 300s isn't enough. The impact is consuming a lot of I/O+CPU in the case of image layer generation for large tenants, and potentially also leaking disk space.

Compaction failures are extremely rare and almost always indicate a bug, frequently a bug that will not let compaction to proceed until it is fixed.

Related: #6738

Summary of changes

• Introduce a CircuitBreaker type
• Add a circuit breaker for compaction, with a policy that after 5 failures, compaction will not be attempted again for 24 hours.
• Add metrics that we can alert on: any >0 value for pageserver_circuit_breaker_broken_total should generate an alert.
• Add a test that checks this works as intended.

Couple notes to reviewers:

• Circuit breakers are intrinsically a defense-in-depth measure: this is not the solution to any underlying issues, it is just a general mitigation for "unknown unknowns" that might be encountered in future.
• This PR isn't primarily about writing a perfect CircuitBreaker type: the one in this PR is meant to be just enough to mitigate issues in compaction, and make it easy to monitor/alert on these failures. We can refine this type in future as/when we want to use it elsewhere.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3054 tests run: 2939 passed, 0 failed, 115 skipped (full report)

---

Code coverage* (full report)

• functions: 32.7% (6972 of 21322 functions)
• lines: 50.1% (54804 of 109498 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
981a288 at 2024-07-11T17:16:29.814Z :recycle:}

[skyzh]

While this prevents failing compaction corrupting/lagging the system, it does not solve the fundamental issue that compaction should clean up resources after failed attempts (#8362). I will come up with more fixes.

Meanwhile, I assume we have set up alerts for pageserver error logs?