Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver: add read depth metrics and test #7464

Merged
merged 4 commits into from
Apr 23, 2024
Merged

Conversation

VladLazar
Copy link
Contributor

@VladLazar VladLazar commented Apr 22, 2024

Problem

We recently went through an incident where compaction was inhibited by a bug.
We didn't observe this until quite late because we did not have alerting on deep reads.

Summary of changes

  • Tweak an existing metric that tracks the depth of a read on the non-vectored read path:
    • Give it a better name
    • Track all layers
    • Larger buckets
  • Add a similar metric for the vectored read path
  • Add a compaction smoke test which uses these metrics. This test would have caught
    the compaction issue mentioned earlier.

Once this merges and we have a better understanding for where the upper bound is
merge these warnings to aid debugging: #7465

Related #7428

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

@VladLazar VladLazar changed the title Vlad/read depth metric pageserver: add read depth metrics and test Apr 22, 2024
@VladLazar VladLazar marked this pull request as ready for review April 22, 2024 16:55
@VladLazar VladLazar requested a review from a team as a code owner April 22, 2024 16:55
Copy link

2778 tests run: 2657 passed, 0 failed, 121 skipped (full report)


Code coverage* (full report)

  • functions: 28.1% (6470 of 23045 functions)
  • lines: 46.8% (45704 of 97615 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
ee2de0a at 2024-04-22T17:23:05.758Z :recycle:

@VladLazar VladLazar merged commit 28e7fa9 into main Apr 23, 2024
53 of 54 checks passed
@VladLazar VladLazar deleted the vlad/read-depth-metric branch April 23, 2024 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants