-
Notifications
You must be signed in to change notification settings - Fork 393
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(per-tenant throttling): exclude throttled time from page_service metrics + regression test #6953
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2490 tests run: 2368 passed, 0 failed, 122 skipped (full report)Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
2dcf5d8 at 2024-03-05T13:05:15.472Z :recycle: |
3860565
to
d417cb9
Compare
This was referenced Feb 29, 2024
…covered, measure overhead first
d417cb9
to
bccf459
Compare
This reverts commit 7afbcd1.
…e callers of log_contains callers
…e callers of log_contains callers
…ttling-observability-fixes
d0eac41
to
a5e0fe5
Compare
…ke tests pass, maybe this was root cause of its flakiness
problame
added a commit
that referenced
this pull request
Mar 1, 2024
…in callers (#6960) Extracted from #6953 Part of #5899 Core Change ----------- In #6953, we need the ability to scan the log _after_ a specific line and ignore anything before that line. This PR changes `log_contains` to returns a tuple of `(matching line, cursor)`. Hand that cursor to a subsequent `log_contains` call to search the log for the next occurrence of the pattern. Other Changes ------------- - Inspect all the callsites of `log_contains` to handle the new tuple return type. - Above inspection unveiled many callers aren't using `assert log_contains(...) is not None` but some weaker version of the code that breaks if `log_contains` ever returns a not-None but falsy value. Fix that. - Above changes unveiled that `test_remote_storage_upload_queue_retries` was using `wait_until` incorrectly; after fixing the usage, I had to raise the `wait_until` timeout. So, maybe this will fix its flakiness.
jcsp
approved these changes
Mar 1, 2024
2ee223f
to
41d57a9
Compare
problame
commented
Mar 5, 2024
41d57a9
to
169065f
Compare
This reverts commit 169065f.
problame
added a commit
that referenced
this pull request
Mar 8, 2024
PR #6953 only excluded throttled time from the handle_pagerequests (aka smgr metrics). This PR implements the deduction for `basebackup ` queries. The other page_service methods either don't use Timeline::get or they aren't used in production.
problame
added a commit
that referenced
this pull request
Mar 9, 2024
…7072) PR #6953 only excluded throttled time from the handle_pagerequests (aka smgr metrics). This PR implements the deduction for `basebackup ` queries. The other page_service methods either don't use Timeline::get or they aren't used in production. Found by manually inspecting in [staging logs](https://neonprod.grafana.net/explore?schemaVersion=1&panes=%7B%22wx8%22:%7B%22datasource%22:%22xHHYY0dVz%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhostname%3D%5C%22pageserver-0.eu-west-1.aws.neon.build%5C%22%7D%20%7C~%20%60git-env%7CERR%7CWARN%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22xHHYY0dVz%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22to%22:%221709919114642%22,%22from%22:%221709904430898%22%7D%7D%7D).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
part of #5899
Problem
Before this PR, the time spent waiting on the throttle was charged towards the higher-level page_service metrics, i.e.,
pageserver_smgr_query_seconds
.The metrics are the foundation of internal SLIs / SLOs.
A throttled tenant would cause the SLI to degrade / SLO alerts to fire.
Changes
micros_spent_throttled
counter.info!
, it's an expected conditionPerformance
I took the same measurements as in #6706 , no meaningful change in CPU overhead.
Future Work
This PR enables us to experiment with the throttle for select tenants without affecting the SLI metrics / triggering SLO alerts.
Before declaring this feature done, we need more work to happen, specifically:
Timeline::get
call, filtered by TaskKindNice-to-haves but probably not worth the time right now: