[enhancement] querier : Add "query_memory_only" to make loki have option to rely only on memory availability like google monarch. #5093
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ion to rely only on memory availability like google monarch.
|
@dannykopping review pr please . |
|
Today s3 is unavailable again for 4 hours。
|
sandeepsukhani
approved these changes
Jan 14, 2022
Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>
…ion to rely only on memory availability like google monarch. grafana#5093
done,thanks |
|
@owen-d review pr please . |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Add "query_memory_only" to make loki have option to rely only on memory availability like google monarch.
1: momery + store mode
2: momery mode
Which issue(s) this PR fixes:
Fixes nil
Special notes for your reviewer:
<Monarch: Google’s Planet-Scale In-Memory
Time Series Database>
https://www.vldb.org/pvldb/vol13/p3181-adams.pdf
Why is important
Last week, due to the 1.5-hour unavailability of the s3 service in our area, the loki cluster operator by me was also completely unavailable for 1.5 hours, which had a very negative impact on my work。
In order to solve this problem, I checked After some information on how other companies solve this problem, I found that google's monarch has an in-memory mode that allows monitoring to be available when storage is unavailable. So I think this mode can help loki operators like me.
ref unavailable loki snapshot：
If there are similar option, when s3 is unavailable, I can adjust the querier to memory mode to downgrade the service of loki, wait for s3 to become available, and then restore to a fully available loki, so that the failure will not be too serious.
Hope similar failures don't happen to other loki operators like me.