Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Series API requests give inconsistent and/or missing results #4547

Closed
slim-bean opened this issue Oct 26, 2021 · 2 comments · Fixed by #4563
Closed

Series API requests give inconsistent and/or missing results #4547

slim-bean opened this issue Oct 26, 2021 · 2 comments · Fixed by #4563
Assignees
@cyriltovena

Comments

@slim-bean
Copy link
Collaborator

Describe the bug

Using grafana label_values() which looks to call the Loki series API has caused some people to realize that results are missing and the behavior changes inconsistently, sometimes changing the timerange returns results, sometimes it doesn't.

To Reproduce

I don't have great info here but based on reports and my attempts to verify the issue I can confirm I saw the issue.

I was able to make logcli requests against old data from the reported timerange and confirmed that labels where not returned which should have been via the Series API

A little bit more playing I did seemed like any completely in the past time range struggled to return results, however the traces and execution time of the query often made it seem like the work was done to get the result yet logcli got an empty result set.

When the timerange included now() as the through time, I was getting results.

It's often seen in a dashboard which uses a variable that reloads when the timerange changes, and then zoom in on a graph and see dropdown values change or disappear.
@cyriltovena cyriltovena self-assigned this Oct 26, 2021
@cyriltovena
Copy link
Contributor

I have a fix for this, I'll send the PR tomorrow.

cyriltovena added a commit to cyriltovena/loki that referenced this issue Oct 27, 2021
@cyriltovena
Fixes the Series function to hanlde properly sharding.
0b838f0 
Before they would try to match the injected label but this label is only used by the store
and not returned in the chunks causing all chunks to be filtered out.

Fixes grafana#4547

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena cyriltovena mentioned this issue Oct 27, 2021
Merged
slim-bean pushed a commit that referenced this issue Oct 27, 2021
@cyriltovena
Fixes the Series function to hanlde properly sharding. (#4563)
165ff31 
Before they would try to match the injected label but this label is only used by the store
and not returned in the chunks causing all chunks to be filtered out.

Fixes #4547

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@jiayi-1994
Copy link

#8034 @cyriltovena Problem recurrence， Are there any good solutions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cyriltovena cyriltovena

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixes the Series function to handle properly sharding. cyriltovena/loki
3 participants
@cyriltovena @slim-bean @jiayi-1994