[new feature] promtail: Add a Promtail stage for probabilistic sampling #7127

liguozhong · 2022-09-10T00:05:32Z

What this PR does / why we need it:

The sampling stage can be directly sampled.
The implementation of sampling is to use the algorithm in jaeger go client

pipeline_stages:
- sampling:
     rate: 0.1

or it can be used with match for precise sampling.

pipeline_stages:
- json:
    expressions:
      app:
- match:
    pipeline_name: "app2"
    selector: "{app=\"poki\"}"
    stages:
    - sampling:
        rate: 0.1

Which issue(s) this PR fixes:
Fixes #6654

Special notes for your reviewer:

The promtail 'rate' stage is also used with the 'match' stage for log filtering.This design makes the code very clean.
Other log agents vector have log filtering built into the sampling operator, which I think is too complicated
https://vector.dev/docs/reference/configuration/transforms/sample/

 transforms:
  my_transform_id:
    type: sample
    inputs:
      - my-source-or-transform-id
    exclude: null
    rate: 10

'rate' stage review suggestions .
#5051

Checklist

Documentation added
Tests updated
Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

grafanabot · 2022-09-10T07:55:59Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

ashwanthgoli · 2022-09-12T11:23:20Z

clients/pkg/logentry/stages/sampling.go

+	}, nil
+}
+
+// dropStage applies Label matchers to determine if the include stages should be run


this needs correction :)

grafanabot · 2022-09-13T06:40:19Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-13T07:24:00Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

ashwanthgoli · 2022-09-14T05:54:22Z

clients/pkg/logentry/stages/sampling.go

+		cfg:              cfg,
+		dropCount:        getDropCountMetric(registerer),
+		samplingBoundary: samplingBoundary,
+		pool: sync.Pool{


I am assuming the reference implementation is using a sync.Pool for storing rand generators since the generator created by NewSource is not safe for concurrent use. And as an added benefit generating random number from this pool would make use of generators created with different seeds.

But in this stage, we have a single go routine that's generating the random numbers which means we don't necessarily need a sync.Pool maybe we should drop this

stale · 2022-11-02T07:58:35Z

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

Mark issues as revivable if we think it's a valid issue but isn't something we are likely
to prioritize in the future (the issue will still remain closed).
Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

# Conflicts: # clients/pkg/logentry/stages/stage.go

jeschkies · 2023-02-09T13:33:59Z

Nice. Could you update the documentation?

liguozhong · 2023-02-09T14:51:03Z

done.

JStickler

[Docs squad} Couple of suggestions and questions for the documentation.

docs/sources/clients/promtail/stages/sampling.md

Co-authored-by: J Stickler <julie.stickler@grafana.com>

liguozhong · 2023-02-10T11:12:37Z

thanks，I modified the document to 2 examples

jeschkies · 2023-02-16T10:50:59Z

I think @JStickler needs to approve 🙂

JStickler

[Docs squad]

liguozhong · 2023-02-28T08:23:40Z

thanks

MasslessParticle · 2023-03-17T21:19:01Z

Awesome, thanks for the contribution!

filip-kolodziej-synerise · 2023-07-25T14:34:39Z

PR was merged 4m ago, any plans for merging it to the release branch as well?

PrayagS · 2023-08-08T06:14:22Z

PR was merged 4m ago, any plans for merging it to the release branch as well?

+1. Looking forward to utilizing this feature. With the limit stage, it's more of a hit-and-miss thing to get the percentage right.

…Promtail stage for probabilistic sampling (#10425) Add PR #7127 to release notes for next release Co-authored-by: Travis Patterson <travis.patterson@grafana.com>

… Add a Promtail stage for probabilistic sampling (grafana#10425) Add PR grafana#7127 to release notes for next release Co-authored-by: Travis Patterson <travis.patterson@grafana.com>

[new feature] promtail: Add a Promtail stage for probabilistic sampling

dbcf4eb

liguozhong requested a review from a team as a code owner September 10, 2022 00:05

pull-request-size bot added the size/L label Sep 10, 2022

liguozhong and others added 12 commits September 10, 2022 08:13

fix lint

e71222c

Update sampling.go

1ffdfcf

Update sampling.go

ed0c2e8

Update sampling.go

d720a63

Update sampling.go

b7a0571

Update sampling.go

f70180e

Update sampling.go

a1ccb59

Update sampling_test.go

a66f641

Update limit_test.go

1ccb1cb

Update sampling_test.go

0a8510f

fix lint

5543c7b

Update sampling_test.go

0ed01d0

ashwanthgoli reviewed Sep 12, 2022

View reviewed changes

delete 'dropStage' word

7bd6dda

Merge branch 'main' into promtail_sampling

179296d

ashwanthgoli reviewed Sep 14, 2022

View reviewed changes

liguozhong added 2 commits September 14, 2022 16:05

remove pool

d8a0371

Merge branch 'main' into promtail_sampling

2315fa9

stale bot added the stale A stale issue or PR that will automatically be closed. label Nov 2, 2022

Merge branch 'main' into promtail_sampling

3a32d06

# Conflicts: # clients/pkg/logentry/stages/stage.go

stale bot removed the stale A stale issue or PR that will automatically be closed. label Feb 9, 2023

fix test

4a59de6

liguozhong added 10 commits February 9, 2023 22:38

Update sampling.md

b2da069

Update sampling.md

47656a2

Update sampling.md

4502df6

Update sampling.md

a91af62

Update sampling.md

0962aed

Update sampling.md

f118389

Update sampling.md

d3cf0ee

Update sampling.md

3bdda1d

Update sampling.md

dbfb86a

Update sampling.md

1c41a4c

JStickler reviewed Feb 9, 2023

View reviewed changes

docs/sources/clients/promtail/stages/sampling.md Outdated Show resolved Hide resolved

docs/sources/clients/promtail/stages/sampling.md Outdated Show resolved Hide resolved

docs/sources/clients/promtail/stages/sampling.md Outdated Show resolved Hide resolved

liguozhong and others added 4 commits February 10, 2023 19:00

Update docs/sources/clients/promtail/stages/sampling.md

8fabe08

Co-authored-by: J Stickler <julie.stickler@grafana.com>

Update docs/sources/clients/promtail/stages/sampling.md

e6eb899

Co-authored-by: J Stickler <julie.stickler@grafana.com>

Update sampling.md

9be2bf7

Update sampling.md

b099e8d

jeschkies approved these changes Feb 16, 2023

View reviewed changes

JStickler approved these changes Feb 16, 2023

View reviewed changes

MasslessParticle merged commit 0295fd4 into grafana:main Mar 17, 2023

MasslessParticle added the add-to-release-notes label Sep 1, 2023

grafanabot mentioned this pull request Sep 1, 2023

[Release Notes Appender] Add PR #7127: [new feature] promtail: Add a Promtail stage for probabilistic sampling #10425

Merged

grafanabot added a commit that referenced this pull request Sep 1, 2023

Add PR #7127 to release notes for next release

48c1744

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[new feature] promtail: Add a Promtail stage for probabilistic sampling #7127

[new feature] promtail: Add a Promtail stage for probabilistic sampling #7127

liguozhong commented Sep 10, 2022 •

edited

Loading

grafanabot commented Sep 10, 2022

ashwanthgoli Sep 12, 2022

grafanabot commented Sep 13, 2022

grafanabot commented Sep 13, 2022

ashwanthgoli Sep 14, 2022

stale bot commented Nov 2, 2022

jeschkies commented Feb 9, 2023

liguozhong commented Feb 9, 2023

JStickler left a comment

liguozhong commented Feb 10, 2023

jeschkies commented Feb 16, 2023

JStickler left a comment

liguozhong commented Feb 28, 2023

MasslessParticle commented Mar 17, 2023

filip-kolodziej-synerise commented Jul 25, 2023

PrayagS commented Aug 8, 2023

[new feature] promtail: Add a Promtail stage for probabilistic sampling #7127

[new feature] promtail: Add a Promtail stage for probabilistic sampling #7127

Conversation

liguozhong commented Sep 10, 2022 • edited Loading

grafanabot commented Sep 10, 2022

ashwanthgoli Sep 12, 2022

Choose a reason for hiding this comment

grafanabot commented Sep 13, 2022

grafanabot commented Sep 13, 2022

ashwanthgoli Sep 14, 2022

Choose a reason for hiding this comment

stale bot commented Nov 2, 2022

jeschkies commented Feb 9, 2023

liguozhong commented Feb 9, 2023

JStickler left a comment

Choose a reason for hiding this comment

liguozhong commented Feb 10, 2023

jeschkies commented Feb 16, 2023

JStickler left a comment

Choose a reason for hiding this comment

liguozhong commented Feb 28, 2023

MasslessParticle commented Mar 17, 2023

filip-kolodziej-synerise commented Jul 25, 2023

PrayagS commented Aug 8, 2023

liguozhong commented Sep 10, 2022 •

edited

Loading