Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Line Filters in StringLabelFilter #8659

Merged
merged 7 commits into from
Mar 3, 2023

Conversation

MasslessParticle
Copy link
Contributor

We've done a ton of work to optimize regexes in line filters but nothing in label filters. To avoid duplicating the work or reworking all of our filter types, this PR creates a new label filter that has an optimized line filter. It uses that against the label value to determine if they match.

Note on Benchmarks: the input string for this is 1114195 bytes long. The not test tells us that our approach is quite a lot slower when the literal pattern isn't present in the input. The size of the input really emphasizes it.

Benchmarks:

name                                        old time/op    new time/op        delta
LineLabelFilters/foo-8                        50.6ns ± 9%        14.3ns ± 1%        -71.67%  (p=0.000 n=10+8)
LineLabelFilters/not-8                        15.7ns ± 1%     64165.6ns ± 2%    +407766.77%  (p=0.000 n=10+10)
LineLabelFilters/(foo)-8                      58.4ns ± 1%        14.8ns ±10%        -74.68%  (p=0.000 n=9+9)
LineLabelFilters/(foo|ba)-8                   64.9ns ± 0%        18.2ns ± 2%        -71.90%  (p=0.000 n=7+9)
LineLabelFilters/(foo|ba|ar)-8                74.8ns ± 7%        21.4ns ± 2%        -71.42%  (p=0.000 n=10+8)
LineLabelFilters/(foo|(ba|ar))-8              65.1ns ± 1%        18.1ns ± 2%        -72.22%  (p=0.000 n=8+8)
LineLabelFilters/foo.*-8                       882ns ± 4%          15ns ±11%        -98.30%  (p=0.000 n=9+10)
LineLabelFilters/.*foo-8                      11.5ns ± 8%        14.9ns ± 7%        +30.39%  (p=0.000 n=10+10)
LineLabelFilters/.*foo.*-8                    5.46µs ± 7%        0.01µs ± 3%        -99.74%  (p=0.000 n=10+9)
LineLabelFilters/(.*)(foo).*-8                6.26µs ± 7%        0.01µs ± 2%        -99.77%  (p=0.000 n=10+9)
LineLabelFilters/(foo.*|.*ba)-8               5.58µs ± 3%        0.02µs ± 8%        -99.67%  (p=0.000 n=9+10)
LineLabelFilters/(foo.*|.*bar.*)-8            7.02µs ± 7%        0.02µs ± 8%        -99.73%  (p=0.000 n=10+10)
LineLabelFilters/.*foo.*|bar-8                5.53µs ± 6%        0.02µs ± 2%        -99.67%  (p=0.000 n=10+8)
LineLabelFilters/.*foo|bar-8                  3.73µs ± 6%        0.02µs ± 3%        -99.52%  (p=0.000 n=10+9)
LineLabelFilters/(?:.*foo.*|bar)-8            5.43µs ± 8%        0.02µs ± 8%        -99.65%  (p=0.000 n=10+10)
LineLabelFilters/(?P<foo>.*foo.*|bar)-8       5.71µs ± 3%        0.02µs ±10%        -99.68%  (p=0.000 n=9+9)
LineLabelFilters/.*foo.*|bar|buzz-8           5.34µs ± 2%        0.02µs ± 3%        -99.65%  (p=0.000 n=10+9)
LineLabelFilters/.*foo.*|bar|uzz-8            5.36µs ± 1%        0.02µs ± 3%        -99.60%  (p=0.000 n=8+8)
LineLabelFilters/foo|bar|b|buzz|zz-8          68.8ns ± 8%        22.6ns ± 7%        -67.19%  (p=0.000 n=10+10)
LineLabelFilters/f|foo|foobar-8               77.1ns ± 1%        18.6ns ± 9%        -75.90%  (p=0.000 n=8+10)
LineLabelFilters/f.*|foobar.*|.*buzz-8        5.64µs ± 7%        0.02µs ±10%        -99.60%  (p=0.000 n=10+10)
LineLabelFilters/((f.*)|foobar.*)|.*buzz-8    6.29µs ± 8%        0.02µs ±10%        -99.65%  (p=0.000 n=10+10)
LineLabelFilters/.*-8                          963ns ± 8%           2ns ±14%        -99.77%  (p=0.000 n=10+10)
LineLabelFilters/.*|.*-8                      4.46µs ± 8%        0.00µs ± 8%        -99.95%  (p=0.000 n=10+10)
LineLabelFilters/.*||||-8                     2.99µs ± 1%        0.00µs ±16%        -99.93%  (p=0.000 n=8+10)
LineLabelFilters/#00-8                        43.5ns ± 8%         2.1ns ± 1%        -95.25%  (p=0.000 n=10+8)
LineLabelFilters/(?i)foo-8                    71.7ns ± 6%        13.1ns ± 1%        -81.73%  (p=0.000 n=10+8)
LineLabelFilters/(?i)界-8                      16.5ns ± 8%       150.2ns ± 8%       +809.55%  (p=0.000 n=10+10)
LineLabelFilters/(?i)ïB-8                     23.8ns ± 1%  14748979.8ns ± 7%  +62074730.81%  (p=0.000 n=8+10)
LineLabelFilters/(?:)foo|fatal|exception-8     185ns ± 2%          21ns ± 2%        -88.38%  (p=0.000 n=9+8)
LineLabelFilters/(?i)foo|fatal|exception-8    78.4ns ± 3%        20.2ns ± 2%        -74.23%  (p=0.000 n=9+9)
LineLabelFilters/(?i)f|foo|foobar-8           94.3ns ±16%        15.4ns ± 9%        -83.66%  (p=0.000 n=10+10)
LineLabelFilters/(?i)f|fatal|e.*-8            65.7ns ± 5%        18.2ns ± 3%        -72.24%  (p=0.000 n=9+8)
LineLabelFilters/(?i).*foo.*-8                6.04µs ± 8%        0.01µs ± 2%        -99.78%  (p=0.000 n=10+10)

name                                        old alloc/op   new alloc/op       delta
LineLabelFilters/foo-8                         0.00B              0.00B                ~     (all equal)
LineLabelFilters/not-8                         0.00B              0.00B                ~     (all equal)
LineLabelFilters/(foo)-8                       0.00B              0.00B                ~     (all equal)
LineLabelFilters/(foo|ba)-8                    0.00B              0.00B                ~     (all equal)
LineLabelFilters/(foo|ba|ar)-8                 0.00B              0.00B                ~     (all equal)
LineLabelFilters/(foo|(ba|ar))-8               0.00B              0.00B                ~     (all equal)
LineLabelFilters/foo.*-8                       0.00B              0.00B                ~     (all equal)
LineLabelFilters/.*foo-8                       0.00B              0.00B                ~     (all equal)
LineLabelFilters/.*foo.*-8                     0.00B              0.00B                ~     (all equal)
LineLabelFilters/(.*)(foo).*-8                 0.00B              0.00B                ~     (all equal)
LineLabelFilters/(foo.*|.*ba)-8                0.00B              0.00B                ~     (all equal)
LineLabelFilters/(foo.*|.*bar.*)-8             0.00B              0.00B                ~     (all equal)
LineLabelFilters/.*foo.*|bar-8                 0.00B              0.00B                ~     (all equal)
LineLabelFilters/.*foo|bar-8                   0.00B              0.00B                ~     (all equal)
LineLabelFilters/(?:.*foo.*|bar)-8             0.00B              0.00B                ~     (all equal)
LineLabelFilters/(?P<foo>.*foo.*|bar)-8        0.00B              0.00B                ~     (all equal)
LineLabelFilters/.*foo.*|bar|buzz-8            0.00B              0.00B                ~     (all equal)
LineLabelFilters/.*foo.*|bar|uzz-8             0.00B              0.00B                ~     (all equal)
LineLabelFilters/foo|bar|b|buzz|zz-8           0.00B              0.00B                ~     (all equal)
LineLabelFilters/f|foo|foobar-8                0.00B              0.00B                ~     (all equal)
LineLabelFilters/f.*|foobar.*|.*buzz-8         0.00B              0.00B                ~     (all equal)
LineLabelFilters/((f.*)|foobar.*)|.*buzz-8     0.00B              0.00B                ~     (all equal)
LineLabelFilters/.*-8                          0.00B              0.00B                ~     (all equal)
LineLabelFilters/.*|.*-8                       0.00B              0.00B                ~     (all equal)
LineLabelFilters/.*||||-8                      0.00B              0.00B                ~     (all equal)
LineLabelFilters/#00-8                         0.00B              0.00B                ~     (all equal)
LineLabelFilters/(?i)foo-8                     0.00B              0.00B                ~     (all equal)
LineLabelFilters/(?i)界-8                       0.00B              0.00B                ~     (all equal)
LineLabelFilters/(?i)ïB-8                      0.00B              0.00B                ~     (all equal)
LineLabelFilters/(?:)foo|fatal|exception-8     0.00B              0.00B                ~     (all equal)
LineLabelFilters/(?i)foo|fatal|exception-8     0.00B              0.00B                ~     (all equal)
LineLabelFilters/(?i)f|foo|foobar-8            0.00B              0.00B                ~     (all equal)
LineLabelFilters/(?i)f|fatal|e.*-8             0.00B              0.00B                ~     (all equal)
LineLabelFilters/(?i).*foo.*-8                 0.00B              0.00B                ~     (all equal)

name                                        old allocs/op  new allocs/op      delta
LineLabelFilters/foo-8                          0.00               0.00                ~     (all equal)
LineLabelFilters/not-8                          0.00               0.00                ~     (all equal)
LineLabelFilters/(foo)-8                        0.00               0.00                ~     (all equal)
LineLabelFilters/(foo|ba)-8                     0.00               0.00                ~     (all equal)
LineLabelFilters/(foo|ba|ar)-8                  0.00               0.00                ~     (all equal)
LineLabelFilters/(foo|(ba|ar))-8                0.00               0.00                ~     (all equal)
LineLabelFilters/foo.*-8                        0.00               0.00                ~     (all equal)
LineLabelFilters/.*foo-8                        0.00               0.00                ~     (all equal)
LineLabelFilters/.*foo.*-8                      0.00               0.00                ~     (all equal)
LineLabelFilters/(.*)(foo).*-8                  0.00               0.00                ~     (all equal)
LineLabelFilters/(foo.*|.*ba)-8                 0.00               0.00                ~     (all equal)
LineLabelFilters/(foo.*|.*bar.*)-8              0.00               0.00                ~     (all equal)
LineLabelFilters/.*foo.*|bar-8                  0.00               0.00                ~     (all equal)
LineLabelFilters/.*foo|bar-8                    0.00               0.00                ~     (all equal)
LineLabelFilters/(?:.*foo.*|bar)-8              0.00               0.00                ~     (all equal)
LineLabelFilters/(?P<foo>.*foo.*|bar)-8         0.00               0.00                ~     (all equal)
LineLabelFilters/.*foo.*|bar|buzz-8             0.00               0.00                ~     (all equal)
LineLabelFilters/.*foo.*|bar|uzz-8              0.00               0.00                ~     (all equal)
LineLabelFilters/foo|bar|b|buzz|zz-8            0.00               0.00                ~     (all equal)
LineLabelFilters/f|foo|foobar-8                 0.00               0.00                ~     (all equal)
LineLabelFilters/f.*|foobar.*|.*buzz-8          0.00               0.00                ~     (all equal)
LineLabelFilters/((f.*)|foobar.*)|.*buzz-8      0.00               0.00                ~     (all equal)
LineLabelFilters/.*-8                           0.00               0.00                ~     (all equal)
LineLabelFilters/.*|.*-8                        0.00               0.00                ~     (all equal)
LineLabelFilters/.*||||-8                       0.00               0.00                ~     (all equal)
LineLabelFilters/#00-8                          0.00               0.00                ~     (all equal)
LineLabelFilters/(?i)foo-8                      0.00               0.00                ~     (all equal)
LineLabelFilters/(?i)界-8                        0.00               0.00                ~     (all equal)
LineLabelFilters/(?i)ïB-8                       0.00               0.00                ~     (all equal)
LineLabelFilters/(?:)foo|fatal|exception-8      0.00               0.00                ~     (all equal)
LineLabelFilters/(?i)foo|fatal|exception-8      0.00               0.00                ~     (all equal)
LineLabelFilters/(?i)f|foo|foobar-8             0.00               0.00                ~     (all equal)
LineLabelFilters/(?i)f|fatal|e.*-8              0.00               0.00                ~     (all equal)
LineLabelFilters/(?i).*foo.*-8                  0.00               0.00                ~     (all equal)

@MasslessParticle MasslessParticle requested a review from a team as a code owner February 28, 2023 23:40
@jeschkies
Copy link
Contributor

Nice find! Would you have a flame graph for the Unicode case? This is quite a drop.

@MasslessParticle MasslessParticle force-pushed the tpatterson/share-regex-optimizations branch from d27bcc0 to 225077c Compare March 1, 2023 18:18
@MasslessParticle
Copy link
Contributor Author

@jeschkies Turns out it's super expensive to call toLower on unicode:

image

@MasslessParticle
Copy link
Contributor Author

MasslessParticle commented Mar 1, 2023

@jeschkies I dug into this today and discovered that Prometheus Label Semantics are different than line filters. Label matches are anchored to the beginning and ending of lines so there is no concept of "contains a literal" for labels. This PR originally broke those semantics so some operations that were essentially no-ops now took a ton of time.

I've fixed the regex simplification to account for the label semantics and the benchmarks make way more sense:

name                                        old time/op  new time/op  delta
LineLabelFilters/foo-8                      52.9ns ± 9%  48.5ns ± 5%   -8.43%  (p=0.000 n=10+10)
LineLabelFilters/(foo)-8                    59.2ns ± 9%  51.1ns ±12%  -13.64%  (p=0.000 n=10+10)
LineLabelFilters/(foo|ba)-8                 63.6ns ± 2%  65.9ns ±10%     ~     (p=0.400 n=9+10)
LineLabelFilters/(foo|ba|ar)-8              71.7ns ± 5%  68.8ns ± 4%   -4.13%  (p=0.001 n=10+9)
LineLabelFilters/(foo|(ba|ar))-8            65.1ns ± 4%  70.1ns ± 9%   +7.66%  (p=0.000 n=9+10)
LineLabelFilters/foo.*-8                     900ns ± 7%    15ns ±11%  -98.33%  (p=0.000 n=10+10)
LineLabelFilters/.*foo.*-8                  5.15µs ± 3%  0.01µs ± 4%  -99.71%  (p=0.000 n=9+9)
LineLabelFilters/(.*)(foo).*-8              5.96µs ± 4%  0.02µs ± 8%  -99.74%  (p=0.000 n=10+10)
LineLabelFilters/(foo.*|.*ba)-8             5.69µs ±10%  0.02µs ± 8%  -99.67%  (p=0.000 n=10+10)
LineLabelFilters/(foo.*|.*bar.*)-8          6.60µs ± 2%  0.02µs ± 9%  -99.71%  (p=0.000 n=8+10)
LineLabelFilters/.*foo.*|bar-8              5.39µs ±13%  5.32µs ± 4%     ~     (p=0.529 n=10+8)
LineLabelFilters/.*foo|bar-8                3.65µs ± 9%  3.63µs ± 2%     ~     (p=0.234 n=10+9)
LineLabelFilters/(?:.*foo.*|bar)-8          5.37µs ± 8%  5.61µs ± 7%   +4.49%  (p=0.020 n=10+10)
LineLabelFilters/(?P<foo>.*foo.*|bar)-8     5.77µs ± 8%  5.51µs ± 7%     ~     (p=0.105 n=10+10)
LineLabelFilters/.*foo.*|bar|buzz-8         5.41µs ± 8%  0.02µs ±10%  -99.65%  (p=0.000 n=10+10)
LineLabelFilters/.*foo.*|bar|uzz-8          5.40µs ± 8%  5.43µs ± 2%     ~     (p=0.203 n=10+8)
LineLabelFilters/foo|bar|b|buzz|zz-8        66.8ns ± 2%  72.9ns ± 7%   +9.09%  (p=0.000 n=8+10)
LineLabelFilters/f|foo|foobar-8             78.3ns ± 2%  19.1ns ± 9%  -75.58%  (p=0.000 n=9+10)
LineLabelFilters/f.*|foobar.*|.*buzz-8      5.50µs ± 7%  0.02µs ±11%  -99.59%  (p=0.000 n=10+10)
LineLabelFilters/((f.*)|foobar.*)|.*buzz-8  6.18µs ± 7%  0.02µs ±10%  -99.64%  (p=0.000 n=10+10)
LineLabelFilters/.*-8                        930ns ± 8%     2ns ±16%  -99.76%  (p=0.000 n=9+10)
LineLabelFilters/.*|.*-8                    4.48µs ± 7%  0.00µs ± 2%  -99.95%  (p=0.000 n=10+8)
LineLabelFilters/.*||||-8                   2.99µs ± 3%  0.00µs ±15%  -99.93%  (p=0.000 n=10+10)
LineLabelFilters/#00-8                      48.0ns ± 9%   2.2ns ± 8%  -95.50%  (p=0.000 n=10+10)
LineLabelFilters/(?i)foo-8                  74.3ns ±16%  66.4ns ±10%  -10.68%  (p=0.006 n=10+10)
LineLabelFilters/(?i)界-8                    17.4ns ±13%  16.1ns ±22%     ~     (p=0.063 n=10+10)
LineLabelFilters/(?i)ïB-8                   24.2ns ±10%  23.6ns ± 4%     ~     (p=0.136 n=9+9)
LineLabelFilters/(?:)foo|fatal|exception-8   186ns ± 8%    71ns ±10%  -62.09%  (p=0.000 n=10+10)
LineLabelFilters/(?i)foo|fatal|exception-8  81.1ns ± 8%  82.4ns ± 8%     ~     (p=0.218 n=10+10)
LineLabelFilters/(?i)f|foo|foobar-8         88.3ns ± 3%  15.7ns ±13%  -82.22%  (p=0.000 n=8+10)
LineLabelFilters/(?i)f|fatal|e.*-8          66.8ns ± 8%  18.2ns ± 2%  -72.73%  (p=0.000 n=10+8)
LineLabelFilters/(?i).*foo.*-8              5.75µs ± 2%  0.01µs ± 8%  -99.76%  (p=0.000 n=8+10)

@MasslessParticle MasslessParticle marked this pull request as draft March 1, 2023 23:46
@MasslessParticle
Copy link
Contributor Author

Converting this to a draft as I work through the implications of the above comment :)

…gex goodness we have worked on in the line filters
- Account for '' special case
- Bug fix in the exists filter
@MasslessParticle MasslessParticle force-pushed the tpatterson/share-regex-optimizations branch from 5c99e5b to 94685f4 Compare March 2, 2023 00:15
@MasslessParticle MasslessParticle marked this pull request as ready for review March 2, 2023 00:15
@MasslessParticle
Copy link
Contributor Author

ok, I think this the last one.

The filters we generate for regexes for trees whose leaf nodes are either a contains filter or a regex filter. Because the label regex semantics anchor the beginning and end of the line to a literal, it's essentially an equal filter.

This last commit adds the equal filter in the case where we're making a label filter that's not a concat operation.

Note on benchmarks -- The benchmarks look great because everything reduces to a noop: most regexes reduce to some pile of equals filters for labels, the equality operator checks length first, and the vast majority of literals aren't of the same length as label we're looking for.

name                                        old time/op  new time/op  delta
LineLabelFilters/foo-8                      52.9ns ± 9%   8.6ns ±23%  -83.66%  (p=0.000 n=10+10)
LineLabelFilters/(foo)-8                    59.2ns ± 9%   7.9ns ± 2%  -86.57%  (p=0.000 n=10+8)
LineLabelFilters/(foo|ba)-8                 63.6ns ± 2%  13.5ns ± 7%  -78.72%  (p=0.000 n=9+10)
LineLabelFilters/(foo|ba|ar)-8              71.7ns ± 5%  19.1ns ± 9%  -73.36%  (p=0.000 n=10+10)
LineLabelFilters/(foo|(ba|ar))-8            65.1ns ± 4%  18.5ns ± 8%  -71.64%  (p=0.000 n=9+10)
LineLabelFilters/foo.*-8                     900ns ± 7%    16ns ±15%  -98.19%  (p=0.000 n=10+10)
LineLabelFilters/.*foo.*-8                  5.15µs ± 3%  0.01µs ±13%  -99.71%  (p=0.000 n=9+10)
LineLabelFilters/(.*)(foo).*-8              5.96µs ± 4%  0.02µs ±18%  -99.74%  (p=0.000 n=10+10)
LineLabelFilters/(foo.*|.*ba)-8             5.69µs ±10%  0.02µs ± 7%  -99.67%  (p=0.000 n=10+10)
LineLabelFilters/(foo.*|.*bar.*)-8          6.60µs ± 2%  0.02µs ±17%  -99.70%  (p=0.000 n=8+9)
LineLabelFilters/.*foo.*|bar-8              5.39µs ±13%  0.02µs ±10%  -99.64%  (p=0.000 n=10+10)
LineLabelFilters/.*foo|bar-8                3.65µs ± 9%  0.02µs ±12%  -99.46%  (p=0.000 n=10+10)
LineLabelFilters/(?:.*foo.*|bar)-8          5.37µs ± 8%  0.02µs ±13%  -99.63%  (p=0.000 n=10+10)
LineLabelFilters/(?P<foo>.*foo.*|bar)-8     5.77µs ± 8%  0.02µs ± 7%  -99.65%  (p=0.000 n=10+10)
LineLabelFilters/.*foo.*|bar|buzz-8         5.41µs ± 8%  0.02µs ±16%  -99.63%  (p=0.000 n=10+10)
LineLabelFilters/.*foo.*|bar|uzz-8          5.40µs ± 8%  0.02µs ±16%  -99.56%  (p=0.000 n=10+10)
LineLabelFilters/foo|bar|b|buzz|zz-8        66.8ns ± 2%  39.8ns ±12%  -40.52%  (p=0.000 n=8+10)
LineLabelFilters/f|foo|foobar-8             78.3ns ± 2%  18.8ns ±10%  -75.98%  (p=0.000 n=9+10)
LineLabelFilters/f.*|foobar.*|.*buzz-8      5.50µs ± 7%  0.02µs ±14%  -99.58%  (p=0.000 n=10+9)
LineLabelFilters/((f.*)|foobar.*)|.*buzz-8  6.18µs ± 7%  0.02µs ±15%  -99.62%  (p=0.000 n=10+10)
LineLabelFilters/.*-8                        930ns ± 8%     2ns ± 9%  -99.77%  (p=0.000 n=9+10)
LineLabelFilters/.*|.*-8                    4.48µs ± 7%  0.00µs ± 3%  -99.95%  (p=0.000 n=10+9)
LineLabelFilters/.*||||-8                   2.99µs ± 3%  0.00µs ±11%  -99.93%  (p=0.000 n=10+10)
LineLabelFilters/#00-8                      48.0ns ± 9%   2.2ns ± 8%  -95.47%  (p=0.000 n=10+10)
LineLabelFilters/(?i)foo-8                  74.3ns ±16%   8.3ns ± 9%  -88.84%  (p=0.000 n=10+10)
LineLabelFilters/(?i)界-8                    17.4ns ±13%   8.2ns ± 9%  -52.91%  (p=0.000 n=10+10)
LineLabelFilters/(?i)ïB-8                   24.2ns ±10%   8.3ns ± 9%  -65.65%  (p=0.000 n=9+10)
LineLabelFilters/(?:)foo|fatal|exception-8   186ns ± 8%    24ns ±10%  -87.34%  (p=0.000 n=10+9)
LineLabelFilters/(?i)foo|fatal|exception-8  81.1ns ± 8%  22.6ns ±13%  -72.15%  (p=0.000 n=10+10)
LineLabelFilters/(?i)f|foo|foobar-8         88.3ns ± 3%  15.6ns ±16%  -82.37%  (p=0.000 n=8+10)
LineLabelFilters/(?i)f|fatal|e.*-8          66.8ns ± 8%  19.7ns ±18%  -70.53%  (p=0.000 n=10+10)
LineLabelFilters/(?i).*foo.*-8              5.75µs ± 2%  0.02µs ±16%  -99.74%  (p=0.000 n=8+10)

@MasslessParticle MasslessParticle merged commit 7f42137 into main Mar 3, 2023
@MasslessParticle MasslessParticle deleted the tpatterson/share-regex-optimizations branch March 3, 2023 18:02
MasslessParticle added a commit that referenced this pull request Mar 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants