Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add query option override for Broker MinGroupTrimSize #11984

Merged
merged 2 commits into from
Nov 16, 2023

Conversation

vvivekiyer
Copy link
Contributor

  1. Added a queryOption override minBrokerGroupTrimSize for config pinot.broker.min.group.trim.size
  2. Reused the generic queryOption override groupTrimThreshold at the broker reduce stage as well

Note: For existing users of groupTrimThreshold queryOption, they can see trimming kicking in going forward. I'm open to suggestions if we want to decouple and create a separate queryOption for broker.

@codecov-commenter
Copy link

codecov-commenter commented Nov 10, 2023

Codecov Report

Attention: 10 lines in your changes are missing coverage. Please review.

Comparison is base (2beb9a4) 61.61% compared to head (ffccff7) 61.61%.
Report is 11 commits behind head on master.

Files Patch % Lines
...inot/core/query/reduce/StreamingReduceService.java 0.00% 7 Missing ⚠️
...e/pinot/core/query/reduce/BrokerReduceService.java 71.42% 0 Missing and 2 partials ⚠️
...e/pinot/common/utils/config/QueryOptionsUtils.java 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             master   #11984    +/-   ##
==========================================
  Coverage     61.61%   61.61%            
- Complexity      207     1150   +943     
==========================================
  Files          2385     2385            
  Lines        129214   129266    +52     
  Branches      20003    20014    +11     
==========================================
+ Hits          79613    79649    +36     
- Misses        43801    43815    +14     
- Partials       5800     5802     +2     
Flag Coverage Δ
custom-integration1 <0.01% <0.00%> (ø)
integration <0.01% <0.00%> (ø)
integration1 <0.01% <0.00%> (ø)
integration2 0.00% <0.00%> (ø)
java-11 61.58% <41.17%> (-0.01%) ⬇️
java-21 61.49% <41.17%> (+0.01%) ⬆️
skip-bytebuffers-false 61.60% <41.17%> (-0.01%) ⬇️
skip-bytebuffers-true 34.90% <41.17%> (+7.32%) ⬆️
temurin 61.61% <41.17%> (+<0.01%) ⬆️
unittests 61.61% <41.17%> (+<0.01%) ⬆️
unittests1 46.96% <41.17%> (+0.02%) ⬆️
unittests2 27.57% <0.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Jackie-Jiang Jackie-Jiang added enhancement Configuration Config changes (addition/deletion/change in behavior) labels Nov 14, 2023
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM otherwise

We may keep the same group trim threshold the same across the query lifecycle. Ideally we should also use the same group trim size everywhere, but we kept them separate for backward compatibility where segment level trim is off by default. We may consider unifying them in the future

Comment on lines 178 to 180
if (queryOptions == null) {
return null;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's follow the same convention by assuming queryOptions is not null

Suggested change
if (queryOptions == null) {
return null;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved this check into (Streaming/Broker)ReduceService. This check is needed because unlike servers (where queryOption cannot be null because of timeout,etc), it can be null for brokers.

@vvivekiyer
Copy link
Contributor Author

I'll update docs once this PR is merged.

@Jackie-Jiang Jackie-Jiang merged commit 1ab9e62 into apache:master Nov 16, 2023
18 of 19 checks passed
@vvivekiyer
Copy link
Contributor Author

vvivekiyer commented Nov 16, 2023

Added query override option to docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Configuration Config changes (addition/deletion/change in behavior) enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants