Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance Broker reducer to handle expression format change #11762

Merged

Conversation

Jackie-Jiang
Copy link
Contributor

Enhance broker reducer to not rely on the column names returned from the servers. This is required to support future expression format change (e.g. do not include single quotes around numeric literals).

  • Drop incompatible data tables for all query types to prevent type incompatible exceptions
  • Remove special handling of distinct data table format introduced before releasing 0.12

@codecov-commenter
Copy link

codecov-commenter commented Oct 9, 2023

Codecov Report

Merging #11762 (3c0ef56) into master (52d16f7) will increase coverage by 0.01%.
Report is 1 commits behind head on master.
The diff coverage is 85.20%.

@@             Coverage Diff              @@
##             master   #11762      +/-   ##
============================================
+ Coverage     63.12%   63.14%   +0.01%     
- Complexity     1118     1120       +2     
============================================
  Files          2342     2343       +1     
  Lines        125917   125976      +59     
  Branches      19370    19388      +18     
============================================
+ Hits          79487    79542      +55     
- Misses        40770    40788      +18     
+ Partials       5660     5646      -14     
Flag Coverage Δ
integration <0.01% <0.00%> (ø)
integration1 <0.01% <0.00%> (ø)
integration2 0.00% <0.00%> (ø)
java-11 63.07% <85.20%> (-0.02%) ⬇️
java-17 62.98% <85.20%> (+48.56%) ⬆️
java-20 62.99% <85.20%> (+0.03%) ⬆️
temurin 63.14% <85.20%> (+0.01%) ⬆️
unittests 63.13% <85.20%> (+0.01%) ⬆️
unittests1 67.29% <85.20%> (+0.03%) ⬆️
unittests2 14.41% <0.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...not/core/query/reduce/GroupByDataTableReducer.java 67.65% <100.00%> (+0.42%) ⬆️
...core/query/selection/SelectionOperatorService.java 95.65% <100.00%> (+1.90%) ⬆️
...t/core/query/reduce/SelectionDataTableReducer.java 95.00% <92.30%> (+27.50%) ⬆️
...t/core/query/selection/SelectionOperatorUtils.java 92.76% <98.71%> (+3.59%) ⬆️
...ot/core/query/reduce/DistinctDataTableReducer.java 96.77% <94.73%> (+9.08%) ⬆️
...e/operator/LeafStageTransferableBlockOperator.java 87.28% <86.66%> (-0.38%) ⬇️
...core/query/reduce/AggregationDataTableReducer.java 68.91% <81.81%> (+1.06%) ⬆️
...inot/core/query/reduce/ReducerDataSchemaUtils.java 87.17% <87.17%> (ø)
...e/pinot/core/query/reduce/BrokerReduceService.java 70.76% <47.61%> (-13.24%) ⬇️
...re/query/reduce/SelectionOnlyStreamingReducer.java 0.00% <0.00%> (ø)

... and 20 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Jackie-Jiang Jackie-Jiang force-pushed the broker_column_name_change_handling branch from 2214fb8 to 3c0ef56 Compare October 9, 2023 18:55
Copy link
Contributor

@walterddr walterddr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm


AggregationDataTableReducer(QueryContext queryContext) {
public AggregationDataTableReducer(QueryContext queryContext) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this and below: is public access needed? (not seeing any changes in how contructors are used

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically it is not required to be public, but it doesn't hurt since we don't really have a reason to prevent it from being reused

// NOTE: When there is no cached data schema, that means all servers encountered exception. In such case, return the
// response with metadata only.
DataSchema cachedDataSchema =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) notes are for line 139?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is about cached data schema, so I feel it is fine putting it here

@Jackie-Jiang Jackie-Jiang merged commit ad0d2b1 into apache:master Oct 12, 2023
19 checks passed
@Jackie-Jiang Jackie-Jiang deleted the broker_column_name_change_handling branch October 12, 2023 01:21
@mosabua mosabua mentioned this pull request Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants