Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the group-by reduce handling when query times out #8450

Merged
merged 2 commits into from
Mar 31, 2022

Conversation

Jackie-Jiang
Copy link
Contributor

Fix #8449

When broker times out, there are multiple places not handled properly:

  • Should check the return value of CountDownLatch.await() to identify if the query times out
  • Future[] length should be the reduce group size
  • When the query times out, the result table won't be set, so reading rows from the result table can cause NPE

@codecov-commenter
Copy link

Codecov Report

Merging #8450 (4c7349c) into master (91c9ce4) will increase coverage by 6.49%.
The diff coverage is 86.95%.

@@             Coverage Diff              @@
##             master    #8450      +/-   ##
============================================
+ Coverage     64.21%   70.70%   +6.49%     
- Complexity     4282     4283       +1     
============================================
  Files          1618     1663      +45     
  Lines         85376    87287    +1911     
  Branches      13004    13210     +206     
============================================
+ Hits          54820    61716    +6896     
+ Misses        26576    21281    -5295     
- Partials       3980     4290     +310     
Flag Coverage Δ
integration1 27.25% <69.56%> (?)
integration2 25.95% <69.56%> (?)
unittests1 67.03% <73.91%> (-0.02%) ⬇️
unittests2 14.14% <0.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...not/core/query/reduce/GroupByDataTableReducer.java 90.55% <86.95%> (+4.03%) ⬆️
.../impl/dictionary/BaseOffHeapMutableDictionary.java 81.33% <0.00%> (-3.34%) ⬇️
.../aggregation/function/ModeAggregationFunction.java 86.48% <0.00%> (-2.17%) ⬇️
...org/apache/pinot/core/data/table/TableResizer.java 91.30% <0.00%> (-1.09%) ⬇️
...t/server/api/resources/DefaultExceptionMapper.java 0.00% <0.00%> (ø)
...g/apache/pinot/server/api/resources/ErrorInfo.java 0.00% <0.00%> (ø)
...ot/plugin/metrics/yammer/YammerMetricsFactory.java 100.00% <0.00%> (ø)
...ache/pinot/server/api/resources/ErrorResponse.java 0.00% <0.00%> (ø)
...rg/apache/pinot/server/starter/ServerInstance.java 88.99% <0.00%> (ø)
...gin/inputformat/avro/SimpleAvroMessageDecoder.java 0.00% <0.00%> (ø)
... and 371 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 91c9ce4...4c7349c. Read the comment docs.

@Jackie-Jiang Jackie-Jiang merged commit 8dcb255 into apache:master Mar 31, 2022
@Jackie-Jiang Jackie-Jiang deleted the fix_groupby_reducer branch March 31, 2022 22:52
}
long reduceTimeoutMs = 1;
BrokerResponseNative brokerResponse =
brokerReduceService.reduceOnDataTable(brokerRequest, brokerRequest, dataTableMap, reduceTimeoutMs, null);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

brokerRequest param is duplicated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. The first arg is the broker side broker request, second arg is the server side broker request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Issue with broker timeout handling results in ArrayOutOfBoundsException
4 participants