Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[multistage][bugfix] fix operator eos pull #11970

Merged
merged 5 commits into from
Nov 9, 2023

Conversation

walterddr
Copy link
Contributor

@walterddr walterddr commented Nov 8, 2023

currently agg, window, transform, fitler operator can potentially pull more than once against input for EOS.

This PR changes the behavior.

  • single-input stop-the-world operator (agg, window, sort) only pull once and check for produced block first
  • streaming operator (filter, transform) keeps the behavior
  • dual-input, right-sided stop-the-world & left-sided stream operator (join, set) will only pull EOS on each input at most once
    • when right side error out first, left side will not be pulled

this guarantees that no more than one EOS block will be pulled; tests are covered by existing resource queries tests

@codecov-commenter
Copy link

codecov-commenter commented Nov 8, 2023

Codecov Report

Merging #11970 (107ba72) into master (45f1869) will decrease coverage by 33.84%.
Report is 8 commits behind head on master.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master   #11970       +/-   ##
=============================================
- Coverage     61.43%   27.60%   -33.84%     
+ Complexity     1135      201      -934     
=============================================
  Files          2385     2385               
  Lines        129150   129179       +29     
  Branches      19994    19998        +4     
=============================================
- Hits          79347    35657    -43690     
- Misses        44049    90837    +46788     
+ Partials       5754     2685     -3069     
Flag Coverage Δ
custom-integration1 <0.01% <0.00%> (ø)
integration <0.01% <0.00%> (ø)
integration1 <0.01% <0.00%> (ø)
integration2 0.00% <0.00%> (ø)
java-11 <0.01% <0.00%> (-61.37%) ⬇️
java-21 27.60% <0.00%> (-33.72%) ⬇️
skip-bytebuffers-false 27.60% <0.00%> (-33.81%) ⬇️
skip-bytebuffers-true 27.58% <0.00%> (-33.71%) ⬇️
temurin 27.60% <0.00%> (-33.84%) ⬇️
unittests 27.59% <0.00%> (-33.84%) ⬇️
unittests1 ?
unittests2 27.59% <0.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...he/pinot/query/runtime/operator/OperatorStats.java 0.00% <ø> (-93.34%) ⬇️
...inot/query/runtime/operator/TransformOperator.java 0.00% <0.00%> (-100.00%) ⬇️
...uery/runtime/operator/WindowAggregateOperator.java 0.00% <0.00%> (-96.00%) ⬇️
...inot/query/runtime/operator/AggregateOperator.java 0.00% <0.00%> (-84.81%) ⬇️
...pinot/query/runtime/operator/HashJoinOperator.java 0.00% <0.00%> (-94.02%) ⬇️
...not/query/runtime/operator/MultiStageOperator.java 0.00% <0.00%> (-85.37%) ⬇️
...ache/pinot/query/runtime/operator/SetOperator.java 0.00% <0.00%> (-73.69%) ⬇️
...e/operator/LeafStageTransferableBlockOperator.java 0.00% <0.00%> (-85.00%) ⬇️
...che/pinot/query/runtime/operator/SortOperator.java 0.00% <0.00%> (-93.75%) ⬇️

... and 1165 files with indirect coverage changes

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

@@ -132,25 +132,22 @@ public String toExplainString() {
@Override
protected TransferableBlock getNextBlock() {
try {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we always need to do this try-catch? Consider moving it into base class?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes i can move it to base

// construct a SET with all the right side rows.
constructRightBlockSet();
}
if (_upstreamErrorBlock != null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably don't need to save this as member variable. Also, the error from right table is not gathered

Copy link
Contributor Author

@walterddr walterddr Nov 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah we should've :-P the issue here is similar to JOIN, fixed

Copy link
Contributor

@Jackie-Jiang Jackie-Jiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -70,7 +79,7 @@ public String getOperatorId() {
}

// Make it protected because we should always call nextBlock()
protected abstract TransferableBlock getNextBlock();
protected abstract TransferableBlock getNextBlock() throws Exception;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(minor) Reformat

@walterddr walterddr merged commit 73d82ec into apache:master Nov 9, 2023
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix multi-stage Related to the multi-stage query engine
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants