Allow missing intervals for Parallel task with hash/range partitioning #10592
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jihoonson
changed the title
|
No comment on the code itself, but very exciting to remove the extra requirement from the user! |
clintropolis
approved these changes
Nov 18, 2020
| final boolean needsInputSampling = | ||
| partitionsSpec.getNumShards() == null | ||
| || ingestionSchemaToUse.getDataSchema().getGranularitySpec().inputIntervals().isEmpty(); | ||
| if (needsInputSampling) { | ||
| // 0. need to determine numShards by scanning the data | ||
| LOG.info("numShards is unspecified, beginning %s phase.", PartialDimensionCardinalityTask.TYPE); |
There was a problem hiding this comment.
this log statement needs a change now
|
|
||
| LOG.info("Automatically determined numShards: " + numShardsOverride); |
There was a problem hiding this comment.
is there an equivalent info being logged somewhere now?
There was a problem hiding this comment.
No, I didn't add one for intervalToNumShards because it could have lots of intervals.
|
@abhishekagarwal87 do you have more comments? |
himanshug
approved these changes
Nov 23, 2020
@jihoonson LGTM |
|
@clintropolis @himanshug @abhishekagarwal87 thanks for the review 👍 |
3 tasks
JulianJaffePinterest
pushed a commit
to JulianJaffePinterest/druid
that referenced
this pull request
Jan 22, 2021
apache#10592) * Allow missing intervals for Parallel task * fix row filter * fix tests * fix log
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR allows Parallel task to run without explicit intervals in granularitySpec. If intervals are missing, the parallel task executes an extra step for input sampling which collects the intervals to index.
This PR additionally fixes a bug when
numShardsis missing in hash partitioning. WhennumShardsis missing, the parallel task computes it by scanning the whole input. However, the computed numShards was ignored when it's serialized into JSON. To fix it, this PR adds another fieldintervalToNumShardsOverridewhich stores the computed numShards per interval so that we can handle data skew well across intervals.This PR has: