SuperSorter: direct merging, increased parallelism. #16775
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Two performance enhancements: 1) Direct merging of input frames to output channels, without any temporary files, if all input frames fit in memory. 2) When doing multi-level merging (now called "external mode"), improve parallelism by boosting up the number of mergers in the penultimate level. To support direct merging, FrameChannelMerger is enhanced such that the output partition min/max values are used to filter input frames. This is necessary because all direct mergers read all input frames, but only rows corresponding to a single output partition.
![github-advanced-security[bot]](https://github.com/avatars/in/57789?s=60&v=4){kind=small}
processing/src/main/java/org/apache/druid/frame/processor/SuperSorter.java
Dismissed
Show dismissed
Hide dismissed
clintropolis
approved these changes
Aug 6, 2024
| } else { | ||
| return UNKNOWN_TOTAL; | ||
| } | ||
| } else if (level > 0 && level == totalMergingLevels - 2) { |
There was a problem hiding this comment.
nit: i wonder if it might be a bit easier to follow if we moved some of these penultimate and ultimate level checks that do math stuff into named functions to be like level == penultimateLevel(totalMergingLevels) or isPenultimateLevel(level) or something since i see them repeated in places, or i guess could also maybe precompute?
There was a problem hiding this comment.
isPenultimateLevel might make sense; I think it'd have to return a tri-state since the answer might be unknown. I/we can think about it in a future patch— simplifying the logic of SuperSorter would be great.
gianm
added a commit
to gianm/druid
that referenced
this pull request
Aug 20, 2024
Without the call to readOnly, each output channel retains a 1 MB allocator, leading to excessive memory use. Fixes regression from apache#16775.
gianm
added a commit
that referenced
this pull request
Aug 21, 2024
Without the call to readOnly, each output channel retains a 1 MB allocator, leading to excessive memory use. Fixes regression from #16775.
hevansDev
pushed a commit
to hevansDev/druid
that referenced
this pull request
Aug 29, 2024
Without the call to readOnly, each output channel retains a 1 MB allocator, leading to excessive memory use. Fixes regression from apache#16775.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Two performance enhancements:
Direct merging of input frames to output channels, without any
temporary files, if all input frames fit in memory.
When doing multi-level merging (now called "external mode"),
improve parallelism by boosting up the number of mergers in the
penultimate level.
To support direct merging, FrameChannelMerger is enhanced such that the output partition min/max values are used to filter input frames. This is necessary because all direct mergers read all input frames, but only rows corresponding to a single output partition.