perf: coalesce ids before executing take #2680
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eddyxu
reviewed
Aug 2, 2024
| @@ -1584,9 +1585,10 @@ impl Scanner { | |||
| projection: &Schema, | |||
| batch_readahead: usize, | |||
| ) -> Result<Arc<dyn ExecutionPlan>> { | |||
| let coalesced = Arc::new(CoalesceBatchesExec::new(input, self.get_batch_size())); | |||
There was a problem hiding this comment.
Do we want to sort row IDs to offer a better chance we can do sequential reads?
There was a problem hiding this comment.
We already do that internally.
wjones127
approved these changes
Aug 2, 2024
westonpace
force-pushed
the
perf/coalesce-take-ids
branch
from
fdd4562
to
e69f0fd
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2680 +/- ##
==========================================
+ Coverage 79.32% 79.34% +0.01%
==========================================
Files 226 226
Lines 66872 66886 +14
Branches 66872 66886 +14
==========================================
+ Hits 53049 53069 +20
- Misses 10720 10724 +4
+ Partials 3103 3093 -10
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
…the number of times we call take which has a number of performance benefits.
westonpace
force-pushed
the
perf/coalesce-take-ids
branch
from
e69f0fd
to
08cd611
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Late materialization is a great benefit when executing a highly selective filter. However, if a filter is highly selective it means that each input batch will probably only have a few matching rows. The current implementation executes take for each filtered batch. E.g. instead of a single call of
take(500, 10000, 300000)we get three callstake(500),take(10000), andtake(300000). This means:On cloud storage I see a 10x plus benefit in scan performance.
We have a benchmark for this (EDA search plot 4) which should assist with preventing regression in the future: https://bencher.dev/console/projects/weston-lancedb/plots