BoundFilter optimizations, and related interface changes. #2727
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
benchmarks: |
gianm
force-pushed
the
optimize-bound-filter
branch
from
8207545
to
a7e4b1b
Compare
|
@gianm should we change default to use roaring as it seems to be much better in most cases |
| } | ||
| } | ||
| ); | ||
| } | ||
|
|
||
| private boolean doesMatch(String input) | ||
| { |
There was a problem hiding this comment.
is this logic just copypasta of what was there before?
There was a problem hiding this comment.
mostly yes, except for the null handling.
|
👍 |
gianm
force-pushed
the
optimize-bound-filter
branch
from
a7e4b1b
to
77fed15
Compare
|
Some of the new tests depend on #2737 to work. |
BoundFilter: - For lexicographic bounds, use bitmapIndex.getIndex to find the start and end points, then union all bitmaps between those points. - For alphanumeric bounds, iterate through dimValues, and union all bitmaps for values matching the predicate. - Change behavior for nulls: it used to be that the BoundFilter would never match nulls, now it matches nulls if "" is allowed by the lower limit and not excluded by the upper limit. Interface changes: - BitmapIndex: add `int getIndex(value)` to make it possible to get the index for a value without retrieving the bitmap. - BitmapIndex: remove `ImmutableBitmap getBitmap(value)`, change callers to `getBitmap(getIndex(value))`. - BitmapIndexSelector: allow retrieving the underlying BitmapIndex through getBitmapIndex. - Clarified contract of indexOf in Indexed, GenericIndexed. Also added tests for SelectorFilter, NotFilter, and BoundFilter.
gianm
force-pushed
the
optimize-bound-filter
branch
from
77fed15
to
2970b49
Compare
| @@ -111,6 +112,17 @@ public BitmapFactory getBitmapFactory() | |||
| } | |||
|
|
|||
| @Override | |||
| public BitmapIndex getBitmapIndex(String dimension) | |||
There was a problem hiding this comment.
Should public BitmapIndex getBitmapIndex(String dimension) and public ImmutableBitmap getBitmapIndex(String dimension, String value) be named differently? Maybe slightly confusing to have functions with a different return type with the same name
There was a problem hiding this comment.
I guess I didn't want to touch too many files in this patch (all of the other filters), but I also didn't want to use a different name for this method, since it's the one that really should be called getBitmapIndex.
I am indifferent though & happy to go with whatever people think is best
There was a problem hiding this comment.
ok, I'm fine with this PR as-is, that rename if desired could be handled in later patch
|
👍 after travis |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
BoundFilter:
then union all bitmaps between those points.
matching the predicate.
now it matches nulls if "" is allowed by the lower limit and not excluded by the
upper limit.
Interface changes:
int getIndex(value)to make it possible to get the index for avalue without retrieving the bitmap.
ImmutableBitmap getBitmap(value), change callers togetBitmap(getIndex(value)).Also added tests for SelectorFilter, NotFilter, and BoundFilter.