add equality, null, and range filter

[clintropolis]

Description

This PR introduces some new filter types which aim to be both more SQL compliant and also work better with various types by accepting match values in types other than String. The new filters are EqualityFilter, NullFilter, and RangeFilter, which serve as replacements for SelectorDimFilter and BoundDimFilter. Effectively SelectorDimFilter is being split into NullFilter and EqualityFilter, and BoundDimFilter is superceded by RangeFilter.

Probably the most notable change in behavior is that the EqualityFilter and RangeFilter will never match null values.

Compare the current behavior of the bound filter:

to the behavior of the new range filter:

Use of these new filters is currently tied by default to the value of SQL compatibility mode, druid.generic.useDefaultValueForNull=false, but is controlled separately through a planner/query context parameter sqlUseBoundAndSelectors, which when true will use the classic filters, but when false will plan into the new filters.

Because the new filters accept all types rather than just strings, it also means that we can now support filtering equality and ranges on ARRAY columns

The null filter also supports proper object predicate matching, so IS NULL/IS NOT NULL work correctly on COMPLEX<json> columns (and any other complex type):

Equality filter

A lot like the SelectorDimFilter, but it will never match null. It also implements array matching, so it can be used for array equality as well.

parameters	description	required
type	must be "equals"	yes
column	column (or virtual column) name to filter for null values	yes
matchValue	Value to match, must not be null	yes
matchValueType	type of the match value, e.g. "STRING", "LONG", "FLOAT", "DOUBLE", "ARRAY<STRING>", "ARRAY<LONG>", etc.	yes
filterTuning	optional filter tuning to control if filtering will use bitmap indexes	no

I did add special handling for COMPLEX<json>, because it is more of an unofficial built-in type than a standard complex, but I can imagine in the future we allow complex types to opt into providing a customized equality check so that they could be supported here as well, but currently at least anything that can be checked with Objects.equals should work correctly.

I haven't really added any tests for json matching since i just snuck it in on the side, so it is totally possible there might be cases where Objects.equals doesn't match things that are equivalent otherwise.

Null filter

The null filter is a dedicated filter for matching NULL, and when enabled in the planner will be used for all IS NULL and (combined with a NotDimFilter) IS NOT NULL filtering, as well as any implicit null/not null. It is separated from the EqualityFilter for future work of making it easier to fix some additional SQL compliance bugs involving filter negation, and is more consistent with SQL behavior.

parameters	description	required
type	must be "null"	yes
column	column (or virtual column) name to filter for null values	yes
filterTuning	optional filter tuning to control if filtering will use bitmap indexes	no

Range filter

A lot like the bound filter, but also never matches null. Instead of accepting a comparator argument, it instead takes a matchValueType, which implies a certain comparator. For range filters on string columns, if the match value type is numeric we use the numeric string comparator, otherwise the lexicographic. For range filters on numeric and array columns, the match value is cast to the columns numeric processor type

parameters	description	required
type	must be "range"	yes
column	column (or virtual column) name to filter for null values	yes
matchValueType	type of the match value, e.g. "STRING", "LONG", "FLOAT", "DOUBLE", etc.	yes
lower	lower bound value, or null if there is no lower bound	no (however, at least one of lower or upper must be specified)
upper	upper bound value, or null if there is no upper bound	no (however, at least one of lower or upper must be specified)
lowerOpen	if the value must be strictly greater than the lower bound, or if it may also be equal	no, defaults to false
upperOpen	if the value must be strictly less than the upper bound, or it may also be equal	no, defaults to false
filterTuning	optional filter tuning to control if filtering will use bitmap indexes	no

Other changes

• Removed isFilterable from ColumnCapabilities. All types of columns should have the chance to participate in filtering without having to set this flag. This mechanism was wired into a short-circuit that would always use a 'nil' column selector when filtering on columns whose isFilterable was set to false.
• Fixed compaction for ARRAY typed columns (by side effect sort of from making column processor factory interface have array handling), added test

Release note

---

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[imply-cheddar]

I've looked through the updates. While there are still things that I would change in here, the only thing that I feel strongly about that I'd kinda rather not be committed if it doesn't have to be is the inclusion of ColumnCapabilities where the TypeStrategy is enough.

Specifically, there were various places where it has a good type strategy, but has to create a ColumnCapabilities object in order to pass it through, the column capabilities that it creates doesn't indicate that it has any indexes or anything, which might be correct and might not, but the API is asking the code to do extra stuff than what is actually needed and I worry about either

1. The current implementation continuing, us coming up with a reason why we want to use the other methods and then be sad that you cannot use the methods because various code paths just create fake objects
2. We never need more than just the type and end up with a proliferation of ColumnCapabilitiesImpl.createDefault().setType() existing where all that was needed was the type.

I'd really rather that we fix/minimize that API to begin with.

I see that you removed the dummy instances, I overlooked that...