Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug preventing numeric casts for MV columns in filters using the multi-stage query engine #13425

Merged
merged 2 commits into from
Jun 22, 2024

Conversation

yashmayya
Copy link
Collaborator

  • Queries like SELECT * FROM mytable WHERE ARRAY_TO_MV(CAST(mv_col AS BIGINT ARRAY)) = 0; fail when run with the multi-stage query engine with an error like:
Unable to execute query plan for request: 11345490000000000 on server: 192.168.29.25@{61393,61394}, ERROR: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Caught exception while submitting request: 11345490000000000, stage: 1
...
...
Caused by: java.lang.RuntimeException: Caught exception while submitting request: 11345490000000000, stage: 1
...
...
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No enum constant org.apache.pinot.spi.data.FieldSpec.DataType.LONG_ARRAY
...
	at org.apache.pinot.query.service.server.QueryServer.lambda$submit$1(QueryServer.java:141)
...

Caused by: java.lang.IllegalArgumentException: No enum constant org.apache.pinot.spi.data.FieldSpec.DataType.LONG_ARRAY
	at java.base/java.lang.Enum.valueOf(Enum.java:273)
	at org.apache.pinot.spi.data.FieldSpec$DataType.valueOf(FieldSpec.java:474)
	at org.apache.pinot.core.query.optimizer.filter.NumericalFilterOptimizer.getDataType(NumericalFilterOptimizer.java:403)
	at org.apache.pinot.core.query.optimizer.filter.NumericalFilterOptimizer.optimizeChild(NumericalFilterOptimizer.java:88)
org.apache.pinot.query.service.dispatch.QueryDispatcher.submit(QueryDispatcher.java:198)
org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:95)
org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:216)
org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:133)
  • That's because the NumericalFilterOptimizer wasn't handling the ARRAY Calcite types that represent MV fields in the v2 query engine.
  • Since MV columns aren't actual arrays and can be queried directly just like SV columns (where rows are matched if any value in the MV field matches the predicate) of the corresponding type, this bug can be fixed by simply stripping the _ARRAY suffix from the target type.

@yashmayya yashmayya added bugfix multi-stage Related to the multi-stage query engine labels Jun 18, 2024
@yashmayya yashmayya requested review from Jackie-Jiang and gortiz and removed request for Jackie-Jiang June 18, 2024 13:27
@yashmayya yashmayya changed the title Fix bug preventing numeric casts for multi-value columns in the multi-stage query engine Fix bug preventing numeric casts for MV columns in filters using the multi-stage query engine Jun 18, 2024
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Jackie-Jiang Jackie-Jiang merged commit 5760d10 into apache:master Jun 22, 2024
20 checks passed
suyashpatel98 pushed a commit to suyashpatel98/pinot that referenced this pull request Jul 6, 2024
@npawar npawar added the v1v2 label Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix multi-stage Related to the multi-stage query engine v1v2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants