Upstream: fe0f4150c9a0156a2d95034fd8f626dfc552b34b #509
Conversation
apache#17380) - This is a non-functional change that moves SqlTaskStatus and its unit test SqlTaskStatusTest from the msq module to the sql module to help class reuse in other places. - This refactor is extracted from this PR to facilitate easier review. - Fix a minor spacing issue in the TaskStartTimeoutFault error message.
…pache#17344) MSQ currently supports only single-valued string dimensions as partition keys. This patch adds a check to ensure that partition keys are single-valued in case this info is available by virtue of segment download for schema inference. During compaction, if MSQ finds multi-valued dimensions (MVDs) declared as part of `range` partitionsSpec, it switches partitioning type to dynamic, ending up in repeated compactions of the same interval. To avoid this scenario, the segment download logic is also updated to always download segments if info on multi-valued dimensions is required.
This patch is extracted from PR 17353. Changes: - Added BrokerClient and BrokerClientImpl to the sql package that leverages the ServiceClient functionality; similar to OverlordClient and CoordinatorClient implementations in the server module. - For now, only two broker API stubs are added: submitSqlTask() and fetchExplainPlan(). - Added a new POJO class ExplainPlan that encapsulates explain plan info. - Deprecated org.apache.druid.discovery.BrokerClient in favor of the new BrokerClient in this patch. - Clean up ExplainAttributesTest a bit and added serde verification.
…tesLimit (apache#17384) * Remove unused coordinator dynamic configs * Update docs and web-console
…d standardize parameters (apache#17383) changes: * adds `SqlBenchmarkDatasets` which contains commonly used benchmark data generator schemas * adds `SqlBaseBenchmark` which contains common benchmark segment generation methods for any benchmark using `SqlBenchmarkDatasets` * adds `SqlBaseQueryBenchmark` and `SqlBasePlanBenchmark` for benchmarks measuring queries and planning respectively * migrate all existing SQL jmh benchmarks to extend `SqlBaseQueryBenchmark`, quite dramatically reducing the boilerplate needed to create benchmarks, and allowing the use of multiple datasources within a benchmark file * adjustments to data generator stuff to allow passing in an ObjectMapper so that the same mapper can be used for both benchmark queries and segment generation, avoiding the need to register stuff with both mappers for benchmarks * adds `SqlProjectionsBenchmark` and `SqlComplexMetricsColumnsBenchmark` for measuring projections and measuring complex metric compression respectively
…pache#17394) * SeekableStreamSupervisor: Use workerExec as the client connectExec. This patch uses the already-existing per-supervisor workerExec as the connectExec for task clients, rather than using the process-wide default ServiceClientFactory pool. This helps prevent callbacks from backlogging on the process-wide pool. It's especially useful for retries, where callbacks may need to establish new TCP connections or perform TLS handshakes. * Fix compilation, tests. * Fix style.
…utput channel in runIncrementally() (apache#17373) WindowOperatorQueryFrameProcessor: Avoid writing multiple frames to output channel in runIncrementally()
Calling toString on newConfig is unnecessary, because it will be done automatically by the logger. This saves some effort under log levels higher than DEBUG.
…ache#17403) Following apache#17394, workerExec can get deadlocked with itself, because it waits for task futures and is also used as the connectExec for the task client. To fix this, we need to never await task futures in the workerExec. There are two specific changes: in "verifyAndMergeCheckpoints" and "checkpointTaskGroup", two "coalesceAndAwait" calls that formerly occurred in workerExec are replaced with Futures.transform (using a callback in workerExec). Because this adjustment removes a source of blocking, it may also improve supervisor responsiveness for high task counts. This is not the primary goal, however. The primary goal is to fix the bug introduced by apache#17394.
…mp column (apache#17411) This PR improves the error message during MSQ ingestion if we encounter a non-long timestamp column.
![github-advanced-security[bot]](https://github.com/avatars/in/57789?s=60&v=4){kind=small}
| final Object[] lastRow = resultSequence.accumulate(null, (accumulated, in) -> in); | ||
| blackhole.consume(lastRow); | ||
| } | ||
| return QUERIES.get(Integer.parseInt(query)); |
Check notice
Code scanning / CodeQL
Missing catch of NumberFormatException Note test
| @Override | ||
| public String getQuery() | ||
| { | ||
| return QUERIES.get(Integer.parseInt(query)); |
Check notice
Code scanning / CodeQL
Missing catch of NumberFormatException Note test
| catch (Throwable ex) { | ||
| log.warn(ex, "failed to count rows"); | ||
| } | ||
| return QUERIES.get(Integer.parseInt(query)); |
Check notice
Code scanning / CodeQL
Missing catch of NumberFormatException Note test
| { | ||
| closer.close(); | ||
| return QUERIES.get(Integer.parseInt(query)); |
Check notice
Code scanning / CodeQL
Missing catch of NumberFormatException Note test
| @Override | ||
| public String getQuery() | ||
| { | ||
| return SqlBenchmark.QUERIES.get(Integer.parseInt(query)); |
Check notice
Code scanning / CodeQL
Missing catch of NumberFormatException Note test
| @Override | ||
| public String getQuery() | ||
| { | ||
| return QUERIES.get(Integer.parseInt(query)); |
Check notice
Code scanning / CodeQL
Missing catch of NumberFormatException Note test
| catch (Exception e) { | ||
| throw new RuntimeException(e); | ||
| } | ||
| return QUERIES.get(Integer.parseInt(query)); |
Check notice
Code scanning / CodeQL
Missing catch of NumberFormatException Note test
No description provided.