MSQ: Allow for worker gaps. #17277
Merged
MSQ: Allow for worker gaps. #17277
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In a Dart query, all Historicals are given worker IDs, but not all of them are going to actually be started or receive work orders. This can create gaps in the set of workers. For example, workers 1 and 3 could have work assigned while workers 0 and 2 do not. This patch updates ControllerStageTracker and WorkerInputs to handle such gaps, by using the set of actual worker numbers, rather than 0..workerCount, in various places.
github-actions
bot
added
Area - Batch Ingestion
Area - MSQ
LakshSingla
approved these changes
Oct 8, 2024
There was a problem hiding this comment.
minor nit: Naming
Per my understanding, sparse means that there are pieces missing from the middle. Something like 2..10 would not be sparse, however, would be created as a "sparse" striped partitions. Maybe something like SpecificStripedReadablePartitions which clarifies that the workers are explicitly specified? This is not a blocker.
cryptoe
pushed a commit
to cryptoe/druid
that referenced
this pull request
Oct 8, 2024
In a Dart query, all Historicals are given worker IDs, but not all of them are going to actually be started or receive work orders. This can create gaps in the set of workers. For example, workers 1 and 3 could have work assigned while workers 0 and 2 do not. This patch updates ControllerStageTracker and WorkerInputs to handle such gaps, by using the set of actual worker numbers, rather than 0..workerCount, in various places. (cherry picked from commit 06bbdb3)
kfaraz
pushed a commit
that referenced
this pull request
Oct 8, 2024
…7282) (#17283) (#17277) (#17285) * MSQ: Allow for worker gaps. (#17277) * DartSqlResource: Sort queries by start time. (#17282) * DartSqlResource: Add controllerHost to GetQueriesResponse. (#17283) * DartWorkerModule: Replace en dash with regular dash. (#17281) * DartSqlResource: Return HTTP 202 on cancellation even if no such query. (#17278) * Upgraded Protobuf to 3.25.5 (#17249) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 7d9e6d3) --------- Co-authored-by: Gian Merlino <gianmerlino@gmail.com> Co-authored-by: Shivam Garg <shigarg@visa.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In a Dart query, all Historicals are given worker IDs, but not all of them are going to actually be started or receive work orders. This can create gaps in the set of workers. For example, workers 1 and 3 could have work assigned while workers 0 and 2 do not.
This patch updates ControllerStageTracker and WorkerInputs to handle such gaps, by using the set of actual worker numbers, rather than 0..workerCount, in various places.