-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update default values in CoordinatorDynamicConfig #14269
Update default values in CoordinatorDynamicConfig #14269
Conversation
Changes may be needed in |
docs/configuration/index.md
Outdated
|`balancerComputeThreads`|Thread pool size for computing moving cost of segments in segment balancing. Consider increasing this if you have a lot of segments and moving segments starts to get stuck.|1| | ||
|`emitBalancingStats`|Boolean flag for whether or not we should emit balancing stats. This is an expensive operation.|false| | ||
|`killDataSourceWhitelist`|List of specific data sources for which kill tasks are sent if property `druid.coordinator.kill.on` is true. This can be a list of comma-separated data source names or a JSON array.|none| | ||
|`killPendingSegmentsSkipList`|List of data sources for which pendingSegments are _NOT_ cleaned up if property `druid.coordinator.kill.pendingSegments.on` is true. This can be a list of comma-separated data sources or a JSON array.|none| | ||
|`maxSegmentsInNodeLoadingQueue`|The maximum number of segments that could be queued for loading to any given server. This parameter could be used to speed up segments loading process, especially if there are "slow" nodes in the cluster (with low loading speed) or if too much segments scheduled to be replicated to some particular node (faster loading could be preferred to better segments distribution). Desired value depends on segments loading speed, acceptable replication time and number of nodes. Value 1000 could be a start point for a rather big cluster. Default value is 100. |100| | ||
|`maxSegmentsInNodeLoadingQueue`|The maximum number of segments that could be queued for loading to any given server. This parameter could be used to speed up segments loading process, especially if there are "slow" nodes in the cluster (with low loading speed) or if too much segments scheduled to be replicated to some particular node (faster loading could be preferred to better segments distribution). Desired value depends on segments loading speed, acceptable replication time and number of nodes. Value 1000 could be a start point for a rather big cluster. |500| |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|`maxSegmentsInNodeLoadingQueue`|The maximum number of segments that could be queued for loading to any given server. This parameter could be used to speed up segments loading process, especially if there are "slow" nodes in the cluster (with low loading speed) or if too much segments scheduled to be replicated to some particular node (faster loading could be preferred to better segments distribution). Desired value depends on segments loading speed, acceptable replication time and number of nodes. Value 1000 could be a start point for a rather big cluster. |500| | |
|`maxSegmentsInNodeLoadingQueue`|The maximum number of segments allowed in a loading queue for any given server. Use this parameter to load the segments faster—for example, if the cluster contains slow-loading nodes, or if there are too many segments to be replicated to a particular node (when faster loading is preferred to better segments distribution). Desired value depends on the loading speed of segments, acceptable replication time, and number of nodes. Value 1000 is a good starting point for a big cluster. |500| |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some suggestions to improve readability.
Thanks for the review, @ektravel ! I have incorporated your feedback. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good from the docs standpoint.
Changes
The defaults of the following config values in the
CoordinatorDynamicConfig
are being updated.1.
maxSegmentsInNodeLoadingQueue = 500
(previous = 100)Rationale: With round-robin segment assignment now being the default assignment technique, the Coordinator can assign a large number of under-replicated/unavailable segments very quickly. Before round-robin, a large queue size would cause the Coordinato to get stuck in
RunRules
duty due to very slow strategy-based cost computations.2.
replicationThrottleLimit = 500
(previous = 10)Rationale: Along with the reasoning given for
maxSegmentsInNodeLoadingQueue
, a very lowreplicationThrottleLimit
can cause clusters to be very slow in getting to full replication, even when there are loading threads sitting idle.Note: It is okay to keep this value equal to
maxSegmentsInNodeLoadingQueue
. Even with equal values, load queues will not get filled up with just replicas, and segments that are completely unavailable will still get a fair chance. This is because while MSINLQ applies to a single server,replicationThrottleLimit
applies to each tier.3.
maxSegmentsToMove = 100
(previous = 5)Rationale: A very low value of this config (say 5) turns out to be very ineffective in balancing especially if there are a large number of segments in a cluster and/or a large skew between usages of two historical servers.
On the other hand, a very large value can cause excessive moves every minute, which might have the following disadvantages:
These defaults will be revisited after #13197 is merged.
Testing
These values have been tried on different production cluster sizes, and have been found to give satisfactory results.
Release note
Update default values of the following coordinator dynamic configs:
maxSegmentsInNodeLoadingQueue = 500
maxSegmentsToMove = 100
replicationThrottleLimit = 500
This PR has: