-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve polling in segment allocation queue #15590
Conversation
...g-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java
Outdated
Show resolved
Hide resolved
Update: Added a few more changes to handle race conditions better and make the code simpler. |
Requests with lineage check cannot be batched. Would it help to not wait before polling for such requests? [Edit] |
Thanks for the suggestion, @AmatyaAvadhanula . To help me understand better, would a more precise statement instead of I ask this because the Either way, my plan is to eventually get rid of Please let me know what you think. |
Yes, it would be a more precise statement. Thank you for the explanation, @kfaraz. I understand the intent of this change better now. |
LGTM @kfaraz! |
Description
When
batchAllocationWaitTime
is set to0
, the segment allocation queue is polled continuously even when it is empty. This would take up cpu cycles unnecessarily.Some existing race conditions would also become more frequent when the
batchAllocationWaitTime
is0
. This PR tries to better address those race conditions as well.Changes
Alternate approach
It would be much simpler to do a long poll of the queue, but we cannot do that here since we first peek at the batch at head of the queue and pop it only if it is due. There doesn't seem to be an equivalent long peek 🙂.
This PR has: