-
Notifications
You must be signed in to change notification settings - Fork 645
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[5.x] Add autoScalingStrategy option #1254
[5.x] Add autoScalingStrategy option #1254
Conversation
de7529a
to
ceaaa58
Compare
Do we really need a new balancing strategy? Could you just define a second supervisor configuration for the other queues you want to ensure have more processes? |
You also state you think it would be nice to allow a custom strategy per supervisor... but isn't that what you have done in this PR? Am I missing something there? Also, can you share an example of a configuration using you new option? |
This was an issue we noticed on our production app. We our running our Laravel monolith on K8s. We scale our worker pods automatically. We have 12 different queues handled by Horizon, running 3 separate supervisors. Using the current Horizon strategy, the I do think that time to clear should play a factor in the balancing strategy, but it shouldn’t impact it this significantly. With a supervisor responsible for 6 queues & having Ideally I think it would be best if the process allocation was based on a combination of time to clear + size of queue, but I'm not sure what that would look like. Because of the current Horizon auto-balance strategy, we see an increase in customer support tickets. It feels like we would be playing whack-a-mole to guess which combinations of queues should share a supervisor. Is it envisioned that production applications have a supervisor for each queue?
I could've added more clarity to that one. Yes, that is sort of what I have done with this PR, but that's still only 2 strategies. Was thinking it might be beneficial for Horizon to allow a consumer application to modify or add their own auto-balancing strategy. (I'm not entirely should what that would look like, but just figured I would include it as a thought that I had).
Here is an example of the configuration so it relies on the new strategy included in the PR: 'supervisor-medium' => [
'connection' => 'redis_medium',
'queue' => [
'ai',
'notifications',
'import',
'other-queue',
],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 10,
'memory' => 64,
'tries' => 3,
'timeout' => 300,
'autoScalingStrategy' => 'size', // now this supervisor will partition # of processes to the queue with the most jobs, not the estimated time to complete
], On Friday, we launched our app to use the If a consumer doesn't set an |
Thanks! |
No no, thank you @taylorotwell |
This PR adds an option to configure how a supervisor auto-scales queues. It allows for
runtime
orsize
, set on the supervisor level under theautoScalingBehavior
key.runtime
is the current behavior (portion the number of available processes based on time to complete a queue / total time to complete all queues), where assize
portions processes based on total number of jobs in queue / total number of jobs in all supervised queues.For some context:
We have a Horizon supervisor set up like this:
How should the queue supervisor balance when there are 2700 jobs in the
ai
queue and 5000 jobs in theimport
queue?We are seeing that, though the supervisor spins up more processes, they are all given to the
ai
queue and not allocated to theimport
queue. So, we would see thatai
queue processes went from 1 to 6 processes, butimport
stayed at 1.To solve this problem, we extended the AutoScaler class to add this behavior. I thought it might be a nice feature for Horizon to have.
Also, I think it's worth noting in the Horizon docs about the time-to-complete behavior. As they currently read:
I think this is easy to misunderstand as it reads like it's balanced on job count, not time to complete. It required digging into the AutoScaler code to understand the behavior we were seeing.
Additionally, I think it might be nice to:
For the second point, I couldn't think of a good way to do that. I was thinking maybe that jobs should be a factor in the calculation, weighted lower than the time to complete. But I didn't spend much time trying things out.