Multi schedulers #143
Multi schedulers #143
Conversation
Schedulers have to acquire a lock before one can schedule jobs. The lock automatically expires in case a scheduler is terminated unexpectedly. Schedulers that cannot acquire a lock will sleep for the duration of polling interval before retrying. refs rq#70
|
@marcinn could you please restore |
|
OK, I'll look at this. I've just resolved the conflict without looking at birth/death. |
|
@selwin I'm thinking over and over again about scheduler locking and I'm not sure about current implementation. The lock is acquired in the loop and stays acquired until end of the work. So in non-burst mode it will be locked constantly, even during sleeping. Maybe the lock should be released before scheduler is going to sleep? It would allow other long-running schedulers to run. The one risk I see relies in not synced clocks between hosts running the scheduler, where each instance may enqueue too many jobs. What do you think? |
|
@marcinn I see your point. However, I'm more inclined to leave it as it now. People mostly request this feature for redundancy reasons. If a scheduler releases its lock everytime it goes to sleep, people with multiple schedulers will have many schedulers firing off at random times, which is not what we want. |
|
Thank you, @selwin |
Here is an updated PR for multiple schedulers support (GH-104).
refs #70