rqscheduler can constantly attempt to register itself.

[ScottSturdivant]

Allow rqscheduler to keep attempting to register itself periodically.

My use case is that I have N identical hosts all running rqworker and rqscheduler processes under the watchful eye of supervisor. As supevisord doesn't support an 'unlimited' value for the 'startretries' program configuration option, eventually the rqscheduler process will be moved into supervisor's failed state. Thus, if the host that was successfully running rqscheduler goes down, none of the other existing hosts can automatically take its place.

Similarly, when performing rolling updates, the new hosts come online and try to launch rqscheduler. This fails because an old host is already executing it. If the rollout is slow, it's possible that the new hosts will again have put rqscheduler into a failed state before the old host is rolled out. This will result in a new deployment where rqscheduler is not running.

This patch will allow rqscheduler itself to keep retrying its registration process. Backwards compatability is preserved by aborting by default.

[selwin]

Would introducing a --burst option be better for this use case?

Similar to RQ worker, running rqscheduler --burst would scheduled all jobs that need to be scheduled and quit on completion.

This means you can schedule N hosts to run rqscheduler --burst every minute via cron on multiple hosts and will retry infinitely.

[lost-theory]

@selwin What would the behavior be with --burst when two schedulers run at the same time? Seems like you'd have the same problem, one process would throw an error. And it would introduce a dependency on cron (one of the reasons people use a system like rq & rq-scheduler is to get away from cron 😄).

FWIW resque-scheduler (the analog of rq-scheduler in the ruby world) allows you to run multiple schedulers and it handles failover automatically:

https://github.com/resque/resque-scheduler#redundancy-and-fail-over

You may want to have resque-scheduler running on multiple machines for redudancy. Electing a master and failover is built in and default. Simply run resque-scheduler on as many machine as you want pointing to the same redis instance and schedule. The scheduler processes will use redis to elect a master process and detect failover when the master dies. Precautions are taken to prevent jobs from potentially being queued twice during failover even when the clocks of the scheduler machines are slightly out of sync (or load affects scheduled job firing time). If you want the gory details, look at Resque::Scheduler::Locking.

I think this is a good approach, and is very similar to @SirScott's patch (all processes continually try to acquire a 'master' lock until one succeeds, and TTLs allow failover to happen when a scheduler process dies unexpectedly).

[selwin]

Sorry, I forgot to reply to this issue.

To be honest, I don't think rq-scheduler as a direct replacement for cron as I think it's one of the most battle tested utility out there. rq-scheduler is meant to be something that lets you schedule jobs programatically.

Yes, two schedulers running at the same time would still create an error. But what I like about the --burst approach is that you won't get two active scheduler processes at the same time (the one that errors out would just die).

However, I've also been thinking about the approach @SirScott suggested and am not opposed to it. Can we have a more descriptive name than --retry though?

[jmmills]

Could it be that you just enqueue jobs that do the schedule poll? That way the loop is shared across the rqworker cluster?

[selwin]

@SirScott thanks for writing this PR, please see my comment here: #70 (comment)