Schedule periodic tasks centrally

[ianb]

We have a couple periodic tasks. Right now these are done with setInterval() in the JS code. This is lame and actually incorrect in production, with multiple application servers running. How should we run these?

If we could leave the work in the application server, but trigger it via an HTTP request (authenticated in some fashion?) that might be best/easiest/most reliable. @relud: thoughts?

[relud]

We could set up a single server to behave differently from the rest and have cron tasks on it. It would be configured such that there is only one, and could be the same server that #1977 gets run on.

Ideally the tasks that it runs would gracefully handle being missed once, or being run an extra time, and the upgrade process for it would make sure there are always at least one, or at most one, of these servers accordingly.

[ianb]

All the periodic tasks can be run extra times without a problem. Will the special server just get a different environmental variable that will tell it to act differently?

[relud]

yeah, and it could be either one of the servers in the ELB, or a separate server (like if we have an admin node)

[ianb]

I did this in 2d4df88 and if the variable CONTROLLER_SINGLETON=true is present then periodic tasks will be performed by this server.