Move query execution to RQ

[arikfr]

Move the execute_query task from Celery to RQ. Once this is done, we're Celery free. 🎉

Depends on #4305 and ability to cancel jobs (rq/rq#684, which we will probably implement in a custom Worker in our codebase and then try to port to RQ).

(Opening this to track the work, but this is currently sparse on details)

[rauchy]

Regarding query cancellation - how awful would it be if we piggy-back ride on the work horse monitoring loop? Effectively this means that queries that get cancelled might only stop after up to job_monitoring_interval (default is 30s, but we can tighten that up).

The alternative would be a thread watching for cancellations in the work horse, but I'm trying to think of ways to avoid making our code complex.

[arikfr]

It doesn't sound too bad if we can lower the interval to something like 5s. What impact it will have?

[rauchy]

From what I can tell, just more heartbeat registrations.

[arikfr]

No impact on CPU or other resources? If so let's lower it and move on.

[rauchy]

I guess also more alarm signals will be registered, but at 5 second intervals, these are extremely negligible.

[arikfr]

I don't know if it complicates things much, but we should consider having an optional handler function (in the query runner) that does the actual cancel and if it's not available then resort to current "kill process".

This will allow for implementing #4410 or MySQL's cancel procedure without the use of threads.

This is a nice to have of course and if it complicates the implementation let's move on with implementing the current behavior (of sending an interrupt signal and on second try a kill signal).

[arikfr]

I guess we can close this?