-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move query execution to RQ #4307
Comments
Regarding query cancellation - how awful would it be if we piggy-back ride on the work horse monitoring loop? Effectively this means that queries that get cancelled might only stop after up to The alternative would be a thread watching for cancellations in the work horse, but I'm trying to think of ways to avoid making our code complex. |
It doesn't sound too bad if we can lower the interval to something like 5s. What impact it will have? |
From what I can tell, just more heartbeat registrations. |
No impact on CPU or other resources? If so let's lower it and move on. |
I guess also more alarm signals will be registered, but at 5 second intervals, these are extremely negligible. |
I don't know if it complicates things much, but we should consider having an optional handler function (in the query runner) that does the actual cancel and if it's not available then resort to current "kill process". This will allow for implementing #4410 or MySQL's cancel procedure without the use of threads. This is a nice to have of course and if it complicates the implementation let's move on with implementing the current behavior (of sending an interrupt signal and on second try a kill signal). |
I guess we can close this? |
Move the
execute_query
task from Celery to RQ. Once this is done, we're Celery free. 🎉Depends on #4305 and ability to cancel jobs (rq/rq#684, which we will probably implement in a custom Worker in our codebase and then try to port to RQ).
(Opening this to track the work, but this is currently sparse on details)
The text was updated successfully, but these errors were encountered: