Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow jobs to be queuable once the existing job starts performing #16

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

lazyatom
Copy link

@lazyatom lazyatom commented Sep 7, 2016

This is useful when you know that the currently running job might not be
taking into account the most recent updates to some of the data it is
operating on, which can occur when the jobs take some time to run.

For example:

  1. update model
  2. queue job to index in ElasticSearch
  3. job starts performing
  4. another update happens to the model
  5. try to queue job to update index

With the default queue locking mechanism, step 5 will only succeed in
enqueuing a new job if the job that started in step 3 has already
completed. However, if that job hasn't finished, but has loaded the
model with only the changes from step 1, then the changes introduced in
step 4 won't get indexed.

With unlock_while_performing set to true, there can always be a single
pending job in the queue, even when there's also a job running, and so
we can be confident that any un-indexed changes to the model will get
indexed once the final job finishes running.

This is useful when you know that the currently running job might not be
taking into account the most recent updates to some of the data it is
operating on, which can occur when the jobs take some time to run.

For example:

  1. update model
  2. queue job to index in ElasticSearch
  3. job starts performing
  4. another update happens to the model
  5. try to queue job to update index

With the default queue locking mechanism, step 5 will only succeed in
enqueuing a new job if the job that started in step 3 has already
completed. However, if that job hasn't finished, but has loaded the
model with only the changes from step 1, then the changes introduced in
step 4 won't get indexed.

With `unlock_while_performing` set to true, there can always be a single
pending job in the queue, even when there's also a job running, and so
we can be confident that any un-indexed changes to the model will get
indexed once the final job finishes running.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant