Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rework for webknossos-worker as client-only #5834

Merged
merged 24 commits into from
Nov 25, 2021
Merged

Rework for webknossos-worker as client-only #5834

merged 24 commits into from
Nov 25, 2021

Conversation

fm3
Copy link
Member

@fm3 fm3 commented Nov 5, 2021 •
edited
Loading

The communication with webknossos-worker for long-running jobs no longer uses flower/celery, but instead webKnossos itself assigns jobs to polling workers.

TODO

  • save workers in postgres
  • structure job state in postgres
  • give out work to worker
  • ensure no duplicates
  • let worker post job status
  • select datastore for job when submitting
  • jobs status report
  • worker status report
  • insert/delete local worker in yarn enable-jobs
  • detect worker death + resurrection
  • in worker, retry posting job status
  • slack notifications
  • remove unused code
  • write postgres evolutions
    • forward
    • reversion
  • remove redundant jobsEnabled field in datastore sql table (should stay in dataSet publicWrites, though)
  • re-build smoke test in worker
  • make CI happy
  • write follow-up issues

Steps to test:

  • enable jobs by calling yarn enable-jobs
  • start a webknossos-worker (version from https://github.com/scalableminds/webknossos-worker/pull/70/ )
  • export a bounding box from a dataset (bbox tab in view mode in right side bar)
  • job should be executed, result should be downloadable from the wk jobs overview page
  • kill worker during job, restart, job should be restarted
  • submit multiple jobs, see that no combination breaks the setup
  • api/jobs/status should show the worker and the jobs with status

Issues:

@fm3 fm3 self-assigned this Nov 5, 2021
@fm3 fm3 changed the title [WIP] rework for webknossos-worker as client-only Rework for webknossos-worker as client-only Nov 22, 2021
@fm3 fm3 marked this pull request as ready for review November 22, 2021 12:05
@fm3 fm3 requested a review from jstriebel November 22, 2021 13:48
jstriebel
jstriebel approved these changes Nov 25, 2021
View reviewed changes
Copy link
Contributor

@jstriebel jstriebel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, LGTM 👍 I like very much how this simplifies the setup 🚀

And thanks for the great walk-through yesterday! I just added one more minor nitpick ;-)

tools/postgres/toggle_jobs.sh Outdated Show resolved Hide resolved
jstriebel
jstriebel reviewed Nov 25, 2021
View reviewed changes
package.json
Comment on lines +122 to +123
"enable-jobs": "sed -i -e 's/jobsEnabled = false/jobsEnabled = true/g' ./conf/application.conf; ./tools/postgres/set_jobs.sh true",
"disable-jobs": "sed -i -e 's/jobsEnabled = true/jobsEnabled = false/g' ./conf/application.conf; ./tools/postgres/set_jobs.sh false",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ Wasn't aware of the commands here, this is nice!

@fm3 fm3 merged commit d355a7b into master Nov 25, 2021
@fm3 fm3 deleted the worker-as-client-only branch November 25, 2021 09:57
This was referenced Nov 25, 2021
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jstriebel jstriebel jstriebel approved these changes

Assignees

@fm3 fm3

Labels
backend jobs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

If the wk worker crashes/restarts during a job, wk forever thinks the job is running
2 participants
@fm3 @jstriebel