-
-
Notifications
You must be signed in to change notification settings - Fork 718
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add TaskState to worker.py #4107
Commits on Sep 10, 2020
-
Add TaskState for some (but not all) relevant dicts
Starting with: * `key` * `duration` * `priority` * `state` * `resource_restrictions` * `exception` * `traceback` Going to get that working, then handle the rest -- `dep_state` seems tricky to get right
Configuration menu - View commit details
-
Copy full SHA for 1eddc52 - Browse repository at this point
Copy the full SHA 1eddc52View commit details -
Configuration menu - View commit details
-
Copy full SHA for eb0d87d - Browse repository at this point
Copy the full SHA eb0d87dView commit details -
Configuration menu - View commit details
-
Copy full SHA for f4ae9ac - Browse repository at this point
Copy the full SHA f4ae9acView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7a36655 - Browse repository at this point
Copy the full SHA 7a36655View commit details -
Configuration menu - View commit details
-
Copy full SHA for 59f2ce2 - Browse repository at this point
Copy the full SHA 59f2ce2View commit details
Commits on Sep 14, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 5e49258 - Browse repository at this point
Copy the full SHA 5e49258View commit details -
First cut at removing
dep_state
,dependencies
anddependents
dictsWorking plan: - All dependencies are also Tasks (they don't necessarily have a `runspec` because they might be gathered from another worker) - `dependents` and `dependencies` are sets of TaskState objects
Configuration menu - View commit details
-
Copy full SHA for 604f23c - Browse repository at this point
Copy the full SHA 604f23cView commit details -
Also treating `who_has` as a Task object level dict, where keys are the dependency keys and workers is a list of workers who have that data This is maybe... wrong? Each dependency TaskState object should be itself (ish) so probably just needs to be a list of workers
Configuration menu - View commit details
-
Copy full SHA for 3ece01f - Browse repository at this point
Copy the full SHA 3ece01fView commit details -
Configuration menu - View commit details
-
Copy full SHA for fff2281 - Browse repository at this point
Copy the full SHA fff2281View commit details
Commits on Sep 16, 2020
-
Configuration menu - View commit details
-
Copy full SHA for a7af14d - Browse repository at this point
Copy the full SHA a7af14dView commit details -
Configuration menu - View commit details
-
Copy full SHA for a5d08ff - Browse repository at this point
Copy the full SHA a5d08ffView commit details
Commits on Sep 17, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 69bac91 - Browse repository at this point
Copy the full SHA 69bac91View commit details -
Configuration menu - View commit details
-
Copy full SHA for e380517 - Browse repository at this point
Copy the full SHA e380517View commit details -
Configuration menu - View commit details
-
Copy full SHA for a35bddc - Browse repository at this point
Copy the full SHA a35bddcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1180f71 - Browse repository at this point
Copy the full SHA 1180f71View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2f05334 - Browse repository at this point
Copy the full SHA 2f05334View commit details -
Configuration menu - View commit details
-
Copy full SHA for 81150e5 - Browse repository at this point
Copy the full SHA 81150e5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 249b196 - Browse repository at this point
Copy the full SHA 249b196View commit details
Commits on Sep 18, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 2012784 - Browse repository at this point
Copy the full SHA 2012784View commit details -
Configuration menu - View commit details
-
Copy full SHA for e441e56 - Browse repository at this point
Copy the full SHA e441e56View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0486146 - Browse repository at this point
Copy the full SHA 0486146View commit details
Commits on Sep 19, 2020
-
Configuration menu - View commit details
-
Copy full SHA for b345125 - Browse repository at this point
Copy the full SHA b345125View commit details
Commits on Sep 21, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 2bf3259 - Browse repository at this point
Copy the full SHA 2bf3259View commit details -
Configuration menu - View commit details
-
Copy full SHA for 112823f - Browse repository at this point
Copy the full SHA 112823fView commit details -
Configuration menu - View commit details
-
Copy full SHA for dc725bb - Browse repository at this point
Copy the full SHA dc725bbView commit details -
Configuration menu - View commit details
-
Copy full SHA for ef4eea4 - Browse repository at this point
Copy the full SHA ef4eea4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6156e9c - Browse repository at this point
Copy the full SHA 6156e9cView commit details -
Configuration menu - View commit details
-
Copy full SHA for a27ed18 - Browse repository at this point
Copy the full SHA a27ed18View commit details -
Configuration menu - View commit details
-
Copy full SHA for d0ffad2 - Browse repository at this point
Copy the full SHA d0ffad2View commit details -
Configuration menu - View commit details
-
Copy full SHA for b98b199 - Browse repository at this point
Copy the full SHA b98b199View commit details -
Configuration menu - View commit details
-
Copy full SHA for f77b33c - Browse repository at this point
Copy the full SHA f77b33cView commit details -
Configuration menu - View commit details
-
Copy full SHA for a9b2ff9 - Browse repository at this point
Copy the full SHA a9b2ff9View commit details -
Merge _dep_transitions and _transitions and remove _dep checks
They are all key checks now
Configuration menu - View commit details
-
Copy full SHA for 9f7cf89 - Browse repository at this point
Copy the full SHA 9f7cf89View commit details -
Add flight -> ready entry in `transitions
This should be the same as flight -> memory
Configuration menu - View commit details
-
Copy full SHA for 2ae9a87 - Browse repository at this point
Copy the full SHA 2ae9a87View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9446af9 - Browse repository at this point
Copy the full SHA 9446af9View commit details
Commits on Sep 22, 2020
-
Add Worker-level count of waiting for data tasks
Need to think about this more, but we don't want to count these up every time the dashboard refreshes
Configuration menu - View commit details
-
Copy full SHA for aee4bae - Browse repository at this point
Copy the full SHA aee4baeView commit details -
Configuration menu - View commit details
-
Copy full SHA for ff05783 - Browse repository at this point
Copy the full SHA ff05783View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0b587d6 - Browse repository at this point
Copy the full SHA 0b587d6View commit details -
Configuration menu - View commit details
-
Copy full SHA for eda1bde - Browse repository at this point
Copy the full SHA eda1bdeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f47d02 - Browse repository at this point
Copy the full SHA 4f47d02View commit details -
Configuration menu - View commit details
-
Copy full SHA for e1dac2f - Browse repository at this point
Copy the full SHA e1dac2fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 64f2af3 - Browse repository at this point
Copy the full SHA 64f2af3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4d6f191 - Browse repository at this point
Copy the full SHA 4d6f191View commit details -
Configuration menu - View commit details
-
Copy full SHA for c5eaf37 - Browse repository at this point
Copy the full SHA c5eaf37View commit details -
It is OK for there to be tasks that aren't in `self.tasks` yet, but not ok if they don't have an assigned `state`
Configuration menu - View commit details
-
Copy full SHA for 4f94290 - Browse repository at this point
Copy the full SHA 4f94290View commit details -
Configuration menu - View commit details
-
Copy full SHA for 28e5694 - Browse repository at this point
Copy the full SHA 28e5694View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5ab969c - Browse repository at this point
Copy the full SHA 5ab969cView commit details -
Configuration menu - View commit details
-
Copy full SHA for b32629c - Browse repository at this point
Copy the full SHA b32629cView commit details
Commits on Sep 23, 2020
-
Update tests to not call
put_key_in_memory
directlyTests (and everything else) should use `update_data` for this kind of (unusual) call. This also avoids special casing an otherwise absent Task object
Configuration menu - View commit details
-
Copy full SHA for d8ce2d0 - Browse repository at this point
Copy the full SHA d8ce2d0View commit details -
continue
if there are missing tasks onready
Comment in code, too, but I was initially treating this as an error, but it can happen as a regular part task execution where a task is re-routed but `self.ready` isn't updated before we get here.
Configuration menu - View commit details
-
Copy full SHA for c3c1110 - Browse repository at this point
Copy the full SHA c3c1110View commit details
Commits on Sep 24, 2020
-
Configuration menu - View commit details
-
Copy full SHA for de950fd - Browse repository at this point
Copy the full SHA de950fdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 24e6ed4 - Browse repository at this point
Copy the full SHA 24e6ed4View commit details -
Add ready -> waiting transition
I think this probably isn't right, but I'm trying it out
Configuration menu - View commit details
-
Copy full SHA for 44d761c - Browse repository at this point
Copy the full SHA 44d761cView commit details
Commits on Sep 25, 2020
-
Configuration menu - View commit details
-
Copy full SHA for ad3c248 - Browse repository at this point
Copy the full SHA ad3c248View commit details -
Configuration menu - View commit details
-
Copy full SHA for 046360e - Browse repository at this point
Copy the full SHA 046360eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4550780 - Browse repository at this point
Copy the full SHA 4550780View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8f1892b - Browse repository at this point
Copy the full SHA 8f1892bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3a24aa8 - Browse repository at this point
Copy the full SHA 3a24aa8View commit details -
Handle missing deps with
TaskState
Also removed adding a dep to `_missing_dep_flight` in one of the transition functions -- that leads to it being excluded from the dep gather, which seems circular in a bad way
Configuration menu - View commit details
-
Copy full SHA for 24108f2 - Browse repository at this point
Copy the full SHA 24108f2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 887081e - Browse repository at this point
Copy the full SHA 887081eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 504a51f - Browse repository at this point
Copy the full SHA 504a51fView commit details
Commits on Sep 28, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 7b85334 - Browse repository at this point
Copy the full SHA 7b85334View commit details
Commits on Sep 29, 2020
-
Configuration menu - View commit details
-
Copy full SHA for fbf776a - Browse repository at this point
Copy the full SHA fbf776aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 349fe8e - Browse repository at this point
Copy the full SHA 349fe8eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0cc88f8 - Browse repository at this point
Copy the full SHA 0cc88f8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 992fbbd - Browse repository at this point
Copy the full SHA 992fbbdView commit details -
Configuration menu - View commit details
-
Copy full SHA for b60d4e5 - Browse repository at this point
Copy the full SHA b60d4e5View commit details -
Configuration menu - View commit details
-
Copy full SHA for ced3fd4 - Browse repository at this point
Copy the full SHA ced3fd4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2ef35ff - Browse repository at this point
Copy the full SHA 2ef35ffView commit details -
Configuration menu - View commit details
-
Copy full SHA for f67353b - Browse repository at this point
Copy the full SHA f67353bView commit details -
Restore
ts
check intransition
And handle the possibility of a `cause` not existing in `self.tasks`
Configuration menu - View commit details
-
Copy full SHA for 173c28d - Browse repository at this point
Copy the full SHA 173c28dView commit details
Commits on Sep 30, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 44e4404 - Browse repository at this point
Copy the full SHA 44e4404View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3ad8972 - Browse repository at this point
Copy the full SHA 3ad8972View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1a2c591 - Browse repository at this point
Copy the full SHA 1a2c591View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5c4aa90 - Browse repository at this point
Copy the full SHA 5c4aa90View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1a3b189 - Browse repository at this point
Copy the full SHA 1a3b189View commit details -
Configuration menu - View commit details
-
Copy full SHA for c2ae508 - Browse repository at this point
Copy the full SHA c2ae508View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5aaac47 - Browse repository at this point
Copy the full SHA 5aaac47View commit details -
Add validate check for "flight"
This doesn't include the `assert ts.key not in self.nbytes` -- since tasks are created via `add_task`, they can have `nbytes` specified before-hand
Configuration menu - View commit details
-
Copy full SHA for fde532c - Browse repository at this point
Copy the full SHA fde532cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 44135db - Browse repository at this point
Copy the full SHA 44135dbView commit details
Commits on Oct 9, 2020
-
Add docstring, change var name for waiting task count
And also restore said count to Worker `repr`
Configuration menu - View commit details
-
Copy full SHA for 5135c1c - Browse repository at this point
Copy the full SHA 5135c1cView commit details -
Apply suggestions from code review
Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for f24365e - Browse repository at this point
Copy the full SHA f24365eView commit details -
Configuration menu - View commit details
-
Copy full SHA for ccfd9eb - Browse repository at this point
Copy the full SHA ccfd9ebView commit details
Commits on Oct 13, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 0877311 - Browse repository at this point
Copy the full SHA 0877311View commit details -
Replace
self.executing
withself.executing_count
This is a redundant bit of state tracking and overcomplicates things. Instead, similar to `Worker.tasks` this maintains a count of the executing tasks for reporting but relies on `TaskState` objects to keep track of the task state. Note that the removal of the `finally:` block, specifically, is because _if_ the key has already been released, then `self.executing_count` is already decremented so that particular check doesn't need to be run.
Configuration menu - View commit details
-
Copy full SHA for f49fbe8 - Browse repository at this point
Copy the full SHA f49fbe8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 508927f - Browse repository at this point
Copy the full SHA 508927fView commit details -
Configuration menu - View commit details
-
Copy full SHA for abce5f6 - Browse repository at this point
Copy the full SHA abce5f6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6f9e6a9 - Browse repository at this point
Copy the full SHA 6f9e6a9View commit details -
Configuration menu - View commit details
-
Copy full SHA for d15e443 - Browse repository at this point
Copy the full SHA d15e443View commit details
Commits on Oct 15, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 1e7de67 - Browse repository at this point
Copy the full SHA 1e7de67View commit details -
Update distributed/diagnostics/tests/test_worker_plugin.py
Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 403ca03 - Browse repository at this point
Copy the full SHA 403ca03View commit details