Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Queue root tasks on scheduler, not workers [with co-assignment] #6598

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from

Commits on Jun 18, 2022

  1. unused: OrderedSet collection

    Idea was that if a `SortedSet` of unrunnable tasks is too expensive, then insertion order is probably _approximately_ priority order, since higher-priority (root) tasks will be scheduled first. This would give us O(1) for all necessary operations, instead of O(logn) for adding and removing.
    
    Interestingly, the SortedSet implementation could be hacked to support O(1) `pop` and `popleft`, and inserting a min/max value. In the most common case (root tasks), we're always inserting a value that's greater than the max. Something like this might be the best tradeoff, since it gives us O(1) in the common case but still maintains the sorted gaurantee, which is easier to reason about.
    gjoseph92 committed Jun 18, 2022
    Configuration menu
    Copy the full SHA
    fdd5fd9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    61b6a12 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    4830cf6 View commit details
    Browse the repository at this point in the history
  4. improve reasonableness of task-state order

    Now task states on the dashboard are listed in the logical order that tasks transition through.
    gjoseph92 committed Jun 18, 2022
    Configuration menu
    Copy the full SHA
    7365df4 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    66999e1 View commit details
    Browse the repository at this point in the history
  6. Only support floats for worker-oversaturation

    Simpler, though I think basically just an int of 1 may be the most reasonable.
    gjoseph92 committed Jun 18, 2022
    Configuration menu
    Copy the full SHA
    f28b9b2 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    1380151 View commit details
    Browse the repository at this point in the history
  8. Queued tasks on info pages

    gjoseph92 committed Jun 18, 2022
    Configuration menu
    Copy the full SHA
    117c082 View commit details
    Browse the repository at this point in the history
  9. driveby: WIP color task graph by worker

    This is just a hack currently, but maybe it would actually be useful?
    gjoseph92 committed Jun 18, 2022
    Configuration menu
    Copy the full SHA
    6a831b2 View commit details
    Browse the repository at this point in the history
  10. Revert "driveby: WIP color task graph by worker"

    This reverts commit df11f719b59aad11f39a27ccae7b2fd4dfd9243a.
    gjoseph92 committed Jun 18, 2022
    Configuration menu
    Copy the full SHA
    f9daba4 View commit details
    Browse the repository at this point in the history
  11. Queued tasks on graph

    gjoseph92 committed Jun 18, 2022
    Configuration menu
    Copy the full SHA
    0edec5d View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    8a4b2db View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    590aa5b View commit details
    Browse the repository at this point in the history
  14. Fix co-assignment logic to consider queued tasks

    When there were multiple root task groups, we were just re-using the last worker for every batch because it had nothing processing on it.
    
    Unintentionally this also fixes dask#6597 in some cases (because the first task goes to processing, but we measure queued, so we pick the same worker for both task groups)
    gjoseph92 committed Jun 18, 2022
    Configuration menu
    Copy the full SHA
    b2e7924 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2022

  1. Revert "unused: OrderedSet collection"

    This reverts commit fdd5fd9.
    gjoseph92 committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    c62f82a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    51ad734 View commit details
    Browse the repository at this point in the history