Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust transfer costs in worker_objective #5326

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from

Commits on Sep 17, 2021

  1. 10ms penalty per transfer

    I'd like to incorporate measured latency somehow too instead of a magic 10ms, but it's a start.
    gjoseph92 committed Sep 17, 2021
    Configuration menu
    Copy the full SHA
    4c67b0b View commit details
    Browse the repository at this point in the history
  2. Amortize transfer cost

    As discussed in dask#5325. The idea is that if a key we need has many dependents, we should amortize the cost of transferring it to a new worker, since those other dependencies could then run on the new worker more cheaply. "We'll probably have to move this at some point anyway, might as well do it now."
    
    This isn't actually intended to encourage transfers though. It's more meant to discourage transferring keys that could have just stayed in one place. The goal is that if A and B are on different workers, and we're the only task that will ever need A, but plenty of other tasks will need B, we should schedule alongside A even if B is a bit larger to move.
    
    But this is all a theory and needs some tests.
    gjoseph92 committed Sep 17, 2021
    Configuration menu
    Copy the full SHA
    b4ebbee View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    cbc145a View commit details
    Browse the repository at this point in the history

Commits on May 4, 2022

  1. Configuration menu
    Copy the full SHA
    7f32a1a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a6c2315 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5f18151 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    01a0e8c View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    e3d62f6 View commit details
    Browse the repository at this point in the history