-
-
Notifications
You must be signed in to change notification settings - Fork 718
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adjust transfer costs in worker_objective
#5326
Draft
gjoseph92
wants to merge
8
commits into
dask:main
Choose a base branch
from
gjoseph92:worker_objective/adjust-transfer-cost
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Adjust transfer costs in worker_objective
#5326
gjoseph92
wants to merge
8
commits into
dask:main
from
gjoseph92:worker_objective/adjust-transfer-cost
Commits on Sep 17, 2021
-
I'd like to incorporate measured latency somehow too instead of a magic 10ms, but it's a start.
Configuration menu - View commit details
-
Copy full SHA for 4c67b0b - Browse repository at this point
Copy the full SHA 4c67b0bView commit details -
As discussed in dask#5325. The idea is that if a key we need has many dependents, we should amortize the cost of transferring it to a new worker, since those other dependencies could then run on the new worker more cheaply. "We'll probably have to move this at some point anyway, might as well do it now." This isn't actually intended to encourage transfers though. It's more meant to discourage transferring keys that could have just stayed in one place. The goal is that if A and B are on different workers, and we're the only task that will ever need A, but plenty of other tasks will need B, we should schedule alongside A even if B is a bit larger to move. But this is all a theory and needs some tests.
Configuration menu - View commit details
-
Copy full SHA for b4ebbee - Browse repository at this point
Copy the full SHA b4ebbeeView commit details -
Configuration menu - View commit details
-
Copy full SHA for cbc145a - Browse repository at this point
Copy the full SHA cbc145aView commit details
Commits on May 4, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 7f32a1a - Browse repository at this point
Copy the full SHA 7f32a1aView commit details -
Configuration menu - View commit details
-
Copy full SHA for a6c2315 - Browse repository at this point
Copy the full SHA a6c2315View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5f18151 - Browse repository at this point
Copy the full SHA 5f18151View commit details -
Configuration menu - View commit details
-
Copy full SHA for 01a0e8c - Browse repository at this point
Copy the full SHA 01a0e8cView commit details -
Configuration menu - View commit details
-
Copy full SHA for e3d62f6 - Browse repository at this point
Copy the full SHA e3d62f6View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.