attempt to also run e2e tests that needs gpus #1070
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hamelsmu
approved these changes
Jan 9, 2024
|
Hopefully this won't block CI pipeline too much |
hamelsmu
requested changes
Jan 9, 2024
|
hamelsmu
reviewed
Jan 9, 2024
|
@winglian I dont have access to the hosted runner, but I think that we need to set nvidia runtime as the default (just trying to interpret the error message). On your self-hosted runner, in {
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}If this is not the case, make the change and then restart docker by running |
winglian
force-pushed
the
docker-e2e-enable
branch
3 times, most recently
from
639de26
to
3f8a9d1
Compare
ensure wandb is dissabled for docker pytests clear wandb env after testing clear wandb env after testing make sure to provide a default val for pop tryin skipping wandb validation tests explicitly disable wandb in the e2e tests explicitly report_to None to see if that fixes the docker e2e tests split gpu from non-gpu unit tests skip bf16 check in test for now build docker w/o cache since it uses branch name ref revert some changes now that caching is fixed skip bf16 check if on gpu w support
winglian
force-pushed
the
docker-e2e-enable
branch
from
a8d3c31
to
bd57059
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.