Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add requeue and QOS slurm options #144

Merged
merged 3 commits into from
Apr 2, 2024

Conversation

marianna13
Copy link
Contributor

  1. Not all clusters support QOS and requeue options so we make them optional.
  2. launch_pickled_pipeline might not work all the time (e.g. if you use a local fork of datatrove)

@guipenedo
Copy link
Collaborator

Hi! Thanks for the changes, they look great!
Was just unsure as to why having a local fork would make the launch pipeline command not work. On my system installing from local source (pip install . -e) adds the command to the environment without issue, does this not happen for you?

@marianna13
Copy link
Contributor Author

Ok pip install . works for launch_pickled_pipeline

@guipenedo
Copy link
Collaborator

Thank you for the PR, I made a small change and fixed the style. Merged!

@guipenedo guipenedo merged commit afadc8f into huggingface:main Apr 2, 2024
4 checks passed
rlrs added a commit to TrustLLMeu/datatrove that referenced this pull request Apr 15, 2024
add requeue and QOS slurm options (huggingface#144)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants