Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix environment variable export bug for MultiNodeRunner #5878

Merged
merged 8 commits into from
Sep 8, 2024

Conversation

TideDra
Copy link
Contributor

@TideDra TideDra commented Aug 8, 2024

In some multi-node environment like SLURM,there are some environment vars that contain special chars and can trigger errors when being exported.

For example, there is a var SLURM_JOB_CPUS_PER_NODE=64(x2) when requesting two nodes with 64 cpus using SLURM.
Using runner.add_export to export this var will add a command export SLURM_JOB_CPUS_PER_NODE=64(x2) when launching subprocesses, while this will cause a bash error since ( is a key word of bash, like:

[2024-08-07 16:56:24,651] [INFO] [runner.py:568:main] cmd = pdsh -S -f 1024 -w server22,server27 export PYTHONPATH=/public/home/grzhang/code/CLIP-2;  export SLURM_JOB_CPUS_PER_NODE=64(x2); ...
server22: bash: -c: 行 0: 未预期的符号“(”附近有语法错误

This PR simply wrap the environment vars with a pair of " to make sure they are treated as string.

@TideDra TideDra requested a review from awan-10 as a code owner August 8, 2024 05:39
@loadams loadams self-assigned this Aug 19, 2024
@jomayeri jomayeri added this pull request to the merge queue Sep 6, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to no response for status checks Sep 7, 2024
@jomayeri jomayeri added this pull request to the merge queue Sep 7, 2024
Merged via the queue into microsoft:master with commit fc22d96 Sep 8, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants