Error "Directory already existed" happen when training with multiple GPUs

[Jelly123456]

🐛 Bug

When I train my own datasets with multiple GPUs, the ["runs/train/exp*" already exists] error always shows.

To Reproduce (REQUIRED)

Input:

Run the command for multiple GPU training for my own datasets:
python -m torch.distributed.launch --nproc_per_node 2 train.py --batch-size 64 --data data/Allcls_one.yaml --weights weights/yolov5l.pt --cfg models/yolov5l_1cls.yaml --epochs 1 --device 0,1

Output:

Start Tensorboard with "tensorboard --logdir runs/train", view at http://localhost:6006/
Traceback (most recent call last):
  File "***/lib/python3.9/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 196, in makedirs
    os.makedirs(path)
  File "***/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: 'runs/train/exp5'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "***/train.py", line 527, in <module>
    tb_writer = SummaryWriter(opt.save_dir)  # Tensorboard
  File "***/lib/python3.9/site-packages/torch/utils/tensorboard/writer.py", line 221, in __init__
    self._get_file_writer()
  File "***/lib/python3.9/site-packages/torch/utils/tensorboard/writer.py", line 251, in _get_file_writer
    self.file_writer = FileWriter(self.log_dir, self.max_queue,
  File "***/lib/python3.9/site-packages/torch/utils/tensorboard/writer.py", line 61, in __init__
    self.event_writer = EventFileWriter(
  File "***/lib/python3.9/site-packages/tensorboard/summary/writer/event_file_writer.py", line 77, in __init__
    tf.io.gfile.makedirs(logdir)
  File "***/lib/python3.9/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 669, in makedirs
    return get_filesystem(path).makedirs(path)
  File "***/lib/python3.9/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 198, in makedirs
    raise errors.AlreadyExistsError(
tensorboard.compat.tensorflow_stub.errors.AlreadyExistsError: Directory already exists
Traceback (most recent call last):
  File "***/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "***/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "***/lib/python3.9/site-packages/torch/distributed/launch.py", line 260, in <module>
    main()
  File "***/lib/python3.9/site-packages/torch/distributed/launch.py", line 255, in main
    raise subprocess.CalledProcessError(returncode=process.returncode,
subprocess.CalledProcessError: Command '['***/bin/python', '-u', 'train.py', '--local_rank=1', '--batch-size', '64', '--data', 'data/Allcls_one.yaml', '--weights', 'weights/yolov5l.pt', '--cfg', 'models/yolov5l_1cls.yaml', '--epochs', '1', '--device', '0,1']' returned non-zero exit status 1.

Expected behavior

It should not show the "directory already exists" error because there is no such directory (runs/train/exp5) in the "runs" folder.

Environment

If applicable, add screenshots to help explain your problem.

• OS: [Ubuntu]
• GPU [Tesla V100s]

Proposed solution

I solved locally by adding the following one-line code before "tb_writer = SummaryWriter(opt.save_dir)" which is in "train.py"
os.makedirs(opt.save_dir, exist_ok=True); This line code is to create the directory explicitly.
I was inspired by this link:
allenai/allennlp#3843

After adding this code, the error disappears, and I can train successfully with multiple GPUs.

[github-actions]

👋 Hello @Jelly123456, thank you for your interest in 🚀 YOLOv5! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://www.ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@Jelly123456 hi thanks for the bug report and for the proposed solution. I'm pretty confused by the error, as the tensorboard summarywriter can certainly be initialized with an existing directory. For example when resuming an interrupted training run with python train.py --resume then tensorboard picks up where it left off (in the same directory) with no problems.

Maybe the error is produced because the directory does not exist?

To allow us to debug could you provide a minimum reproducible example? Thanks!
https://docs.ultralytics.com/help/minimum_reproducible_example/

BTW a pro tip: the --weights, --cfg, and --data files are searched locally, so if they are unique you can simply pass the filename and we will find it automatically, i.e. as in the colab notebook example here:
https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb#scrollTo=1NcFxRcFdJ_O&line=2&uniqifier=1

[glenn-jocher]

@Jelly123456 also that's strange that your error says file exists. runs/train/exp should be a directory, not a file.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[slimwangyue]

@Jelly123456 Just mkdir yourself before you run the codes and manually denote the --name as the directory you just built. Note that the codes will automatically change --name by appending numbers to it. So be sure to comment that line.

[glenn-jocher]

@slimwangyue @Jelly123456 you can log to the same directory every time using the --exist-ok flag. This prevents the log directory from incrementing:

python train.py --project PROJECT --name NAME --exist-ok