Update DDP for torch.distributed.run with gloo backend

[glenn-jocher]

@NanoCode012 I'm experimenting with a few DDP updates here.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhanced logging and introduced distributed training modifications.

📊 Key Changes

• Introduced colorstr function to colorize the printed options in detect.py, export.py, and test.py.
• Removed set_logging calls from conditionals to allow logging across different modes.
• Added LOCAL_RANK, RANK, and WORLD_SIZE global variables for distributed data parallel (DDP) support in train.py.
• Refactored train.py to use DDP-friendly WORLD_SIZE, RANK, and LOCAL_RANK variables.
• Modified create_dataloader in utils/datasets.py to support DDP by removing the world_size argument.
• Added dist.barrier() calls in utils/torch_utils.py to synchronize DDP processes.
• Updated the WandbLogger to handle DDP rank conditions using the environment variable RANK.

🎯 Purpose & Impact

• 📈 Improved Logging: Colorized output enhances readability, helping users track the progress and settings easily.
• 💻 DDP Support: Changes create a solid foundation for distributed training, allowing efficient scaling across multiple GPUs and improving training times.
• 🔄 Easier Collaboration: These updates could facilitate collaborative development and training on different systems or clusters.

[NanoCode012]

Hi glenn, a quick run from this branch on the Docker image (commit 382ce4f) gives me the below. It somewhat trains but stopped at the end.

Output

[INFO] 2021-06-18 12:16:14,143 run: Running torch.distributed.run with args: ['/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py', '--master_port', '9980', '--nproc_per_node', '2', 'train.py', '--weights', 'yolov5s.pt', '--epochs', '3', '--img', '320', '--device', '0,1']
[INFO] 2021-06-18 12:16:14,149 run: Using nproc_per_node=2.
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
[INFO] 2021-06-18 12:16:14,150 api: Starting elastic_operator with launch configs:
  entrypoint       : train.py
  min_nodes        : 1
  max_nodes        : 1
  nproc_per_node   : 2
  run_id           : none
  rdzv_backend     : static
  rdzv_endpoint    : 127.0.0.1:9980
  rdzv_configs     : {'rank': 0, 'timeout': 900}
  max_restarts     : 3
  monitor_interval : 5
  log_dir          : None
  metrics_cfg      : {}

[INFO] 2021-06-18 12:16:14,153 local_elastic_agent: log directory set to: /tmp/torchelastic_nxmozyoe/none_tmcpzgfk
[INFO] 2021-06-18 12:16:14,153 api: [default] starting workers for entrypoint: python
[INFO] 2021-06-18 12:16:14,153 api: [default] Rendezvous'ing worker group
[INFO] 2021-06-18 12:16:14,153 static_tcp_rendezvous: Creating TCPStore as the c10d::Store implementation
/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:52: FutureWarning: This is an experimental API and will be changed in future.
  warnings.warn(
[INFO] 2021-06-18 12:16:14,159 api: [default] Rendezvous complete for workers. Result:
  restart_count=0
  master_addr=127.0.0.1
  master_port=9980
  group_rank=0
  group_world_size=1
  local_ranks=[0, 1]
  role_ranks=[0, 1]
  global_ranks=[0, 1]
  role_world_sizes=[2, 2]
  global_world_sizes=[2, 2]

[INFO] 2021-06-18 12:16:14,160 api: [default] Starting worker group
[INFO] 2021-06-18 12:16:14,160 __init__: Setting worker0 reply file to: /tmp/torchelastic_nxmozyoe/none_tmcpzgfk/attempt_0/0/error.json
[INFO] 2021-06-18 12:16:14,161 __init__: Setting worker1 reply file to: /tmp/torchelastic_nxmozyoe/none_tmcpzgfk/attempt_0/1/error.json
github: skipping check (Docker image), for updates see https://github.com/ultralytics/yolov5
{'RANK': 1, 'LOCAL_RANK': 1, 'WORLD_SIZE': 2}
YOLOv5 🚀 v5.0-223-g382ce4f torch 1.9.0+cu102 CUDA:0 (Tesla V100-SXM2-32GB, 32510.5MB)
                                              CUDA:1 (Tesla V100-SXM2-32GB, 32510.5MB)

{'RANK': 0, 'LOCAL_RANK': 0, 'WORLD_SIZE': 2}
Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for 2 nodes.
Namespace(adam=False, artifact_alias='latest', batch_size=8, bbox_interval=-1, bucket='', cache_images=False, cfg='', data='data/coco128.yaml', device='0,1', entity=None, epochs=3, evolve=False, exist_ok=False, hyp='data/hyp.scratch.yaml', image_weights=False, img_size=[320, 320], label_smoothing=0.0, linear_lr=False, multi_scale=False, name='exp', noautoanchor=False, nosave=False, notest=False, project='runs/train', quad=False, rect=False, resume=False, save_dir='runs/train/exp10', save_period=-1, single_cls=False, sync_bn=False, total_batch_size=16, upload_dataset=False, weights='yolov5s.pt', workers=8)
hyperparameters: lr0=0.01, lrf=0.2, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0
tensorboard: Start with 'tensorboard --logdir runs/train', view at http://localhost:6006/
wandb: Install Weights & Biases for YOLOv5 logging with 'pip install wandb' (recommended)

                 from  n    params  module                                  arguments                     
  0                -1  1      3520  models.common.Focus                     [3, 32, 3]                    
  1                -1  1     18560  models.common.Conv                      [32, 64, 3, 2]                
  2                -1  1     18816  models.common.C3                        [64, 64, 1]                   
  3                -1  1     73984  models.common.Conv                      [64, 128, 3, 2]               
  4                -1  1    156928  models.common.C3                        [128, 128, 3]                 
  5                -1  1    295424  models.common.Conv                      [128, 256, 3, 2]              
  6                -1  1    625152  models.common.C3                        [256, 256, 3]                 
  7                -1  1   1180672  models.common.Conv                      [256, 512, 3, 2]              
  8                -1  1    656896  models.common.SPP                       [512, 512, [5, 9, 13]]        
  9                -1  1   1182720  models.common.C3                        [512, 512, 1, False]          
 10                -1  1    131584  models.common.Conv                      [512, 256, 1, 1]              
 11                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 12           [-1, 6]  1         0  models.common.Concat                    [1]                           
 13                -1  1    361984  models.common.C3                        [512, 256, 1, False]          
 14                -1  1     33024  models.common.Conv                      [256, 128, 1, 1]              
 15                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 16           [-1, 4]  1         0  models.common.Concat                    [1]                           
 17                -1  1     90880  models.common.C3                        [256, 128, 1, False]          
 18                -1  1    147712  models.common.Conv                      [128, 128, 3, 2]              
 19          [-1, 14]  1         0  models.common.Concat                    [1]                           
 20                -1  1    296448  models.common.C3                        [256, 256, 1, False]          
 21                -1  1    590336  models.common.Conv                      [256, 256, 3, 2]              
 22          [-1, 10]  1         0  models.common.Concat                    [1]                           
 23                -1  1   1182720  models.common.C3                        [512, 512, 1, False]          
 24      [17, 20, 23]  1    229245  models.yolo.Detect                      [80, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Model Summary: 283 layers, 7276605 parameters, 7276605 gradients, 17.1 GFLOPs

Transferred 362/362 items from yolov5s.pt
Scaled weight_decay = 0.0005
Optimizer groups: 62 .bias, 62 conv.weight, 59 other
train: Scanning '../coco128/labels/train2017' images and labels...128 found, 0 missing, 2 empty, 0 corrupted: 100%|███████████████| 128/128 [00:00<00:00, 1334.50it/s]
train: New cache created: ../coco128/labels/train2017.cache
train: Scanning '../coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|██████████████████| 128/128 [00:00<?, ?it/s]
val: Scanning '../coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|████████████████████| 128/128 [00:00<?, ?it/s][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
val: Scanning '../coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|████████████████████| 128/128 [00:00<?, ?it/s]
val: Scanning '../coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|████████████████████| 128/128 [00:00<?, ?it/s]
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
Plotting labels... 
val: Scanning '../coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|████████████████████| 128/128 [00:01<?, ?it/s]
val: Scanning '../coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|████████████████████| 128/128 [00:01<?, ?it/s]
val: Scanning '../coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|████████████████████| 128/128 [00:01<?, ?it/s]
val: Scanning '../coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|████████████████████| 128/128 [00:01<?, ?it/s]
val: Scanning '../coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|████████████████████| 128/128 [00:01<?, ?it/s]
val: Scanning '../coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|████████████████████| 128/128 [00:01<?, ?it/s]

autoanchor: Analyzing anchors... anchors/target = 3.97, Best Possible Recall (BPR) = 0.9580. Attempting to improve anchors, please wait...
autoanchor: WARNING: Extremely small objects found. 35 of 929 labels are < 3 pixels in size.
autoanchor: Running kmeans for 9 anchors on 927 points...
autoanchor: thr=0.25: 0.9623 best possible recall, 3.58 anchors past thr
autoanchor: n=9, img_size=320, metric_all=0.252/0.634-mean/best, past_thr=0.474-mean: 11,12,  31,34,  74,42,  46,87,  132,90,  78,161,  192,151,  173,273,  305,189
autoanchor: Evolving anchors with Genetic Algorithm: fitness = 0.6718: 100%|█████████████████████████████████████████████████████| 1000/1000 [00:01<00:00, 938.65it/s]
autoanchor: thr=0.25: 0.9925 best possible recall, 3.71 anchors past thr
autoanchor: n=9, img_size=320, metric_all=0.261/0.672-mean/best, past_thr=0.478-mean: 6,6,  10,13,  25,28,  64,45,  43,87,  64,130,  139,118,  184,194,  313,214
autoanchor: New anchors saved to model. Update model *.yaml to use these anchors in the future.

Image sizes 320 train, 320 test
Using 4 dataloader workers
Logging results to runs/train/exp10
Starting training for 3 epochs...

     Epoch   gpu_mem       box       obj       cls     total    labels  img_size
       0/2    0.644G   0.08607   0.03836   0.03987    0.1643       111       320:  12%|██████▏                                          | 1/8 [00:08<01:01,  8.78s/it]Reducer buckets have been rebuilt in this iteration.
       0/2     1.59G   0.07716   0.04663   0.03862    0.1624        86       320: 100%|█████████████████████████████████████████████████| 8/8 [00:11<00:00,  1.41s/it]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|███████████████████████████████████████████| 8/8 [00:03<00:00,  2.04it/s]
                 all        128        929      0.387      0.485      0.404      0.225

     Epoch   gpu_mem       box       obj       cls     total    labels  img_size
       1/2     1.59G   0.06844   0.04651   0.03561    0.1506        80       320: 100%|█████████████████████████████████████████████████| 8/8 [00:02<00:00,  3.29it/s]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|███████████████████████████████████████████| 8/8 [00:02<00:00,  3.32it/s]
                 all        128        929      0.453      0.488       0.46      0.275

     Epoch   gpu_mem       box       obj       cls     total    labels  img_size
       2/2     1.59G   0.06269   0.05292   0.03293    0.1485       150       320: 100%|█████████████████████████████████████████████████| 8/8 [00:02<00:00,  3.02it/s]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|███████████████████████████████████████████| 8/8 [00:04<00:00,  1.80it/s]
                 all        128        929       0.51       0.49      0.493        0.3
3 epochs completed in 0.012 hours.

Optimizer stripped from runs/train/exp10/weights/last.pt, 14.8MB
Optimizer stripped from runs/train/exp10/weights/best.pt, 14.8MB

---

I'll try to check your changes. Right now, I notice a few things.

• my training stuck on the last line with Optimizer stripped.... Perhaps a worker did not end or an update with wandb that I haven't been following recently

Output when Cntrl+C

Optimizer stripped from runs/train/exp10/weights/best.pt, 14.8MB
^CTraceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 637, in <module>
    main()
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 629, in main
    run(args)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 621, in run
    elastic_launch(
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 116, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 238, in launch_agent
    result = agent.run()
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper
    result = f(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py", line 700, in run
    result = self._invoke_run(role)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py", line 828, in _invoke_run
    time.sleep(monitor_interval)
KeyboardInterrupt

• There are a lot of extra logs with the new run module. I wonder if we can hide them.
• Warning: Leaking Caffe2 thread-pool after fork. <- This warning while loading dataset

Unfortunately, since this is quite new, there aren't any examples we can follow.

[NanoCode012]

nccl should be the faster backend for ddp. I recall that Windows only support gloo however.

[glenn-jocher]

@NanoCode012 I see the same hanging at the end of training. This warning seems to imply that the hanging may be related to dist.barrier() we use in torch_distributed_zero_first()

[W ProcessGroupNCCL.cpp:1569] Rank 3 using best-guess GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.

yolov5/utils/torch_utils.py

Lines 27 to 37 in 2729761

	@contextmanager
	def torch_distributed_zero_first(local_rank: int):
	"""
	Decorator to make all processes in distributed training wait for each local_master to do something.
	"""
	if local_rank not in [-1, 0]:
	torch.distributed.barrier()
	yield
	if local_rank == 0:
	torch.distributed.barrier()

[glenn-jocher]

@NanoCode012 agreed there are a lot of warnings/messages everywhere.

Part of this newly appeared in torch 1.9:

/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)

And this part maybe either be due to 1.9 or the Docker image.

[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)

This is just my own debug code to remove before merge.

{'RANK': 0, 'LOCAL_RANK': 0, 'WORLD_SIZE': 2}

The rest of the extra stuff is due to torch.distributed.run

[NanoCode012]

@NanoCode012 I see the same hanging at the end of training. This warning seems to imply that the hanging may be related to dist.barrier() we use in torch_distributed_zero_first()

[W ProcessGroupNCCL.cpp:1569] Rank 3 using best-guess GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.

I think this warning is because of this comment in the run module.
https://github.com/pytorch/pytorch/blob/d5988c5eca0221e9ef58918e4f0b504940cb926a/torch/distributed/run.py#L210-L212

However, I notice when I re-ran the latest code from this branch, I also get this error but not in my earlier run. Perhaps it's something you changed from commit 382ce4f ?

Edit: I think the cause of this error is when you changed to nccl 8ae9ea1 . Checkout to the commit before does not give me that warning. This is quite confusing..

[glenn-jocher]

@NanoCode012 ok I found a fix. torch.destroy_process_group() needs to be outside of the main train() function, and now it works, with both nccl and gloo, using both torch.distributed.launch and torch.distributed.run.

EDIT: I can't profile this right now as there's no EC2 spot availability but I will try to profile this week and we can revert to nccl if need be, or even introduce an if statement with if torch.distributed.is_nccl_available().

[lleye]

Hi @glenn-jocher @NanoCode012 i'm having the same issues here.

[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)

[E ProcessGroupNCCL.cpp:566] [Rank #] Watchdog caught collective operation timeout: WorkNCCL(OpType=BROADCAST, Timeout(ms)=60000) ran for 60960 milliseconds before timing out.

RuntimeError: replicas[0][0] in this process with sizes [48, 12, 3, 3] appears not to match sizes of the same param in process 0.

[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'

training on multiple a100s using ddp in docker

[glenn-jocher]

@glenn-jocher this is a merged PR. If you have an issue that meets the criteria below I recommend you open a new issue with code to reproduce.

How to create a Minimal, Reproducible Example

When asking a question, people will be better able to provide help if you provide code that they can easily understand and use to reproduce the problem. This is referred to by community members as creating a minimum reproducible example. Your code that reproduces the problem should be:

• ✅ Minimal – Use as little code as possible that still produces the same problem
• ✅ Complete – Provide all parts someone else needs to reproduce your problem in the question itself
• ✅ Reproducible – Test the code you're about to provide to make sure it reproduces the problem

In addition to the above requirements, for Ultralytics to provide assistance your code should be:

• ✅ Current – Verify that your code is up-to-date with current GitHub master, and if necessary git pull or git clone a new copy to ensure your problem has not already been resolved by previous commits.
• ✅ Unmodified – Your problem must be reproducible without any modifications to the codebase in this repository. Ultralytics does not provide support for custom code ⚠️.

If you believe your problem meets all of the above criteria, please close this issue and raise a new one using the 🐛 Bug Report template and providing a minimum reproducible example to help us better understand and diagnose your problem.

Thank you! 😃