DDP Multi-GPU --resume bug fix

[glenn-jocher]

Multi-GPU --resume bug fix for issue #851. This retains the values of opt.global_rank and opt.local_rank when DDP training is resumed.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Improved resume functionality in YOLOv5 training script.

📊 Key Changes

• Introduced code to ensure distributed training settings are correctly retained when resuming training from a checkpoint.

🎯 Purpose & Impact

• The change fixes a potential issue where training resumption could disregard the distributed training rank settings.
• Users relying on distributed training will experience a more consistent and reliable resumption of their training process, particularly in environments using multiple GPUs or nodes. 🔄
• Makes the process of resuming training smoother and less error-prone, which can save time and computational resources. 💻🕒

[glenn-jocher]

Running test on 2x T4 GCP VM. Test scenario:

# 1. Start DDP training
python -m torch.distributed.launch --nproc_per_node 2 train.py --batch-size 80 --data coco128.yaml --epochs 50 --weights yolov5s.pt

# 2. sudo reboot after about 25 epochs

# 3. Resume DDP training
python -m torch.distributed.launch --nproc_per_node 2 train.py --resume

Test results as expected, training resumed properly from epoch 25 after forced reboot. nvidia-smi check confirmed multi-GPUs in use. Everything seems to work. Note not tested with W&B enabled.

[glenn-jocher]

CI passing and resume test scenario passing. Merging.

To --resume with DDP (2 GPUs)

ONLY add --resume or --resume path/to/last.pt. --resume accepts no other arguments.

python -m torch.distributed.launch --nproc_per_node 2 train.py --resume