Implementation of Early Stopping for DDP training

[giacomoguiduzzi]

This edit correctly uses the broadcast_object_list function to send slave processes a bool so to end the training phase if the variable is True, thus allowing the master process to destroy the process group and terminate.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Implementation of a unified early stopping mechanism for both single-GPU and DDP training in the YOLOv5 model.

📊 Key Changes

• 🤖 Added a single stop flag alongside the existing EarlyStopping object for better early stop control.
• 🔄 The early stopping check now updates the stop flag which is used consistently across the training loop.
• 🚀 Optimized the training loop to use the new stop flag, removing previous early stop code for single-GPU and DDP.
• 📡 Implemented broadcasting the stop flag in DDP (Distributed Data Parallel) training to ensure all processes stop simultaneously.

🎯 Purpose & Impact

• 🎓 Provides a cleaner, more consistent early stopping implementation for developers and researchers.
• 🌐 Enhances the integrity of distributed training by ensuring all training processes stop at the same time when the criteria are met.
• 🏃‍♂️ May speed up training by preventing unnecessary epochs, leading to faster experimentation cycles for users.
• 🔍 Ensures more robust training across different training setups, potentially improving the reliability of model training for users.

[github-actions]

👋 Hello @giacomoguiduzzi, thank you for submitting a YOLOv5 🚀 PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

• ✅ Verify your PR is up-to-date with upstream/master. If your PR is behind upstream/master an automatic GitHub Actions merge may be attempted by writing /rebase in a new comment, or by running the following code, replacing 'feature' with the name of your local branch:

git remote add upstream https://github.com/ultralytics/yolov5.git
git fetch upstream
# git checkout feature  # <--- replace 'feature' with local branch name
git merge upstream/master
git push -u origin -f

• ✅ Verify all Continuous Integration (CI) checks are passing.
• ✅ Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." -Bruce Lee

[glenn-jocher]

@giacomoguiduzzi thanks for the PR! Is there any way to drop some or all of this code into the stopper() method itself? The class can access the global RANK variable by defining it in utils/torch_utils if required, i.e.:

RANK = int(os.getenv('RANK', -1))

[giacomoguiduzzi]

@giacomoguiduzzi thanks for the PR! Is there any way to drop some or all of this code into the stopper() method itself? The class can access the global RANK variable by defining it in utils/torch_utils if required, i.e.:

RANK = int(os.getenv('RANK', -1))

Hi @glenn-jocher, no problem! I think if you wanted to drop this code into the EarlyStopper __call__ method you'd need every device to call that function. At the moment, on line 438 the instruction stop = stopper(epoch=epoch, fitness=fi) is inside an if branch that is only executed by the master process. If you move this call outside of the if branch, so that every device calls the stopper() method, then it is possible for every device to execute the broadcast_object_list() function inside the EarlyStopper class. Looking at the definition of the stopper variable on line 299 it looks like every process has it defined already. I can see though that the EarlyStopper class uses the fi variable that is currently computed only by the master device as it is the result of the evaluation phase.

Summing up, I think it could be possible but it is necessary to broadcast the fi variable in the __call__ method, so that every process is able to compute it's stop boolean value. If this is done it is not necessary to broadcast the stop variable anymore as the slave processes are now synchronized and every one of them will have stop = True if it is the case, thus every process will terminate with the master.

Let me know if you want me to look into it.

[glenn-jocher]

@giacomoguiduzzi I've cleaned up the PR a bit while maintaining the functionality, I think. Can you test on your side to verify that everything still works correctly? If it all looks good after your review I will proceed to merge. Thanks!

[glenn-jocher]

@giacomoguiduzzi further cleaned up in 58bc763. I think this is ok but have not tested with DDP earlystopping.

[giacomoguiduzzi]

Hi @glenn-jocher, I've just tested your edits and the early stopping feature is working as intended. You're right, my code wasn't very pythonic...

[glenn-jocher]

@giacomoguiduzzi got it! PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐