Implement --save-period locally

[glenn-jocher]

This PR adds a new training argument --save-period to save training checkpoints every x epochs. To save training every 50 epochs for example:

python train.py --save-period 50  # saves epoch50.pt, epoch100.pt, epoch150.pt, ... etc.

This saved checkpoints in addition to existing last.pt and best.pt checkpoints and does not affect their behavior. Default value is -1, i.e. disabled.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Implementing periodic checkpoint saving during model training in YOLOv5.

📊 Key Changes

• Added code to save model checkpoints periodically, controlled by a new command-line argument called --save-period.
• Organized Weights & Biases (W&B) related arguments under a separate comment in the argument parser for better readability.

🎯 Purpose & Impact

• This change provides flexibility by allowing users to save model checkpoints at specified intervals (--save-period), improving resource management and providing intermediate points to resume training if necessary.
• Cleanup and organization of W&B related code enhances maintainability and clarity for developers working with these options.
• Users now have the option to potentially avoid large gaps in recovery points during long training runs, reducing the risk of significant data loss in case of unexpected interruptions. 🛡️

[glenn-jocher]

@AyushExel FYI this PR applies the W&B --save_period argument (renamed to --save-period no underscore to conform to the rest of the YOLO arguments but turns into underscore by argparser) to local checkpointing as well. Should not affect W&B ops.

[glenn-jocher]

Verified correct operation in Colab.

NOTE: 'Epoch' checkpoints are not optimizer-stripped, so will be larger than last.pt and best.pt upon training completion. This is intended behavior to allow them to be used to --resume a run, i.e. python train.py --resume path/to/epoch50.pt.

[AyushExel]

@glenn-jocher awesome. Now, both cloud based and local checkpointing are aligned!!
Just to confirm, last.pt will still be the latest checkpoint updated after after every epoch right?

[glenn-jocher]

@AyushExel yes that's correct. last.pt and best.pt are not affected by this change.