Self-contained checkpoint --resume

[glenn-jocher]

Make checkpoints fully contained, allow resume directly from any checkpoint.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Improved checkpoint handling and enhanced URL checks in training scripts.

📊 Key Changes

• Hyperparameters (hyp) are now saved with checkpoint for improved reproducibility.
• Added is_url function check to verify if a given string is a valid URL.
• Simplified argument parsing in parse_opt function.
• Enhanced resume function to better handle interrupted training sessions and to deal with URL data sources correctly.
• Updated error messages and logging for clarity when resuming training.

🎯 Purpose & Impact

• The ability to save hyperparameters directly with the checkpoints ensures that all necessary information for reproducing a training session is contained within the checkpoint itself. 🔄
• The improved URL check (is_url) provides more robust validation and ensures that training data can be correctly accessed from remote sources. 🌍
• Simplifying the argument parser makes the code more readable and maintainable. 🧹
• Better resume functionality aids users in restarting interrupted training sessions without hassle, thereby saving time and computational resources. ⏱
• Clearer error messaging and logging enhance the user's experience by providing better guidance and debugging information. 📝

[glenn-jocher]

@kalenmike ok I think has all we need. The idea is you can take one of these new fully-contained partially-trained checkpoints, download it, and then python train.py --resume path/to/partially_trained_checkpoint.pt.

It will create save_dir and resume training from where it left off automatically. For HUB I think what we need on the server/API side is that when ultralytics.start(key) sees an existing partially trained model in the database that it downloads it's last.pt and then runs python train.py --resume last.pt.

So for the user the command to start a new training or resume an existing training would be the same, and it would be up to us to handle them correctly depending on the model status in the database (fully or partially trained).