Fine tuning my custom dataset on already trained yolo

[usamatariq70]

❔Question

I trained my yolov5 and I had 7000 images with 33000 annotations. Now I want to finetune my that model and I have almost 5000 images and annotations.
Can you please guide me which layers should I freeze to finetune it and how?

Additional context

[glenn-jocher]

@usamatariq70 to start training from a previous model simply pass it to the --weights argument:

python train.py --data custom.yaml --weights path/to/ckpt.pt

[glenn-jocher]

@usamatariq70 for freezing layers see Transfer Learning with Frozen Layers Tutorial:

YOLOv5 Tutorials

• Train Custom Data  🚀 RECOMMENDED
• Weights & Biases Logging  🌟 NEW
• Multi-GPU Training
• PyTorch Hub  ⭐ NEW
• ONNX and TorchScript Export
• Test-Time Augmentation (TTA)
• Model Ensembling
• Model Pruning/Sparsity
• Hyperparameter Evolution
• Transfer Learning with Frozen Layers  ⭐ NEW
• TensorRT Deployment

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[SISTMrL]

@usamatariq70 to start training from a previous model simply pass it to the --weights argument:

python train.py --data custom.yaml --weights path/to/ckpt.pt

just this command, it doesn't need to modify learning rate or other parameters? thanks!!!! looking forward to your reply

[glenn-jocher]

@SISTMrL 👋 Hello! Thanks for asking about resuming training. YOLOv5 🚀 Learning Rate (LR) schedulers follow predefined LR curves for the fixed number of --epochs defined at training start (default=300), and are designed to fall to a minimum LR on the final epoch for best training results. For this reason you can not modify the number of epochs once training has started.

If your training was interrupted for any reason you may continue where you left off using the --resume argument. If your training fully completed, you can start a new training from any model using the --weights argument. Examples:

Resume Single-GPU

You may not change settings when resuming, and no additional arguments other than --resume should be passed, with an optional path to the checkpoint you'd like to resume from. If no checkpoint is passed the most recently updated last.pt in your yolov5/ directory is automatically found and used:

python train.py --resume  # automatically find latest checkpoint (searches yolov5/ directory)
python train.py --resume path/to/last.pt  # specify resume checkpoint

Resume Multi-GPU

Multi-GPU DDP trainings must be resumed with the same GPUs and DDP command, i.e. assuming 8 GPUs:

python -m torch.distributed.launch --nproc_per_node 8 train.py --resume  # resume latest checkpoint
python -m torch.distributed.launch --nproc_per_node 8 train.py --resume path/to/last.pt  # specify resume checkpoint

Start from Pretrained

If you would like to start training from a fully trained model, use the --weights argument, not the --resume argument:

python train.py --weights path/to/best.pt  # start from pretrained model

Good luck and let us know if you have any other questions!

[dbkalaria]

@usamatariq70 to start training from a previous model simply pass it to the --weights argument:

python train.py --data custom.yaml --weights path/to/ckpt.pt

@glenn-jocher I've tried fine-tuning my custom model using this command but instead of fine-tuned model I get a newly trained model.

scenario: I've trained a model on 3 classes with 2 classes containing around 2500 images and 3rd class with only 500 images due to lack of data. Later I've gathered 1700 more for class 3 and fine-tuned my model on only images of 3rd class and while evaluating the fine-tuned model, it only predicts the 3rd class and some false positives for other two classes (predicts on 3rd class).

Can you please tell me what went wrong?

[glenn-jocher]

@dhruv312 everything is correct. You must train on all classes you expect your model to predict in deployment.

[dbkalaria]

@glenn-jocher so in fine-tuning also I need to give all the classes right?

[glenn-jocher]

@dhruv312 yes, during training the model is updated on the training data and nothing else. Whatever it's learned in the past is not actively retained, in fact quite the opposite weight decay by itself will bring all weights to zero even with zero losses.

[lchunleo]

can i check when we do the training on the pretrained model, are the layers all frozen? checking on the training code, it seem to me that the layers are not unfrozen, pls correct me if am wrong. thank you

[glenn-jocher]

@lchunleo no layers are frozen by default, they are all trainable, but you can use --freeze when training to freeze part of the model. See Transfer Learning with Frozen Layers Tutorial for details.

Tutorials

• Train Custom Data  🚀 RECOMMENDED
• Tips for Best Training Results  ☘️
  RECOMMENDED
• ClearML Logging 🌟 NEW
• Weights & Biases Logging
• Roboflow for Datasets, Labeling, and Active Learning  🌟 NEW
• Multi-GPU Training
• PyTorch Hub  ⭐ NEW
• TFLite, ONNX, CoreML, TensorRT Export 🚀
• Test-Time Augmentation (TTA)
• Model Ensembling
• Model Pruning/Sparsity
• Hyperparameter Evolution
• Transfer Learning with Frozen Layers  ⭐ NEW
• Architecture Summary  ⭐ NEW

Good luck 🍀 and let us know if you have any other questions!