Pre-trained weights not happy, map0.5 plateauing

[edeyejedi]

Hi there,

First ever post from a somewhat inexperienced person in terms of ML (but not a total novice).
I am being cheeky and using 1 post to address 2 problems I am having. The first is priority.

1. When i introduce the YV5 pre-retrained weights in to my training everything comes to a halt due to:
   Traceback (most recent call last):
   File "train.py", line 474, in
   train(hyp, tb_writer, opt, device)
   File "train.py", line 135, in train
   model.load_state_dict(ckpt['model'], strict=False)
   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 847, in load_state_dict
   self.class.name, "\n\t".join(error_msgs)))
   RuntimeError: Error(s) in loading state_dict for Model:
   size mismatch for model.18.weight: copying a param with shape torch.Size([255, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([18, 128, 1, 1]).
   size mismatch for model.18.bias: copying a param with shape torch.Size([255]) from checkpoint, the shape in current model is torch.Size([18]).
   size mismatch for model.22.weight: copying a param with shape torch.Size([255, 256, 1, 1]) from checkpoint, the shape in current model is torch.Size([18, 256, 1, 1]).
   size mismatch for model.22.bias: copying a param with shape torch.Size([255]) from checkpoint, the shape in current model is torch.Size([18]).
   size mismatch for model.26.weight: copying a param with shape torch.Size([255, 512, 1, 1]) from checkpoint, the shape in current model is torch.Size([18, 512, 1, 1]).
   size mismatch for model.26.bias: copying a param with shape torch.Size([255]) from checkpoint, the shape in current model is torch.Size([18]).

I have had this working previously, and I can not find what I have changed to create this issue. I don't know if changes to the code or weights have changed as I run everything on Colab and just pull down the code/weights as required. Despite reverting to a setting that (I think) worked previously I have played, toyed and adjusted settings for hours and I cant seem to get this rolling again. a bit of research tells me there can be a conflict between classes, but I don't know. Any help would be appreciated. TIA

2. I have been running YV5 heaps trying to improve my model training, I plateau around map0.5 ~=0.4. Not exactly desirable. My next steps were to lower the learning rate and perhaps switch the optimizer. Does that sound like a reasonable thing to do? one thing to bare in mind is that i have very small objects - I even get a warning while training that tells me this - thanks. so is there any particular config in YV5 anyone has found particularly effective for small objects.

Thanks heaps

[github-actions]

Hello @edeyejedi, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Jupyter Notebook , Docker Image, and Google Cloud Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note that Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

• Cloud-based AI systems operating on hundreds of HD video streams in realtime.
• Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
• Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

[edeyejedi]

I can add more if requiredL
!python train.py --img 640 --batch 8 --epochs 300 --data '../data.yaml' --cfg /content/mydrive/YOLOv5/custom_yolov5s.yaml --weights yolov5s.pt --name result --nosave --cache

[glenn-jocher]

@edeyejedi thanks for raising the issue. We were able to reproduce your bug by trying to train on voc from pretrained weights:

python train.py --batch 32 --cfg yolov5s.yaml --weights yolov5s.pt --data voc.yaml --img 512 --nosave

We've recently pushed a fix, so please try again with the latest code!

In terms of training, start from https://github.com/ultralytics/yolov5#training and see the custom training tutorial. https://docs.ultralytics.com/yolov5/tutorials/train_custom_data

I don't have time to get into details, but one thing that clearly sticks out is that batch 8 is far too small. We've never given guidance to use small batches like this.

[edeyejedi]

Thank you Glenn.
Ran a quick test and all is working well.
Yeah batch size/model size I need to sort out but I just dropped it down for testing whilst dealing with the mismatch error.
Cheers