stacking in validation

[ShAmoNiA]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Validation

Bug

Greeting,
I attempted to run YOLO in evaluation mode, but unfortunately, every time I executed it, it became stuck in the validation process after a few generations, resulting in the following issue:

  Epoch    GPU_mem   box_loss   obj_loss   cls_loss  Instances       Size
    9/9      1.79G    0.06118    0.08289    0.04812         59        640: 100%|##########| 16/16 [00:01<00:00, 10.35it/s]
             Class     Images  Instances          P          R      mAP50   mAP50-95:  75%|#######5  | 6/8 [00:00<00:00,  9.51it/s]

I ran it with this command:

nohup python train.py --data coco128.yaml --weights 'yolov5s.pt' --cfg yolov5s.yaml --batch-size 8 --name GA --epoch 10 --evolve 500  &> GA_s.out &

How can I go about fixing this issue?

Environment

yolov5 - rtx 2080ti

Minimal Reproducible Example

No response

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[ShAmoNiA]

@glenn-jocher would you please assist on this topic?

[glenn-jocher]

@ShAmoNiA hello! It seems that the training process got stuck in the final validation process. This sometimes occurs if an error in the annotations is present or if the validation set is too large. To prevent this, you can try setting smaller --batch-size. Let me know if this helps!

[ShAmoNiA]

@glenn-jocher It is coco128 dataset. I don't think this issue was about the validation size.
I Changed the batch size to 2; However this issue happened again.

 Epoch    GPU_mem   box_loss   obj_loss   cls_loss  Instances       Size
      19/19     0.554G    0.05693    0.06772    0.05346          9        640: 100%|##########| 64/64 [00:02<00:00, 26.88it/s]
                 Class     Images  Instances          P          R      mAP50   mAP50-95:  88%|########7 | 28/32 [00:00<00:00, 34.79it/s]

[glenn-jocher]

@ShAmoNiA If you've already tried changing the batch size and it's still getting stuck, it could be an issue with the dataset itself. Have you checked that your image files are in the correct format and that each image is correctly annotated with at least one object? It might also be helpful to double-check that the annotations are correct and that there are no syntax errors or typos in your annotation file. Another thing to try would be to verify that the images and annotations are aligned by using a visualization tool to look at the images and their corresponding annotations.

In addition, I recommend reviewing the YOLOv5 documentation for advice on training issues and performance troubleshooting. Additionally, there are some performance-related tweaks you can make to speed up the training, such as using mixed precision training (--amp) and using larger batch sizes (--batch-size). Let me know if you have any other questions or concerns!

[ShAmoNiA]

Thank you very much for your response glenn

[glenn-jocher]

@ShAmoNiA you're welcome! If you have any further questions or run into issues, feel free to ask. I will do my best to help.