train_batch.jpg labels missing

[mfruhner]

🐛 Bug

Hello, on my custom dataset I noticed some missing labels in random images in train_batch*.jpg images. I then tried it out with coco128 and i noticed the same behaviour. Most labels are correct, but some are just missing. It always appears to be a complete mosaic (or rect for --rect) without labels inside a batch. See the following images with the cake and the dog/elephant:

To Reproduce (REQUIRED)

Download latest docker image.
Run:

python train.py --img 640 --batch 16 --epochs 1 --data coco128.yaml --weights yolov5s.pt --nosave --cache

Output:

Some images are missing their labels.
The questions is also, if it is just a rendering bug, or if the missing labels are contributing to the training processing, resulting in inferior results.

Expected behavior

All images have their correct bounding boxes

Environment

If applicable, add screenshots to help explain your problem.

• OS: Ubuntu, Docker Image

[github-actions]

Hello @mfruhner, thank you for your interest in 🚀 YOLOv5! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://www.ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab Notebook with free GPU:
• Kaggle Notebook with free GPU: https://www.kaggle.com/ultralytics/yolov5
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Docker Image https://hub.docker.com/r/ultralytics/yolov5. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@mfruhner thanks for the bug report! I'd noticed the same behavior recently. The mosaic plotting function has undergone several recent updates, so it may be a bug there, perhaps related to de-normalization of the labels. I will add a TODO here for us to try to reproduce and find a fix.

In any case, this is almost certainly a plotting-related issue, as it only appeared within the last several weeks, and the COCO128 labels have never changed.

[glenn-jocher]

TODO: reproduce and fix mosaic plotting bug.

[glenn-jocher]

@mfruhner I can reproduce this issue in a Colab notebook by running your command after running the setup cell. I can see in train_batch2.jpg specifically an image for which many objects have labels, like train, refrigerator, that are not being plotted properly. I will try to figure out why this is happening.

[glenn-jocher]

Ok I think I see the problem. A check is made to determine if the labels are in normalized coordinates before scaling them up to pixels. The check is failing sometimes due to numerical precision issues it seems:

True 1.0
True 1.0
True 1.0
True 1.0
True 1.0
True 1.0
True 1.0
True 1.0
True 1.0
True 1.0
True 1.0
True 1.0
False 1.0000001
True 1.0
True 1.0

I think the solution is to introduce an eps term to account for precision issues, perhaps with a default value of eps=0.01.

[glenn-jocher]

Also I should add, that comically the result is that all the boxes are indeed plotted all of the time, but the 'missing' boxes are simply plotted entirely inside the 0,0 pixel.

[glenn-jocher]

Problem appears solved now in PR branch:

[glenn-jocher]

@mfruhner problem should be resolved now after PR #1647. Thank you for your contributions, and please let us know if you spot any other problems or areas for improvement!

[ozett]

i have a dataset.yaml and observe the same effect that no labels are plotted on the mosaic while training.

i assume from this issue that one of my 800 .txt files for annotation has an error...
(maybe because of using a buggy annotation tool ?).

but how do i check this? i thought a fix for checking wrong labeling coordinates was already made into yolov5..

my labels.txt: labels.zip

...

root@gpu-2:/usr/src/yolov5/data# cat dataset800.yaml
#names:
#- marlo-tuxedo
#- charly-black
#nc: 2
nc: 83   # count starts with 0             // glen-jocher: this is very simple. 25 classes are zero indexed 0-24 in your labels.txt files.
path: /usr/src/datasets/dataset800
train: images/train
val: images/val
# names: ['marlo-tuxedo', 'charly-black']

names: ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light',
        'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow',
        'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee',
        'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard',
        'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple',
        'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch',
        'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone',
        'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear',
        'hair drier', '79', '80', 'marlo-tuxedo' , 'charly-black']  # class names
root@gpu-2:/usr/src/yolov5/data#

[ozett]

wanted to crosscheck my yaml and dataset with training on coco128.yaml defaults.

python3 train.py --data data/coco128.yaml --epochs 2 --project custom_yolo5_126 --bbox_interval 1 --save-period 1 --weights yolov5m.pt --batch-size 14

no labels plotting on coco128 also...
something broken?
what do i check to fix this?

edit: did a new checkout from git for yolo. did not help.
maybe something with my pytorch version on Cuda 11.6 -> #5815 (comment)

or simply my misunderstanding?
to clarifiy: i am missing "label-names" on the train-mosaic plot. "label-numbers" are there.

but maybe no bug at all - normal?

instead only later (after training) "label-names" appear only the validation-mosaic?
looks more like this ...

[glenn-jocher]

@ozett validation is plotting names and training is plotting class indices.

[ozett]

Yes, thanks. yolov5 works perfectly and fast.

as an ML-baby i still have to take my first steps and gain more experience.
thanks for the confirmation. i am now trying to go the next step.

neil amstrong reverse, so to say: a giant leap for mankind, but a small step for a ML-baby

[ecarrenolozano]

Hi dear community,

I'm experiencing the same issue, labels are plotted just in validation, not in training.
Implementation details:

• Yolov5 version:
  Your branch is up to date with 'origin/master'. commit 5c91daeaecaeca709b8b6d13bd571d068fdbd003 (HEAD -> master, origin/master, origin/HEAD) Author: Glenn Jocher <glenn.jocher@ultralytics.com> Date: Mon Feb 27 19:55:23 2023 -0800
• Dataset: coco128

Training: train_batch0.jpg

Validation: val_batch0.jpg

Is this a normal and intended behaviour or there is a bug?

Thanks in advance!