Val.py: No prediction on image follows in wrong AP calculation

[UnglvKitDe]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Validation

Bug

The problem exists on images on which the model has not made a prediction.
Then the following lines are called:

yolov5/val.py

Lines 228 to 231 in 29d79a6

	if npr == 0:
	if nl:
	stats.append((correct, *torch.zeros((3, 0), device=device)))
	continue

The problem now is that when the tcls array is calculated here:

yolov5/val.py

Line 264 in 29d79a6

stats = [torch.cat(x, 0).cpu().numpy() for x in zip(*stats)] # to numpy

, these labels are not taken into account. This results in a wrong AP calculation because the number of total positives is wrong. In the example shown below, the length of tcls should be 4, but it is only 1.

Environment

YOLOv5
torch 1.11 and torch 1.12
python 3.8

Minimal Reproducible Example

stats = list()
device = 'cpu'
correct = torch.zeros(1, 10, dtype=torch.bool, device=device)
stats.append((correct, *torch.zeros((3, 1), device=device)))
for _ in range(3):
  correct = torch.zeros(0, 10, dtype=torch.bool, device=device)
  stats.append((correct, *torch.zeros((3, 0), device=device)))
[torch.cat(x, 0).cpu().numpy() for x in zip(*stats)]

Additional

Possible Solutionin line 230 in val.py:
stats.append((correct, *torch.zeros((2, 0), device=device), labels[:, 0]))

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[glenn-jocher]

@UnglvKitDe hi, thank you for your feature suggestion on how to improve YOLOv5 🚀!

Maybe this can improve our mAP, which is still slightly off from pycocotools on COCO. Can you please submit a PR with this fix and provide some before and after results on COCO? If you don't have COCO just submit the PR and we can test on our side. Thanks!

Please see our ✅ Contributing Guide to get started.