mAP when testing is very different with pycocotools

[itruonghai]

I try to test my model on
!python test.py --weights /content/drive/My\ Drive/0TruongHai/TA_YOLOV5/exp0/weights/best.pt --data mydata.yaml --img 800 --task test --verbose --save-json
But the results of mAP@.5:.95 benchmark seem to be very differents

Namespace(augment=False, batch_size=32, conf_thres=0.005, data='./data/coco128.yaml', device='', img_size=800, iou_thres=0.65, save_conf=False, save_dir='runs/test', save_json=True, save_txt=False, single_cls=False, task='test', verbose=True, weights=['/content/drive/My Drive/0TruongHai/TA_YOLOV5/exp0/weights/best.pt'])
Using CUDA device0 _CudaDeviceProperties(name='Tesla P100-PCIE-16GB', total_memory=16280MB)

Fusing layers... 
Model Summary: 284 layers, 8.8431e+07 parameters, 0 gradients
Scanning labels /content/dataset/labels/test.cache (450 found, 0 missing, 0 empty, 10 duplicate, for 450 images): 450it [00:00, 10969.01it/s]
               Class      Images     Targets           P           R      mAP@.5  mAP@.5:.95: 100% 15/15 [00:15<00:00,  1.03s/it]
                 all         450    1.11e+03        0.54       0.689       0.722       0.476 <---
                   1         450         153       0.502        0.68       0.731       0.459
                   2         450         203       0.587       0.818       0.842       0.581
                   3         450          57       0.763       0.684       0.745       0.502
                   4         450         106       0.421       0.726       0.701       0.505
                   5         450         185       0.434       0.573        0.59       0.382
                   6         450         308       0.642       0.662        0.73       0.451
                   7         450         102       0.431       0.676       0.718       0.454
Speed: 21.0/1.9/22.9 ms inference/NMS/total per 800x800 image at batch-size 32

COCO mAP with pycocotools... saving runs/test/detections_val2017_best_results.json...
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.94s).
Accumulating evaluation results...
DONE (t=0.21s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.309 <---
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.622
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.267
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.244
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.627
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.606
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.295
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.369
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.371
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.312
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.675
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.610
Results saved to runs/test

As comparision, 0.309 in pycocotools and 0.47 with the default evaluator.

[github-actions]

Hello @itruonghai, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Jupyter Notebook , Docker Image, and Google Cloud Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

• Cloud-based AI systems operating on hundreds of HD video streams in realtime.
• Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
• Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

[ZwNSW]

@itruonghai Hello, how did you achieve AR printing to the terminal? I tried it for a long time without realizing it. Looking forward to your reply. Thank you

[itruonghai]

@itruonghai Hello, how did you achieve AR printing to the terminal? I tried it for a long time without realizing it. Looking forward to your reply. Thank you

just --save-json and it will use pycocotools

[ZwNSW]

@itruonghai But I But I ran into this problem。
“COCO mAP with pycocotools... saving detections_val2017__results.json...
ERROR: pycocotools unable to run: invalid literal for int() with base 10: 'Image_20200930140952222”
can you help me?

[itruonghai]

@itruonghai But I But I ran into this problem。
“COCO mAP with pycocotools... saving detections_val2017__results.json...
ERROR: pycocotools unable to run: invalid literal for int() with base 10: 'Image_20200930140952222”
can you help me?

maybe use change name of image?

[ZwNSW]

@itruonghai Changing the name of the picture is useless. Are you using the COCO dataset?

[itruonghai]

@itruonghai Changing the name of the picture is useless. Are you using the COCO dataset?

the image name is just id only, not have String.

[ZwNSW]

@itruonghai Now I encountered this problem.
"COCO mAP with pycocotools... saving runs\test\detections_val2017_best_results.json...
ERROR: pycocotools unable to run: list index out of range"

[itruonghai]

@itruonghai try to change the link of json file in test.py

[ZwNSW]

@itruonghai Sorry, I don't quite understand what you mean. Can you be more careful?

[glenn-jocher]

@itruonghai pycocotools is only intended for COCO dataset mAP. With 7 classes you don't appear to meet that constraint.

[glenn-jocher]

@itruonghai upon review, I think this may be related to a recent update to our mAP code in #1206

Can you try to update these two lines in utils/general.py to see if the pycocotools discrepancy remains?

yolov5/utils/general.py

Lines 331 to 334 in 15a1060

	# Append sentinel values to beginning and end
	mrec = recall # np.concatenate(([0.], recall, [recall[-1] + 1E-3]))
	mpre = precision # np.concatenate(([0.], precision, [0.]))

You should update them to this:

    # Append sentinel values to beginning and end
    mrec = np.concatenate(([0.], recall, [recall[-1] + 1E-3]))
    mpre = np.concatenate(([0.], precision, [0.]))

Let me know the results after this test please!

[summit1993]

I try to test my model on
!python test.py --weights /content/drive/My\ Drive/0TruongHai/TA_YOLOV5/exp0/weights/best.pt --data mydata.yaml --img 800 --task test --verbose --save-json
But the results of mAP@.5:.95 benchmark seem to be very differents

Namespace(augment=False, batch_size=32, conf_thres=0.005, data='./data/coco128.yaml', device='', img_size=800, iou_thres=0.65, save_conf=False, save_dir='runs/test', save_json=True, save_txt=False, single_cls=False, task='test', verbose=True, weights=['/content/drive/My Drive/0TruongHai/TA_YOLOV5/exp0/weights/best.pt'])
Using CUDA device0 _CudaDeviceProperties(name='Tesla P100-PCIE-16GB', total_memory=16280MB)

Fusing layers... 
Model Summary: 284 layers, 8.8431e+07 parameters, 0 gradients
Scanning labels /content/dataset/labels/test.cache (450 found, 0 missing, 0 empty, 10 duplicate, for 450 images): 450it [00:00, 10969.01it/s]
               Class      Images     Targets           P           R      mAP@.5  mAP@.5:.95: 100% 15/15 [00:15<00:00,  1.03s/it]
                 all         450    1.11e+03        0.54       0.689       0.722       0.476 <---
                   1         450         153       0.502        0.68       0.731       0.459
                   2         450         203       0.587       0.818       0.842       0.581
                   3         450          57       0.763       0.684       0.745       0.502
                   4         450         106       0.421       0.726       0.701       0.505
                   5         450         185       0.434       0.573        0.59       0.382
                   6         450         308       0.642       0.662        0.73       0.451
                   7         450         102       0.431       0.676       0.718       0.454
Speed: 21.0/1.9/22.9 ms inference/NMS/total per 800x800 image at batch-size 32

COCO mAP with pycocotools... saving runs/test/detections_val2017_best_results.json...
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.94s).
Accumulating evaluation results...
DONE (t=0.21s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.309 <---
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.622
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.267
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.244
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.627
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.606
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.295
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.369
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.371
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.312
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.675
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.610
Results saved to runs/test

As comparision, 0.309 in pycocotools and 0.47 with the default evaluator.

there is a bug in test.py, in line 171, change xy center to top-left corner, it is different from coco box format, delete it may be ok.

[itruonghai]

@summit1993 format of coco is true in the github, if change like you, it different from coco format and the result gets 0.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.