mAP different between YOLO and COCO

[tangzhe1995]

❔Question

Hi, I confuse that the mAP results are different between test.py of YOLOv5 and pycocotools of COCO. Can anyone help with this question?

Namespace(augment=False, batch_size=32, conf_thres=0.001, data='./data/coco.yaml', device='', img_size=640, iou_thres=0.65, save_json=True, save_txt=False, single_cls=False, task='val', verbose=False, weights=['yolov5x.pt'])
Using CUDA device0 _CudaDeviceProperties(name='Tesla P100-PCIE-16GB', total_memory=16280MB)

Fusing layers... Model Summary: 284 layers, 8.89222e+07 parameters, 0 gradients
Scanning labels ../coco/labels/val2017.cache (4952 found, 0 missing, 48 empty, 0 duplicate, for 5000 images): 5000it [00:00, 17761.74it/s]
               Class      Images     Targets           P           R      mAP@.5  mAP@.5:.95: 100% 157/157 [02:34<00:00,  1.02it/s]
                 all       5e+03    3.63e+04       0.409       0.754       0.669       0.476
Speed: 23.6/1.6/25.2 ms inference/NMS/total per 640x640 image at batch-size 32

COCO mAP with pycocotools... saving detections_val2017__results.json...
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.492
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.676
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.534
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.318
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.541
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.633
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.376
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.616
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.670
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.493
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.723
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.812

Thank you.

Additional context

[github-actions]

Hello @tangzhe1995, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Jupyter Notebook , Docker Image, and Google Cloud Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

• Cloud-based AI systems operating on hundreds of HD video streams in realtime.
• Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
• Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

[glenn-jocher]

@tangzhe1995 pycocotools is the official metric of the COCO dataset, i.e. 0.492 above.

For reasons unclear, our in-house mAP calculation is typically about 1-2% below pycocotools results, i.e. 0.476 above.

[tangzhe1995]

@glenn-jocher Thanks for quick reply. I Got it.

[glenn-jocher]

@tangzhe1995 BTW, a recent PR closed this gap a bit, see #1206

[Mary14-design]

Can you share the code you used for the COCO mAP for your official page? @glenn-jocher

[glenn-jocher]

@Mary14-design, we're glad you're interested in the details of our evaluation! The mAP reported on the official page was obtained by using the test.py script provided in the YOLOv5 repository, followed by the COCO API evaluation with the generated results files. Here's a quick overview:

1. Use test.py to evaluate your dataset and save the results:

python test.py --data coco.yaml --weights yolov5x.pt --save-json

2. This script will save a .json file in the yolov5/runs/test/ directory.

3. The saved .json file is then evaluated using the COCO API for the official mAP scores.

This process ensures adherence to COCO's evaluation standards. Hopefully, this clarifies your query! Let me know if you have further questions. 😊