Evaluation on torch hub model

[joangog]

Hi I have a question regarding the yolov5s model from torchhub. I'm trying to evaluate the model and other torchvision models (e.g. SSD, Faster RCNN) in a similar way to val.py. I tried using the evaluation function (with some modifiications) from pytorch/vision/references/detection/engine.py and ended up with the following:
https://github.com/joangog/object-detection-assets/blob/main/scripts/engine.py
The relevant function is evaluate() which uses a COCOeval object to calculate different metrics such as MAP for each IoU.
The thing is, when I evaluated the model with this function I got different results when compared to val.py (map=0.69 and map=0.73 respectively for my dataset). For the life of me, I can't find the exact formula that pycocotools uses in their code to calculate the metrics. Does someone know what the difference is in how pycocotools calculates the metrics versus how yolov5 does so?

I'm dropping a few more links in case they prove to be useful:

• Aux functions used in engine.py for handling pycocotools COCOeval object: https://github.com/joangog/object-detection-assets/blob/main/scripts/coco_eval.py
• Definition of COCOeval object: https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py
• Yolov5 metrics: https://github.com/ultralytics/yolov5/blob/master/utils/metrics.py

[glenn-jocher]

@joangog see #2258

We have an open issue with a mAP misalignment with pycocotools. The result is that pycocotools will evaluate to higher mAP on the COCO dataset than we do using the internal YOLOv5 mAP functions in the repo, despite both tools using the exact same predictions.

[joangog]

I see... Weirdly it is the opposite for my dataset (Medical masks on faces). I get higher AP with the yolov5 evaluation than the coco one. I'm going to check out the issue and see if I can find anything.

[glenn-jocher]

@joangog any info you can find would be helpful.

[Audrey528]

@joangog Do you find anything useful?

[joangog]

@Audrey528
No unfortunately...I decided to stick to coco evaluation so that my analysis on the comparison to torchvision's models is more reliable. If I have time I might try to search further but for now that's what I'm doing.