Why the detect result of pytorch model is different from onnx model？It seems different deep learning framework has different output

[zoubaihan]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hello, here is a question that confuse me a lot. I download the model yolov5m.onnx and yolov5m.pt , and the test command is as follow:

onnx model tested on COCO:
python detect.py --source /datasets/coco128/images/train2017 --weights /data/zoubaihan/pyproject/yolov5/yolov5m.onnx

One of the image detected by onnx model is as follow:

pytorch model tested on COCO:
python detect.py --source /datasets/coco128/images/train2017 --weights /data/zoubaihan/pyproject/yolov5/yolov5m.pt

While the same input image detected by pytorch model is as follow:

Jesus! Why the same model and the same weight under different deeplearning framework output different results?

I have also tried to convert some other deep learning models between different deep learning frameworks, but the output results are often different, or even very different. Could you please explain why this is?

I also tried to use export.py to convert model from pytorch to onnx, but their results is also not equal. The command is as follow:
python export.py --weights yolov5m.pt --include onnx

But another confusing problem came out: the file size of the output onnx model is not equal to the onnx model I download from your github. Is the yolov5m.onnx model you put in the release really converted by you by running export.py?

Additional

I tried the export.py and other convert tools, but all not work well. Their output results are not the same.
Anybody know why?
Here is the collection of model converter:
https://github.com/ysh329/deep-learning-model-convertor

[github-actions]

👋 Hello @zoubaihan, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@zoubaihan anecdotal results are not very meaningful. We have a benchmarking branch that validates and profiles exported models on a validation set. All export formats that are working here on CPU return near-identical mAPs:

https://github.com/ultralytics/yolov5/blob/updates/benchmarks/utils/benchmarks.py

benchmarks: weights=/usr/src/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/usr/src/yolov5/data/coco128.yaml
...

Benchmarks complete (782.19s)
                   Weights  mAP@0.5:0.95  Inference time (ms)
0               yolov5s.pt      0.407554           102.777191
1      yolov5s.torchscript      0.402908           132.848348
2             yolov5s.onnx      0.402908            89.061137
3   yolov5s_openvino_model      0.402908            67.093970
4                   engine           NaN                  NaN
5                   coreml           NaN                  NaN
6      yolov5s_saved_model      0.402908           133.983964
7               yolov5s.pb      0.402908           101.405423
8      yolov5s-fp16.tflite      0.402851           502.289245
9                  edgetpu           NaN                  NaN
10                    tfjs           NaN                  NaN

[knwng]

Hi @glenn-jocher , thanks for the information! I've just faced the same problem. Do you know what mechanism in onnx probably lower the mAP?(although it's not that much). I used to consider pt2onnx as a lossless conversion.

[zoubaihan]

According to my experiments, almost all deep learning model conversion tools on github will cause changes in model performance. I don't know why they insist their models all perform the same... There are also many people who have raised the similar issue on github, hoping to attract attention!

Here is the collection of model converter:
https://github.com/ysh329/deep-learning-model-convertor

[glenn-jocher]

@zoubaihan not sure what's up with the thumbs down. Is benchmarking every single export format for mAP and speed a bad thing?

[glenn-jocher]

@knwng the main difference I can think of is that PyTorch models are capable of rectangular inference while the export formats are at 640x640, so there are padding differences on the images that are passed to the models.

Also note that benchmarks.py runs on COCO128 by default for speed. If you truly want to know COCO mAP you can simply run it with --data coco.yaml, though since these are CPU benchmarks you will be waiting much longer for the results.

[glenn-jocher]

@zoubaihan @knwng good news 🙂!! I've confirmed that YOLOv5 produces identical exports. The only difference with PyTorch inference as I thought earlier was the rectangular inference, this was the cause of the mAP difference. I've updated benchmarks to force square inference for all formats and now mAP is identical.

See PR #6613 for details.

Colab++ High-RAM CPU Results

benchmarks: weights=/content/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml
Checking setup...
YOLOv5 🚀 v6.0-282-g8310a17 torch 1.10.0+cu111 CPU
Setup complete ✅ (8 CPUs, 51.0 GB RAM, 44.7/166.8 GB disk)

Benchmarks complete (637.94s)
                   Format  mAP@0.5:0.95  Inference time (ms)
0                 PyTorch      0.402908           111.652056
1             TorchScript      0.402908           142.402692
2                    ONNX      0.402908            64.537143
3                OpenVINO      0.402908            69.528472
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel      0.402908           150.990861
7     TensorFlow GraphDef      0.402908           123.970838
8         TensorFlow Lite      0.402851           229.984051
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

MacOS Intel CPU Results (CoreML-capable)

benchmarks: weights=/Users/glennjocher/PycharmProjects/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/Users/glennjocher/PycharmProjects/yolov5/data/coco128.yaml
Checking setup...
Setup complete ✅ (8 CPUs, 32.0 GB RAM, 750.5/931.6 GB disk)
YOLOv5 🚀 v6.0-282-g8310a17 torch 1.10.0 CPU

Benchmarks complete (671.88s)
                   Format  mAP@0.5:0.95  Inference time (ms)
0                 PyTorch      0.402908           283.422085
1             TorchScript      0.402908           291.616663
2                    ONNX      0.402908            94.682336
3                OpenVINO      0.402908            96.916517
4                TensorRT           NaN                  NaN
5                  CoreML      0.402076            73.937317
6   TensorFlow SavedModel      0.402908           172.185935
7     TensorFlow GraphDef      0.402908           145.169625
8         TensorFlow Lite      0.402851           208.791604
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Ultralytics Hyperplane EPYC Milan AMD CPU Results

benchmarks: weights=yolov5s.pt, imgsz=640, batch_size=1, data=/usr/src/yolov5/data/coco128.yaml
Checking setup...
YOLOv5 🚀 v6.0-291-g6adb9cd torch 1.10.1+cu113 CPU
Setup complete ✅ (96 CPUs, 1007.7 GB RAM, 258.5/3519.3 GB disk)

Benchmarks complete (470.78s)
                   Format  mAP@0.5:0.95  Inference time (ms)
0                 PyTorch      0.402908            65.982247
1             TorchScript      0.402908            75.219663
2                    ONNX      0.402908            68.812262
3                OpenVINO      0.402908            27.210791
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel      0.402908           118.363027
7     TensorFlow GraphDef      0.402908            93.722841
8         TensorFlow Lite      0.402851           140.936984
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

[abdullahabid10]

the main difference I can think of is that PyTorch models are capable of rectangular inference while the export formats are at 640x640, so there are padding differences on the images that are passed to the models.

@glenn-jocher if we export a PyTorch model to CoreML, the exported model won't be able to run rectangular inference? Wouldn't that impact the performance a lot, like you showed here

[glenn-jocher]

@abdullahabid10 all formats support export at any size and shape

[abdullahabid10]

the main difference I can think of is that PyTorch models are capable of rectangular inference while the export formats are at 640x640, so there are padding differences on the images that are passed to the models.

@glenn-jocher I'm referring to your comment ^ where you've said that only PyTorch models are capable of rectangular inference. Just want to know that if I export to CoreML, it won't be able to run rectangular inference? For example, if I export to CoreML with --imgsz parameter set to [320, 320], the model won't be able to take advantage of rectangular inference by letterboxing images to 320x192?

[glenn-jocher]

@abdullahabid10 outdated comment.

As I already mentioned above

@abdullahabid10 all formats support export at any size and shape