AssertionError: Not equal to tolerance rtol=0.001, atol=1e-05

[GeneralJing]

Recently I am converting the pytorch segmentation model to onnx model。I can export the onnx model, pass the onnx.checker.check_model() and use the onnxruntime to do inference. But when I use np.testing.assert_allclose(to_numpy(torch_out), ort_outs[0], rtol=1e-03, atol=1e-05) to compare ONNX Runtime and PyTorch results, there is an AssertionError, like follows:

AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-05

Mismatched elements: 20827169 / 20971520 (99.3%)
Max absolute difference: 1.8859415
Max relative difference: 1008390.8
x: array([[[[ 1.165803e+01, 1.163278e+01, 1.160753e+01, ...,
1.179392e+01, 1.176985e+01, 1.174578e+01],
[ 1.167064e+01, 1.164517e+01, 1.161970e+01, ...,...
y: array([[[[11.636896, 11.6166 , 11.596304, ..., 12.943967, 12.909642,
12.875318],
[11.656967, 11.636346, 11.615723, ..., 12.954525, 12.920053,...

The code snippet to export the model is as follows：

model.eval()
batch_size = 1
input_shape = (3, 512, 512)

# x = torch.autograd.Variable(torch.randn(batch_size, *input_shape))

x = torch.rand(batch_size, 3, 512, 512, requires_grad=True)
torch.onnx.export(model, x, model_file_name + '.onnx', export_params=True, opset_version=11, verbose=False)

In this tutorial, https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html, it said, if the results do not match then there is an issue in the ONNX exporter. But i don't know where is the mistake.

cc @BowenBao @sekyondaMeta @svekars @carljparker @NicolasHug @kit1980 @subramen

[Samsonboadi]

/assigntome

[bhavyashahh]

/assigntome

[bhavyashahh]

Hello, can you please help me with the specific pytorch-segmentation model you are trying to use and the other code settings, so that I can try and reproduce your issue on my end?
Apart from that, I tried a sample snippet for the pretrained fcn_resnet50 model within torchvision(0.15.2+cu118).

import torch
import torchvision
import torch.onnx
import onnx
import onnxruntime
import numpy as np

model = torchvision.models.segmentation.fcn_resnet50(pretrained=True)
model.eval()
batch_size = 1

x = torch.rand(batch_size, 3, 512, 512, requires_grad=True)
torch.onnx.export(model, x, 'exported_model.onnx', export_params=True, opset_version=11, verbose=True)

# PyTorch results
torch_out = model(x)

onnx_model = onnx.load("exported_model.onnx")
onnx.checker.check_model(onnx_model)

ort_session = onnxruntime.InferenceSession("exported_model.onnx")

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

# compute ONNX Runtime output prediction
ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(x)}
ort_outs = ort_session.run(None, ort_inputs)

# compare ONNX Runtime and PyTorch results
np.testing.assert_allclose(to_numpy(torch_out['out']), ort_outs[0], rtol=1e-03, atol=1e-05)

There is no assertion error related to tolerance anymore, and the two output results fall under the range.
Version I use that work for me are:

• torch: 2.0.1+cu118
• onnx: 1.14.0
• onnxruntime: 1.15.0

I would suggest you to check your versions and retry first.

[svekars]

This issue has been unassigned due to inactivity. If you are still planning to work on this, you can still send a PR referencing this issue.

[bjhargrave]

/assigntome

[bjhargrave]

I don't think this is an issue with the tutorial. The tutorial works as expected and #1272 (comment) shows that it also works for a segmentation model. Without any additional details from the submitter, we should close this issue.

cc: @svekars

[svekars]

Thanks @bjhargrave - will close and grant half credit for investigation.