Strange output from the CoreML model converted from the Pytorch model

[zhengliu0829]

Hi,
I trained a YOLOV5S model to detect one object. I converted the model to TensorFlow, TensorFlow lite and CoreML formats using the export.py code. The TensorFlow and TensorFlow lite model work well. But the output from the CoreML model is wrong. See the screenshot below. Can anybody please give me some clue why the output is like this? Is it wrong? Or extra postprocessing steps are needed before feeding this output to the NMS function?

Thanks!

[glenn-jocher]

@zhengliu0829 yes this model would require additional steps from the Detect() layer as well as NMS to run inference.

[zhengliu0829]

Thanks @glenn-jocher. Do you know why the Detect() layer is not exported when converting from Pytorm model to CoreML? Is it because the Pytorch "trace" can not trace the for-loop for the bounding boxes calculation? If I convert the model from TensorFlow .pb format to CoreML, will the Detect() layer be available in the CoreML model?