Conversion to CoreML fails when running with --batch 2

[guy4261]

Hello! I'm trying to convert a model with batchsize of 2. This is with latest CoreML built from their master (per your suggestion in other threads).

1. Converting to batch > 1 results in a CoreML error: generic_reshape_kernel: Invalid bottom shape (80 80 255 1 1) for reshape to (80 80 85 3 2).

2. Despite the error, the script acts as if the conversion worked well (it did - torch -> torchscript -> onnx, but not to coreml).

Repro:

$ git pull  # using your latest master
$ python models/export.py --batch 2
...
Translating MIL ==> MLModel Ops: 100%|███████████████████████████████████████████████████████| 686/686 [00:01<00:00, 654.18 ops/s]
/Users/grapaport/ms/lib/python3.6/site-packages/coremltools/models/model.py:118: RuntimeWarning:
You will not be able to run predict() on this Core ML model.
Underlying exception message was:
Error compiling model:
"compiler error:  Espresso exception: "Invalid argument":
generic_reshape_kernel: Invalid bottom shape (80 80 255 1 1) for reshape to (80 80 85 3 2)".
  RuntimeWarning,
CoreML export success, saved as ./yolov5s.mlmodel

Export complete (14.63s). Visualize with https://github.com/lutzroeder/netron.

[github-actions]

👋 Hello @guy4261, thank you for your interest in 🚀 YOLOv5! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://www.ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@guy4261 export works correctly for me in Colab, unable to reproduce.

!git clone https://github.com/ultralytics/yolov5  # clone repo
%cd yolov5
%pip install -qr requirements.txt coremltools  # install dependencies

import torch
from IPython.display import Image, clear_output  # to display images

clear_output()
print('Setup complete. Using torch %s %s' % (torch.__version__, torch.cuda.get_device_properties(0) if torch.cuda.is_available() else 'CPU'))

!python models/export.py --batch 2

[glenn-jocher]

It appears you may have environment problems. Please ensure you meet all dependency requirements if you are attempting to run YOLOv5 locally. If in doubt, create a new virtual Python 3.8 environment, clone the latest repo (code changes daily), and pip install -r requirements.txt again. We also highly recommend using one of our verified environments below.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are passing. These tests evaluate proper operation of basic YOLOv5 functionality, including training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu.

[guy4261]

Thanks @glenn-jocher , will check.

I'm getting zero exitcode for !python models/export.py --batch 2 as well, still I get this error.

Thanks for the effort of running this - but did you also try to load the CoreML converted model and infer over a batch of 2 images? I will try this from scratch and report. There's the "status" badge saying export.py was tested - but where are those tests defined? Can't find them in the repo.

Thanks again!

[guy4261]

EDIT: works in Colab for me. I'll keep this issue open until I figure out what happened in my local environment (OS X 10.15) and update here. My apologies!

[glenn-jocher]

@guy4261 ah yes, CI tests run export.py but since coremltools is not part of requirements the export is skipped here (under Tests Workflow):
https://github.com/ultralytics/yolov5/runs/1996038442?check_suite_focus=true

[guy4261]

I started a docker container of ultralytics/yolov5:latest on my machine.

I pip installed the requirements for CoreML conversion.

Conversion finishes well (python models/export.py --batch 2), with no warning messages.

However, when I copy the model (docker cp) outside and try to load it using coremltools, I get an error:

>>> import coremltools as ct

>>> model = ct.models.MLModel("yolov5s.mlmodel")
~/miniconda3/envs/py38/lib/python3.8/site-packages/coremltools/models/model.py:114: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error:  Espresso exception: "Invalid argument": generic_reshape_kernel: Invalid bottom shape (80 80 255 1 1) for reshape to (80 80 85 3 2)".
  _warnings.warn(

Not sure if there is anything you can do about it, but things seem broken right now :(

BTW - when I load the model on the docker image, I'm not getting any errors; however the model is not usable on Linux anyway as this is how CoreML works (trying to model.predict() I get: Exception: Model prediction is only supported on macOS version 10.13 or later.).

@glenn-jocher please close if you consider this irreproducible or something that should not be fixed. Thanks!

[glenn-jocher]

@guy4261 we have fully working CoreML export pipelines and reference solutions (i.e. iDetection source code) for business clients, but these are all batch-size 1 solutions.

I'm not actually aware of any batched inference examples for iOS, and it's also unclear to me what performance difference you might see at batch sizes larger than 1 on the ANE. The lowest risk implementation pipeline might of course be to run inference in a swift for loop for whatever contents you need.

[guy4261]

Thanks! Closing this, then.