10-Intro-resnet50-convert-compare.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convert and Compare Torch, ONNX, and CoreML Resnet50  \n",
    "The notebooks in this folder are code for, and notes about, converting a \"standard\" **PyTorch** **Resnet50** model to **CoreML**.  Because there is no direct conversion from Pytorch to CoreML,  ONNX is used as an intermediate format.  The conversion flow is:\n",
    "\n",
    "    Torch->ONNX,  ONNX->CoreML,  then CoreML->iOS using XCode\n",
    "\n",
    "This was primarily a learning exercise, and secondarily an experiment to investigate if an off-the-shelf  *Torch Resnet50* - e.g. from the Python *torchvision*  distribution - would convert successfully (... The answer is: no, it does not - but for now I'm assuming that the root cause is me, and that I need to learn more.). \n",
    "\n",
    "In addition to demonstrating the conversion process, and CoreML model repair where necessary, these notebooks show and compare predictions generated by the original models and their converted descendants."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why?\n",
    "\n",
    "I hear you ask ...\n",
    "\n",
    "I tried to port my own ResNet50 model from Torch(Fastai) to CoreML. Unsuccessfully.\n",
    "\n",
    "As part of investigating how and why that conversion went astray, I thought it might be enlightening to convert a \"standard\" Pytorch Resnet50 model to CoreML, and see what happened.   I also included a \"standard\" ONNX Resnet50 model, because as long as I was doing this, why not? ... I wanted to see how its predictions would compare to the others.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview\n",
    "Here is the general approach:\n",
    "- Obtain the intial,off-the-shelf model (By downloading from web or loading from a package).\n",
    "- Sanity-check that I've set up the model correctly by generating a few predictions and confirming that they are correct.\n",
    "- Convert,repair when necessary, and verify results until we generate a working CoreML model.\n",
    "- Compare the prediction results of the original model(s) to those of the converted descendants.\n",
    "\n",
    "Test images were selected randomily from a collection of dogs, cats, plants and food images that reside on my local machine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of Conversion Results\n",
    "\n",
    "*Torch->Onnx->CoreML*\n",
    "\n",
    "Conversion of the Torch Resnet50 model `torchvision.models.resnet50(pretrained=True)` to  ONNX was successful. The conversion of that ONNX model to CoreML failed. The *ONNX->CoreML* step created a CoreML model that raised shape-mismatch errors when run. See [20-convert-torch-to-coreml][20-convert].\n",
    "\n",
    "To get the faulty CoreML model to work, I edited the problematic layers (using `coremltools` and `coreml_help`) and re-verified in Python.\n",
    "\n",
    "*Onnx->CoreML*\n",
    "\n",
    "As part of investigating the error above, I attempted to convert an off-the-shelf ONNX Resnet50 to CoreML (see [30-convert-onnx-to-coreml][30-convert] ).  That conversion was successful.  One interpretation of such a result is that  the *Torch -> ONNX* conversion generates an ONNX model that is either not entirely correct, or not handled by the *ONNX->CoreML* conversion tools, or both.  However,  *more likely*  is that I introduced the error thru misunderstanding some part of the conversion procedure.  I'm still looking into this.\n",
    "\n",
    "[20-convert]:20-convert-torch-to-coreml.ipynb\n",
    "[30-convert]:30-convert-onnx-to-coreml.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of Comparision Results\n",
    "After conversion and repair, I compared predictions from the CoreML-converted-from-Torch Resnet50 model to predictions generated by the originals, the intermediate models, and to predictions from \"off-the-shelf\" ONNX and CoreML Resnet50 models. See [40-Compare-torch-onnx-coreml](40-Compare-torch-onnx-coreml.ipynb).\n",
    "\n",
    "**95% Agreement - the \"Torch Family\"** - The predictions generated by the original Torch model, the intermediate ONNX model dervied from it, and the final (after repair) CoreML model - call these three together the *Torch Family*  - were very consistent. About 95% or more of the predictions agreed.\n",
    "\n",
    "*60% Agreement - the \"ONNX Family\"* - The \"Model Zoo\" ONNX model and its CoreML descendant - call these two the *\"ONNX Family\"* - generated modest agreement. A bit surprising that it was not higher.\n",
    "\n",
    "*53% Agreement* - between *Torch* and *ONNX* and *CoreML*\n",
    "\n",
    "\n",
    "#### Conclusion\n",
    "  * Any conversion from Torch to CoreML will probably require some amount of CoreML \"model surgery\" followed by comprehensive testing.\n",
    "  * The converted model should behave very much like the Torch model, which is good news.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notebooks\n",
    "**[10-Intro-resnet50-convert-compare](10-Intro-resnet50-convert-compare.ipynb)**\n",
    "\n",
    "This notebook.\n",
    "\n",
    "**[20-convert-torch-to-coreml](20-convert-torch-to-coreml.ipynb)**\n",
    "\n",
    "Load the packaged Torch Resnet50 (`torchvision.models.resnet50(pretrained=True)`), convert to ONNX, then convert to CoreML, checking at each step that the models work.  The first conversion,*Torch->ONNX*, succeeds, but the second one, *ONNX->CoreML*, fails. Surgery is required to fix the CoreML model. Surgery is attempted, patient survives, and the CoreML model successfully validated.  A brief analysis of agreement between the models is followed by a display of images and the resulting predictions\n",
    "\n",
    "**[30-convert-onnx-to-coreml](30-convert-onnx-to-coreml.ipynb)**\n",
    "\n",
    "The notebook converts an off-the-shelf ONNX Resnet50 to CoreML.  Successfully. \n",
    "\n",
    "The initial attempt at converting from ONNX->CoreML failed. Is this because of a fault in the ONNX->CoreML conversion tools, or is it  because the original model started life as a Torch model? This notebook is an experiment to investigate.\n",
    "\n",
    "This conversion was successful. Which could mean something about the two-step (Torch->ONNX, ONNX->CoreML) conversion process is breaking. Or that I don't know enough yet.\n",
    "\n",
    "**[40-compare-torch-onnx-coreml](40-compare-torch-onnx-coreml.ipynb)**\n",
    "\n",
    "This notebook compares the predictions of six Resnet50 Models. \n",
    "It loads the original, intermediate and final models created by the conversions described previously. \n",
    "\n",
    "The models are named:   \n",
    "```\n",
    " torchm Torch model from the torchvision distribution.  \n",
    " t2o    An ONNX Model. The result of converting \"torchm\" to ONNX.  \n",
    " t2c    A CoreML Model. The result of converting the intermediate model \"t2o\" to CoreML.  \n",
    " onnxm  The ONNX Resnet50 model from the ONNX Model Zoo on Github.  \n",
    " o2c    A CoreML Model. The result of converting \"onnxm\" to CoreML.  \n",
    " cml    A CoreML Model. For comparison, I added the \"native\" CoreML Resnet50 model.  \n",
    "```\n",
    "Agreements between the predictions of all models are tabulated and presented in an \"Agreement Matrix\". \n",
    "A selection of images and the predictions made for them by each model is shown.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "#### Tools and Packages\n",
    "\n",
    "- **[Netron](https://github.com/lutzroeder/netron)** for examining models\n",
    "- [torch, torchvision](https://onnx.ai)\n",
    "- [onnx](https://onnx.ai) for conversion\n",
    "- [onnx_coreml](https://github.com/onnx/onnx-coreml) to convert to CoreML\n",
    "- [onnxruntime](https://microsoft.github.io/onnxruntime/) for verification and debugging.\n",
    "- [coremltools](https://github.com/apple/coremltools) for verification and debugging and model editing\n",
    "- [coreml_help](https://mcsieber.github.io/coreml_help.html) (minor) Python helper tools that I wrote. A convenience. Not required.\n",
    "- [pred_help](https://mcsieber.github.io/pred_help.html) More (minor) python helper tools. Not required.\n",
    "- [XCode 10](https://developer.apple.com/xcode/) to include the model in an iOS app and load the app onto my iPhone\n",
    "\n",
    "#### Helpful Books, Articles, Tutorials etc ...\n",
    "\n",
    "- [**Core ML Survival Guide**](https://leanpub.com/coreml-survival-guide) by [*Matthijs Holleman*](https://github.com/hollance)  for real help with CoreML\n",
    "- [How I Shipped a Neural Network on iOS with CoreML, PyTorch, and React Native](https://attardi.org/pytorch-and-coreml)\n",
    "- [raywenderlich.com](https://www.raywenderlich.com) for iOS sample apps and tutorials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}