Custom_real_time_Object_Detection.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Custom real-time Object Detection.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "EPtldBFZZkrk"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/danielsoy/data/blob/main/Custom_real_time_Object_Detection.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a-KVjA7GrZbS"
      },
      "source": [
        "<img align=\"left\" src=\"https://cdn-images-1.medium.com/max/1200/1*iDQvKoz7gGHc6YXqvqWWZQ.png\" width=\"85\" height=\"80\" >\n",
        "<br>\n",
        "\n",
        "#  Custom Real-Time Object Detection\n",
        "\n",
        "<br>\n",
        "\n",
        "\n",
        "\n",
        "This notebook is part of the tutorial [Custom real-time object detection in the browser using TensorFlow.js](https://medium.com/@zaninihugo/custom-real-time-object-detection-in-the-browser-using-tensorflow-js-5ca90538eace) and aims to train a custom Object Detection model using the TensorFlow 2 Object Detection API.\n",
        "\n",
        "Later, this model is going to be converted to TensorFlow.js  to run directly in a web browser. Refer to the tutorial post to see how."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BzZ0iAH--6k6"
      },
      "source": [
        "### Installation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UaUG3D6m_FEY"
      },
      "source": [
        "### **Clone and install the Tensorflow Object Detection API** \n",
        "\n",
        "In order to use the TensorFlow Object Detection API, we need to clone it's GitHub Repo.\n",
        "<br>\n",
        "\n",
        "#### **Dependencies**\n",
        "\n",
        "\n",
        "Most of the dependencies required come preloaded in Google Colab.  No extra installation is needed.\n",
        "<br>\n",
        "\n",
        "#### **Protocol Buffers**\n",
        "\n",
        "\n",
        "\n",
        "The TensorFlow Object Detection API relies on what are called `protocol buffers` (also known as `protobufs`). Protobufs are a language neutral way to describe information. That means you can write a protobuf once and then compile it to be used with other languages, like Python, Java or C [5].\n",
        "\n",
        "The `protoc` command used below is compiling all the protocol buffers in the `object_detection/protos` folder for Python."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xbkuAjhU-3e6"
      },
      "source": [
        "!git clone https://github.com/tensorflow/models.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ukzhAgID_gyP"
      },
      "source": [
        "%cd /content/models/research/\n",
        "!protoc object_detection/protos/*.proto --python_out=.\n",
        "# Install TensorFlow Object Detection API.\n",
        "!cp object_detection/packages/tf2/setup.py .\n",
        "!python -m pip install ."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jkZtxPYCXLHu"
      },
      "source": [
        "Run the model builder test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XzfZSmpSXMxS"
      },
      "source": [
        "!python /content/models/research/object_detection/builders/model_builder_tf2_test.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SIj4L4S1neYu"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dOgMOdcmOzVc"
      },
      "source": [
        "### Getting the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tyNB-m_5O0kh"
      },
      "source": [
        "!pip install -q kaggle\n",
        "!pip install -q kaggle-cli"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sTrBqJRKrfMh"
      },
      "source": [
        "import os\n",
        "os.environ['KAGGLE_USERNAME'] = 'yourusername'\n",
        "os.environ['KAGGLE_KEY'] = 'yourkey'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qBG_9zNwtBQd"
      },
      "source": [
        "%%bash\n",
        "mkdir /content/dataset\n",
        "cd /content/dataset\n",
        "kaggle datasets download -d hugozanini1/kangaroodataset --unzip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xrT-BLUHFIe5"
      },
      "source": [
        "##### Create a `labelmap.pbtxt` file, following the pattern:\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "item {\n",
        "  name: \"kangaroo\"\n",
        "  id: 1\n",
        "}\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zff5msFLAm8g"
      },
      "source": [
        "##### Converting data to TFRecord\n",
        "The dataset contains two files `train_labels.csv` and `test_labels.csv` which need to be converted into TFRecord format so that it can be fed into Tensorflow’s 2 Object Detection API."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5yIkwN0MCbAR"
      },
      "source": [
        "%cd /content/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1VqGdNpF7kwU"
      },
      "source": [
        "!wget https://github.com/raw/hugozanini/object-detection/master/generate_tf_record.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IRYscfzGEgl3"
      },
      "source": [
        "!python generate_tf_record.py -l labelmap.txt -o dataset/train.record -i dataset/images -csv dataset/train_labels.csv\n",
        "!python generate_tf_record.py -l labelmap.txt -o dataset/test.record -i dataset/images -csv dataset/test_labels.csv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qwpHRbZUngms"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0mzXmPHfPJsP"
      },
      "source": [
        "### Configuring train"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r5vys2yVG7_m"
      },
      "source": [
        "We are going to use the pretrained TF2 MobileNet V2 model as the feature extractor in the SSD MobileNet V2 Object Detection model. So the next logical step is to download and untar the pretrained TF2 MobileNet V2 model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cE5PyJUpP7F8"
      },
      "source": [
        "#### Downloading MobileNet weights"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qafiqrBJG8pF"
      },
      "source": [
        "%cd /content\n",
        "!wget http://download.tensorflow.org/models/object_detection/classification/tf2/20200710/mobilenet_v2.tar.gz\n",
        "!tar -xvf mobilenet_v2.tar.gz\n",
        "!rm mobilenet_v2.tar.gz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6zRvdLDfRGA_"
      },
      "source": [
        "!wget https://github.com/raw/tensorflow/models/master/research/object_detection/configs/tf2/ssd_mobilenet_v2_320x320_coco17_tpu-8.config\n",
        "!mv ssd_mobilenet_v2_320x320_coco17_tpu-8.config mobilenet_v2.config"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K5gIl9aSVdGq"
      },
      "source": [
        "#### Defining training parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YVps_k4JRGa7"
      },
      "source": [
        "num_classes = 1\n",
        "batch_size = 96 #16\n",
        "num_steps = 7500 #1500\n",
        "num_eval_steps = 1000\n",
        "\n",
        "train_record_path = '/content/dataset/train.record'\n",
        "test_record_path = '/content/dataset/test.record'\n",
        "model_dir = '/content/training/'\n",
        "labelmap_path = '/content/labelmap.pbtxt'\n",
        "\n",
        "pipeline_config_path = 'mobilenet_v2.config'\n",
        "fine_tune_checkpoint = '/content/mobilenet_v2/mobilenet_v2.ckpt-1'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BOiWE_mCVgap"
      },
      "source": [
        "#### Editing config file\n",
        "\n",
        "The next cell is modification of the code available at [4]."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hH9pVN4qfbAb"
      },
      "source": [
        "import re\n",
        "\n",
        "with open(pipeline_config_path) as f:\n",
        "    config = f.read()\n",
        "\n",
        "with open(pipeline_config_path, 'w') as f:\n",
        "\n",
        "  # Set labelmap path\n",
        "  config = re.sub('label_map_path: \".*?\"', \n",
        "  'label_map_path: \"{}\"'.format(labelmap_path), config)\n",
        "  \n",
        "  # Set fine_tune_checkpoint path\n",
        "  config = re.sub('fine_tune_checkpoint: \".*?\"',\n",
        "                  'fine_tune_checkpoint: \"{}\"'.format(fine_tune_checkpoint),\n",
        "                  config)\n",
        "  \n",
        "  # Set train tf-record file path\n",
        "  config = re.sub('(input_path: \".*?)(PATH_TO_BE_CONFIGURED/train)(.*?\")', \n",
        "                  'input_path: \"{}\"'.format(train_record_path), config)\n",
        "  \n",
        "  # Set test tf-record file path\n",
        "  config = re.sub('(input_path: \".*?)(PATH_TO_BE_CONFIGURED/val)(.*?\")', \n",
        "                  'input_path: \"{}\"'.format(test_record_path), config)\n",
        "  \n",
        "  # Set number of classes.\n",
        "  config = re.sub('num_classes: [0-9]+',\n",
        "                  'num_classes: {}'.format(num_classes), config)\n",
        "  \n",
        "  # Set batch size\n",
        "  config = re.sub('batch_size: [0-9]+',\n",
        "                  'batch_size: {}'.format(batch_size), config)\n",
        "  \n",
        "  # Set training steps\n",
        "  config = re.sub('num_steps: [0-9]+',\n",
        "                  'num_steps: {}'.format(num_steps), config)\n",
        "  \n",
        "  f.write(config)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U_FmiIbinigI"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jHxMKaxfhGyU"
      },
      "source": [
        "### Training the Network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Dx_wEykfpuv"
      },
      "source": [
        "!python /content/models/research/object_detection/model_main_tf2.py \\\n",
        "    --pipeline_config_path={pipeline_config_path} \\\n",
        "    --model_dir={model_dir} \\\n",
        "    --alsologtostderr \\\n",
        "    --num_train_steps={num_steps} \\\n",
        "    --sample_1_of_n_eval_examples=1 \\\n",
        "    --num_eval_steps={num_eval_steps}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Iqb33ihknpCs"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6UBxLRR5RttK"
      },
      "source": [
        "### Validation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sGtfi97VCMOp"
      },
      "source": [
        "Here we're going yo run the code through a loop that waits for checkpoints to evaluate. Once the evaluation finishes, you're going to see the message:\n",
        "\n",
        "`INFO:tensorflow:Waiting for new checkpoint at /content/training/`\n",
        "\n",
        "Then you can stop the cell\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NR_ASWHFRvEt"
      },
      "source": [
        "!python /content/models/research/object_detection/model_main_tf2.py \\\n",
        "    --pipeline_config_path={pipeline_config_path} \\\n",
        "    --model_dir={model_dir} \\\n",
        "    --checkpoint_dir={model_dir}           ##This is passed to run ONLY EVALUATION"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rd2lZkVjnrBV"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JxvpnTruphye"
      },
      "source": [
        "### Training and validation Metrics "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e_ZSdvMzRGsq"
      },
      "source": [
        "%load_ext tensorboard\n",
        "%tensorboard --logdir '/content/training/'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8R_znCjFXdhj"
      },
      "source": [
        "### Export the Inference Graph\n",
        "\n",
        "The below code cell adds a line to the tf_utils.py file. This is a temporary fix to a exporting issue occuring when using the OD API with Tensorflow 2. This code will be removed as soon as the OD Team puts out a fix.\n",
        "\n",
        "All credit goes to the Github users [Jacobsolawetz](https://github.com/Jacobsolawetz) and [ Tanner Gilbert](https://github.com/TannerGilbert), who provided this [temporary fix](https://github.com/tensorflow/models/issues/8841#issuecomment-657647648)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AsrZNbomXfzA"
      },
      "source": [
        "with open('/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/utils/tf_utils.py') as f:\n",
        "    tf_utils = f.read()\n",
        "\n",
        "with open('/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/utils/tf_utils.py', 'w') as f:\n",
        "  # Set labelmap path\n",
        "  throw_statement = \"raise TypeError('Expected Operation, Variable, or Tensor, got ' + str(x))\"\n",
        "  tf_utils = tf_utils.replace(throw_statement, \"if not isinstance(x, str):\" + throw_statement)\n",
        "  f.write(tf_utils)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cRJxDZ36XdKw"
      },
      "source": [
        "output_directory = 'inference_graph'\n",
        "\n",
        "!python /content/models/research/object_detection/exporter_main_v2.py \\\n",
        "    --trained_checkpoint_dir {model_dir} \\\n",
        "    --output_directory {output_directory} \\\n",
        "    --pipeline_config_path {pipeline_config_path}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RQ4nCG2QaUrk"
      },
      "source": [
        "##### Downloading weights"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BqgiuAVsHFIK"
      },
      "source": [
        "!zip -r /content/saved_model.zip /content/inference_graph/saved_model/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2ecN2ZoXaWJp"
      },
      "source": [
        "from google.colab import files\n",
        "files.download(\"/content/saved_model.zip\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t5gaL9ODnu-x"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pnGFb9nleIJ8"
      },
      "source": [
        "###  Testing the trained model\n",
        "\n",
        "Based on [Object Detection API Demo](https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/object_detection_tutorial.ipynb) and [Inference from saved model tf2 colab](https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/inference_from_saved_model_tf2_colab.ipynb)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sn1qzpWKRG0R"
      },
      "source": [
        "!wget https://github.com/raw/hugozanini/object-detection/master/inferenceutils.py\n",
        "from inferenceutils import *"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DD4pogqTnRvv"
      },
      "source": [
        "##### Loading the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xiqxI1qK2Smu"
      },
      "source": [
        "output_directory = 'inference_graph/'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iejp6C20mndZ"
      },
      "source": [
        "category_index = label_map_util.create_category_index_from_labelmap(labelmap_path, use_display_name=True)\n",
        "tf.keras.backend.clear_session()\n",
        "model = tf.saved_model.load(f'/content/{output_directory}/saved_model')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "geqTTdirnU6K"
      },
      "source": [
        "##### Selecting the images to test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NCA1RWcrl-Ea"
      },
      "source": [
        "import pandas as pd\n",
        "test = pd.read_csv('dataset/test_labels.csv')\n",
        "images = list(test['filename'][0:20])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zXd0Bh9Go0C8"
      },
      "source": [
        "##### Doing inference"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yi5oxETyk8PP"
      },
      "source": [
        "images = ['canguru-1.jpeg', 'canguru-2.jpeg', 'canguru-3.jpeg']\n",
        "\n",
        "for image_name in images:\n",
        "  \n",
        "  image_np = load_image_into_numpy_array('dataset/images/' + image_name)\n",
        "  output_dict = run_inference_for_single_image(model, image_np)\n",
        "  vis_util.visualize_boxes_and_labels_on_image_array(\n",
        "      image_np,\n",
        "      output_dict['detection_boxes'],\n",
        "      output_dict['detection_classes'],\n",
        "      output_dict['detection_scores'],\n",
        "      category_index,\n",
        "      instance_masks=output_dict.get('detection_masks_reframed', None),\n",
        "      use_normalized_coordinates=True,\n",
        "      line_thickness=8)\n",
        "  display(Image.fromarray(image_np))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xsQsFG_jnw34"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EPtldBFZZkrk"
      },
      "source": [
        "### References\n",
        "\n",
        "[0] Tanner, G. (2020, July 27). Tensorflow Object Detection with Tensorflow 2: Creating a custom model. Retrieved December 28, 2020, from https://gilberttanner.com/blog/tensorflow-object-detection-with-tensorflow-2-creating-a-custom-model\n",
        "\n",
        "---\n",
        "\n",
        "[1] Rafiq, H. (2020, December 13). Image Object Detection - TensorFlow 2 Object Detection API. Retrieved December 28, 2020, from https://medium.com/swlh/image-object-detection-tensorflow-2-object-detection-api-af7244d4c34e\n",
        "\n",
        "---\n",
        "\n",
        "[2] TensorFlow 2 Detection Model Zoo. (2020, September 09). Tensorflow/models. Retrieved December 28, 2020, from https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md\n",
        "\n",
        "---\n",
        "\n",
        "[3] Tensorflow 2 Models Configs. (2020, August 11). Tensorflow/models. Retrieved December 28, 2020, from https://github.com/tensorflow/models/tree/master/research/object_detection/configs/tf2\n",
        "\n",
        "---\n",
        "[4] Solawetz, J. (2020, December 09). How to Train YOLOv4 on a Custom Dataset. Retrieved December 28, 2020, from https://blog.roboflow.com/training-yolov4-on-a-custom-dataset/\n",
        "\n",
        "---\n",
        "\n",
        "[5] Google Colaboratory. (n.d.). Retrieved January 13, 2021, from https://colab.research.google.com/github/cloud-annotations/google-colab-training/blob/master/object_detection.ipynb#:~:text=The%20TensorFlow%20Object%20Detection%20API%20relies%20on%20what%20are%20called,like%20Python%2C%20Java%20or%20C.\n"
      ]
    }
  ]
}