coco_extract_subset.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cxi8K9mXwl5t"
      },
      "source": [
        "# Extract annotations from COCO Dataset annotation file\n",
        "This notebook was created to answer a question from stackoverflow: [https://stackoverflow.com/questions/69722538/extract-annotations-from-coco-dataset-annotation-file](https://stackoverflow.com/questions/69722538/extract-annotations-from-coco-dataset-annotation-file)\n",
        "\n",
        "> I want to train on a subset of COCO dataset. For the images, I have created a folder of first 30k images of train2017 folder. Now I need annotations of those 30k images (extracted from instances_train2017.json) in a separate json file so that I can train it. How can I do it?\n",
        "\n",
        "The reason for the question is that Coco stores all of the annotations in one long json file, so there is no simple way to extract only the ones that you need. PyLabel can help with this task by importing the dataset, filtering the annotations to the images you care about, and then exporting back to a coco json file. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [],
      "source": [
        "import logging\n",
        "logging.getLogger().setLevel(logging.CRITICAL)\n",
        "!pip install pylabel > /dev/null"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "AYwWkeF4z1Sc"
      },
      "outputs": [],
      "source": [
        "from pylabel import importer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Download sample dataset \n",
        "For this example we can use a sample dataset stored in coco format. The general approach can later be applied to the full coco dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2021-11-01 07:52:48--  https://github.com/pylabelalpha/notebook/blob/main/BCCD_coco.zip?raw=true\n",
            "Resolving github.com (github.com)... 192.30.255.112\n",
            "Connecting to github.com (github.com)|192.30.255.112|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://github.com/pylabelalpha/notebook/raw/main/BCCD_coco.zip [following]\n",
            "--2021-11-01 07:52:48--  https://github.com/pylabelalpha/notebook/raw/main/BCCD_coco.zip\n",
            "Reusing existing connection to github.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://github.com/raw/pylabelalpha/notebook/main/BCCD_coco.zip [following]\n",
            "--2021-11-01 07:52:48--  https://github.com/raw/pylabelalpha/notebook/main/BCCD_coco.zip\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 7625693 (7.3M) [application/zip]\n",
            "Saving to: ‘data/BCCD_coco.zip’\n",
            "\n",
            "data/BCCD_coco.zip  100%[===================>]   7.27M  21.2MB/s    in 0.3s    \n",
            "\n",
            "2021-11-01 07:52:50 (21.2 MB/s) - ‘data/BCCD_coco.zip’ saved [7625693/7625693]\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>img_folder</th>\n",
              "      <th>img_filename</th>\n",
              "      <th>img_path</th>\n",
              "      <th>img_id</th>\n",
              "      <th>img_width</th>\n",
              "      <th>img_height</th>\n",
              "      <th>img_depth</th>\n",
              "      <th>ann_segmented</th>\n",
              "      <th>ann_bbox_xmin</th>\n",
              "      <th>ann_bbox_ymin</th>\n",
              "      <th>...</th>\n",
              "      <th>ann_area</th>\n",
              "      <th>ann_segmentation</th>\n",
              "      <th>ann_iscrowd</th>\n",
              "      <th>ann_pose</th>\n",
              "      <th>ann_truncated</th>\n",
              "      <th>ann_difficult</th>\n",
              "      <th>cat_id</th>\n",
              "      <th>cat_name</th>\n",
              "      <th>cat_supercategory</th>\n",
              "      <th>split</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>id</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td></td>\n",
              "      <td>BloodImage_00315.jpg</td>\n",
              "      <td>None</td>\n",
              "      <td>0</td>\n",
              "      <td>640</td>\n",
              "      <td>480</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>164.0</td>\n",
              "      <td>261.0</td>\n",
              "      <td>...</td>\n",
              "      <td>13699.0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>Unspecified</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>RBC</td>\n",
              "      <td>None</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td></td>\n",
              "      <td>BloodImage_00315.jpg</td>\n",
              "      <td>None</td>\n",
              "      <td>0</td>\n",
              "      <td>640</td>\n",
              "      <td>480</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>66.0</td>\n",
              "      <td>...</td>\n",
              "      <td>13699.0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>Unspecified</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>RBC</td>\n",
              "      <td>None</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td></td>\n",
              "      <td>BloodImage_00315.jpg</td>\n",
              "      <td>None</td>\n",
              "      <td>0</td>\n",
              "      <td>640</td>\n",
              "      <td>480</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>13.0</td>\n",
              "      <td>234.0</td>\n",
              "      <td>...</td>\n",
              "      <td>11781.0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>Unspecified</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>RBC</td>\n",
              "      <td>None</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td></td>\n",
              "      <td>BloodImage_00315.jpg</td>\n",
              "      <td>None</td>\n",
              "      <td>0</td>\n",
              "      <td>640</td>\n",
              "      <td>480</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>239.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>...</td>\n",
              "      <td>11960.0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>Unspecified</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>RBC</td>\n",
              "      <td>None</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td></td>\n",
              "      <td>BloodImage_00315.jpg</td>\n",
              "      <td>None</td>\n",
              "      <td>0</td>\n",
              "      <td>640</td>\n",
              "      <td>480</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>542.0</td>\n",
              "      <td>109.0</td>\n",
              "      <td>...</td>\n",
              "      <td>10290.0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>Unspecified</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>RBC</td>\n",
              "      <td>None</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 24 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   img_folder          img_filename img_path  img_id  img_width  img_height  \\\n",
              "id                                                                            \n",
              "0              BloodImage_00315.jpg     None       0        640         480   \n",
              "1              BloodImage_00315.jpg     None       0        640         480   \n",
              "2              BloodImage_00315.jpg     None       0        640         480   \n",
              "3              BloodImage_00315.jpg     None       0        640         480   \n",
              "4              BloodImage_00315.jpg     None       0        640         480   \n",
              "\n",
              "    img_depth ann_segmented  ann_bbox_xmin  ann_bbox_ymin  ...  ann_area  \\\n",
              "id                                                         ...             \n",
              "0           3             0          164.0          261.0  ...   13699.0   \n",
              "1           3             0           15.0           66.0  ...   13699.0   \n",
              "2           3             0           13.0          234.0  ...   11781.0   \n",
              "3           3             0          239.0            3.0  ...   11960.0   \n",
              "4           3             0          542.0          109.0  ...   10290.0   \n",
              "\n",
              "    ann_segmentation  ann_iscrowd     ann_pose  ann_truncated ann_difficult  \\\n",
              "id                                                                            \n",
              "0               None         None  Unspecified              0             0   \n",
              "1               None         None  Unspecified              0             0   \n",
              "2               None         None  Unspecified              0             0   \n",
              "3               None         None  Unspecified              0             0   \n",
              "4               None         None  Unspecified              1             0   \n",
              "\n",
              "   cat_id cat_name cat_supercategory split  \n",
              "id                                          \n",
              "0       0      RBC              None        \n",
              "1       0      RBC              None        \n",
              "2       0      RBC              None        \n",
              "3       0      RBC              None        \n",
              "4       0      RBC              None        \n",
              "\n",
              "[5 rows x 24 columns]"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import os \n",
        "import zipfile\n",
        "\n",
        "#Download and import sample coco dataset \n",
        "os.makedirs(\"data\", exist_ok=True)\n",
        "!wget \"https://github.com/pylabelalpha/notebook/blob/main/BCCD_coco.zip?raw=true\" -O data/BCCD_coco.zip\n",
        "with zipfile.ZipFile(\"data/BCCD_coco.zip\", 'r') as zip_ref:\n",
        "   zip_ref.extractall(\"data\")\n",
        "\n",
        "#Specify path to the coco.json file\n",
        "path_to_annotations = \"data/BCCD_Dataset.json\"\n",
        "#Specify the path to the images (if they are in a different folder than the annotations)\n",
        "path_to_images = \"\"\n",
        "\n",
        "#Import the dataset into the pylable schema \n",
        "dataset = importer.ImportCoco(path_to_annotations, path_to_images=path_to_images, name=\"BCCD_coco\")\n",
        "dataset.df.head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "PyLabel imports the annotations into a pandas dataframe. Now you can filter this dataframe to the rows related to the images that you care about. There are 364 images in this dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5R1rOJVL420b",
        "outputId": "ae94525b-c6d5-4a4f-d56d-7f1bd29a7411"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of images: 364\n",
            "Class counts:\n",
            "RBC          4155\n",
            "WBC           372\n",
            "Platelets     361\n",
            "Name: cat_name, dtype: int64\n"
          ]
        }
      ],
      "source": [
        "print(f\"Number of images: {dataset.analyze.num_images}\")\n",
        "print(f\"Class counts:\\n{dataset.analyze.class_counts}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Extract images\n",
        "Lets copy some images to another directory to to represent the images that we care about. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "mkdir: data/100Images/: File exists\n"
          ]
        }
      ],
      "source": [
        "#Copy 100 images from the BCCD_Dataset/BCCD/JPEGImages/ to BCCD_Dataset/BCCD/100Images/ \n",
        "!mkdir data/100Images/ \n",
        "!ls data/*.jpg | head -100 | xargs -I{} cp {} data/100Images/ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create a list with all of the files in this directory. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 files including BloodImage_00000.jpg\n"
          ]
        }
      ],
      "source": [
        "#Store a list of all of the files in the directory \n",
        "files = sorted(os.listdir('data/100Images/'))\n",
        "print(f\"{len(files)} files including {files[0]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now filter the dataframe to only images in the list of files."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of images 100\n"
          ]
        }
      ],
      "source": [
        "dataset.df = dataset.df[dataset.df.img_filename.isin(files)].reset_index()\n",
        "print(f\"Number of images {dataset.df.img_filename.nunique()}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Export annotations back as a coso json file"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['data/100Images/100Images_coco.json']"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset.path_to_annotations = 'data/100Images/'\n",
        "dataset.name = '100Images_coco'\n",
        "\n",
        "dataset.export.ExportToCoco()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "import2.ipynb",
      "provenance": []
    },
    "interpreter": {
      "hash": "de5e368ab5494158da905fb8b1c2e4fef8844fbfaace7cceca0beab983df3a80"
    },
    "kernelspec": {
      "display_name": "Python 3.8.5 64-bit ('penv': venv)",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}