4_yolo_eval_performace.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. YOLO RarePlanes Model Scoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This last notebook converts the ground truth annotations to the bounding box format whcih YOLO creates in order to compare them with the predictions. Then, precision, recall, and f1 are calculated by class and as a whole. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    }
   ],
   "source": [
    "from shapely.geometry import box\n",
    "import argparse\n",
    "import geopandas as gpd\n",
    "import pandas as pd\n",
    "import os\n",
    "import glob\n",
    "from tqdm.notebook import tqdm\n",
    "import numpy as np\n",
    "from solaris.eval.iou import calculate_iou\n",
    "from solaris.eval.vector import mF1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert Ground Truth to Bounding Boxes\n",
    "\n",
    "The following scripts take the geojson ground truths for the test set and convert them to bounding boxes. Then, in the two cells, the same custom classes are created in the ground truth. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 66/66 [00:03<00:00, 16.67it/s]\n"
     ]
    }
   ],
   "source": [
    "input_folder = \"/home/ubuntu/src/yolo_planes/wdata/test/geojson_aircraft\"\n",
    "output_folder = \"/home/ubuntu/src/yolo_planes/wdata/test/geojson_aircraft_bbox\"\n",
    "os.makedirs(output_folder,exist_ok=True)\n",
    "os.chdir(input_folder)\n",
    "geojsons = glob.glob(\"*.geojson\")\n",
    "for geojson in tqdm(geojsons):\n",
    "    gdf = gpd.read_file(geojson)\n",
    "    box_geoms= []\n",
    "    for _, row in gdf.iterrows():\n",
    "        x = row.geometry.bounds\n",
    "        bbox = box(x[0], x[1], x[2], x[3])\n",
    "        box_geoms.append(bbox)\n",
    "    gdf['box_geom'] = box_geoms\n",
    "    gdf = gdf.drop('geometry',axis=1)\n",
    "    gdf = gpd.GeoDataFrame(gdf, geometry='box_geom')\n",
    "    out_name=os.path.join(output_folder, geojson)\n",
    "    gdf.to_file(out_name, driver='GeoJSON', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_unique_index(df, by):\n",
    "    return df.groupby(by).size().reset_index().rename(columns={0: 'count'})\n",
    "\n",
    "def create_custom_classes(all_annotations_geojson, geojson_dir, output_path, category_attributes):\n",
    "    \"\"\" parse the geojson files and create custom classes based upon\n",
    "    unique variatons of the RarePlanes attributes.\n",
    "        -all_annotations_geojson (str): The path to the\n",
    "        `RarePlanes_Public_All_Annotations.geojson` file.\n",
    "        - geojson_dir (str): directory containing the geojson files\n",
    "        for individual images or tiles\n",
    "        - output_path (str): directory to output the customized geojsons. Need to provide the absolute path.\n",
    "        - category_attributes (list): A list of attributes to combine\n",
    "        to create a custom class.  Choose any combintaion of the following:\n",
    "        ['role','num_engines', 'propulsion', 'canards', 'num_tail_fins',\n",
    "       'wing_position', 'wing_type', 'faa_wingspan_class']\n",
    "    :returns\n",
    "        - new geojsons with a custom_id for each combination of unique\n",
    "        attributes.\n",
    "        -A lookup table for each classes custom_id.\n",
    "    \"\"\"\n",
    "    os.makedirs(output_path, exist_ok=True)\n",
    "    gdf = gpd.read_file(all_annotations_geojson)\n",
    "    lookup_gdf = count_unique_index(gdf, category_attributes)\n",
    "    lookup_gdf['custom_id'] = list(range(0, len(lookup_gdf)))\n",
    "    lookup_gdf.drop(columns=['count'], inplace=True)\n",
    "    lookup_gdf.to_csv(os.path.join(output_path, \"custom_class_lookup.csv\"))\n",
    "    os.chdir(geojson_dir)\n",
    "    geojsons = glob.glob(\"*.geojson\")\n",
    "    for geojson in tqdm(geojsons):\n",
    "        gdf = gpd.read_file(geojson)\n",
    "        gdf = pd.merge(gdf, lookup_gdf, on=category_attributes, how='left')\n",
    "        gdf[\"custom_id\"] = pd.to_numeric(gdf[\"custom_id\"], downcast='float')\n",
    "        gdf.to_file(os.path.join(output_path, geojson), driver=\"GeoJSON\", encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 66/66 [00:03<00:00, 18.73it/s]\n"
     ]
    }
   ],
   "source": [
    "all_annotations_geojson = '/home/ubuntu/src/yolo_planes/wdata/RarePlanes_Public_All_Annotations.geojson'\n",
    "geojson_dir_test = '/home/ubuntu/src/yolo_planes/wdata/test/geojson_aircraft_bbox'\n",
    "\n",
    "output_path_test_one = '/home/ubuntu/src/yolo_planes/wdata/test/yolo_class_one_truth_bbox'\n",
    "\n",
    "class_one = ['num_engines', 'propulsion']\n",
    "\n",
    "create_custom_classes(all_annotations_geojson, geojson_dir_test, output_path_test_one, class_one)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "The following cells execute the scoring of the predicitons vs the ground truth. The first cell outputs total recall, precision, and F1, while the second cell outputs the lookup table and now includes the class by class recall, precision, and F1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "getting unique objects...\n",
      "calculating recall...\n",
      "mRecall: 0.6511768568429169\n",
      "calculating precision...\n",
      "mPrecision: 0.7047816872010533\n",
      "\n",
      "calculating F1 scores...\n",
      "mF1: 0.6732098637317354\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/ipykernel_launcher.py:26: RuntimeWarning: Mean of empty slice\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/ipykernel_launcher.py:364: RuntimeWarning: invalid value encountered in double_scalars\n"
     ]
    }
   ],
   "source": [
    "proposal_polygons_dir =\"/home/ubuntu/src/yolo_planes/yolov5/inference/class_one_out/bounding_boxes\"\n",
    "gt_polygons_dir = \"/home/ubuntu/src/yolo_planes/wdata/test/yolo_class_one_truth_bbox\"\n",
    "mF1, f1s_by_class, precision_iou_by_obj, precision_by_class, mPrecision, recall_iou_by_obj, recall_by_class, mRecall, object_subset, prop_objs, all_objs = mF1(proposal_polygons_dir, gt_polygons_dir, prediction_cat_attrib=\"class_id\", gt_cat_attrib='custom_id', object_subset=[], all_outputs=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   num_engines propulsion  recall_by_class  precision_by_class  f1s_by_class\n",
      "0            0  unpowered         0.729730            0.729730      0.729730\n",
      "1            1        jet         0.250000            0.250000      0.250000\n",
      "2            1  propeller         0.933941            0.933941      0.933941\n",
      "3            2        jet         0.978015            0.978015      0.978015\n",
      "4            2  propeller         0.885305            0.885305      0.885305\n",
      "5            3        jet         0.000000            0.000000      0.000000\n",
      "6            4        jet         0.690000            0.690000      0.690000\n",
      "7            4  propeller         0.742424            0.742424      0.742424\n"
     ]
    }
   ],
   "source": [
    "lookup_table = \"/home/ubuntu/src/yolo_planes/geojsons_test/yolo_class_one/custom_class_lookup.csv\"\n",
    "lookup_table = pd.read_csv(lookup_table)\n",
    "lookup_table.drop(lookup_table.columns[lookup_table.columns.str.contains('unnamed',case = False)],axis = 1, inplace = True)\n",
    "object_subset_int = [int(i) + 1 for i in object_subset]\n",
    "out_dict = {\"custom_id\":object_subset_int,\"f1\":f1s_by_class,\"precision\": precision_by_class, \"recall\": recall_by_class}\n",
    "out_df = pd.DataFrame.from_dict(out_dict)\n",
    "out_df = pd.merge(out_df, lookup_table, on='custom_id', how='left')\n",
    "out_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, using these fused geojsons, you can visulaize the predictions by downloading the `/home/ubuntu/src/yolo_planes/yolov5/inference/class_one_out/bounding_boxes` folder to your local machine and using a geographic infromation system like QGIS and overlaying the bounding boxes onto the real image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Solaris",
   "language": "python",
   "name": "solaris"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}