1_yolo_start.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. YOLO RarePlanes Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the first of four tutorial notebooks that implement the YOLOv5 object detection network on the RarePlanes dataset to identify aircraft characteristics of interest. \n",
    "\n",
    "In this notebook, you will be creating and detecting a custom class that combines the number of engines and propulsion type. Creating meaningful custom classes can boost model performace in areas of intrest as it forces the model to differentiate in a specific way but, will also find information on how to create your own custom classes or how to use the prebuilt RarePlanes classes with YOLO. \n",
    "\n",
    "This ML pipeline uses a modified implementation of the YOLOv5 implementation found [here](https://github.com/ultralytics/yolov5). The RarePlanes dataset can be found [here](https://www.cosmiqworks.org/rareplanes/) and helper functions for the dataset can be found [here](https://github.com/aireveries/RarePlanes). \n",
    "\n",
    "In this notebook you will create a custom class to detect, pre-process the dataset images into YOLO format, and create the YOLO specific `.yaml` files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/ubuntu/anaconda3/envs/solaris/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    }
   ],
   "source": [
    "import solaris as sol\n",
    "import numpy as np\n",
    "import geopandas as gpd\n",
    "import os\n",
    "import sys\n",
    "import pandas as pd\n",
    "import gdal\n",
    "import glob\n",
    "import shapely\n",
    "import shutil\n",
    "import datetime\n",
    "import rasterio \n",
    "import argparse\n",
    "from solaris.vector.mask import footprint_mask\n",
    "from solaris.vector.polygon import geojson_to_px_gdf, get_overlapping_subset\n",
    "from solaris.utils.core import _check_gdf_load\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Custom Classes\n",
    "\n",
    "The following functions come from the RarePlanes GitHub (link above) and are used to create custom classes using any combination of 'role','num_engines', 'propulsion', 'canards', 'num_tail_fins','wing_position', 'wing_type', 'faa_wingspan_class'. More informaiton on the classes and the dataset can be found [here](https://medium.com/the-downlinq/rareplanes-dataset-paper-and-code-release-5b0cba300a0d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "### From https://github.com/aireveries/RarePlanes/blob/master/tools/create_custom_classes.py ###\n",
    "\n",
    "def count_unique_index(df, by):\n",
    "    return df.groupby(by).size().reset_index().rename(columns={0: 'count'})\n",
    "\n",
    "def create_custom_classes(all_annotations_geojson, geojson_dir, output_path, category_attributes):\n",
    "    \"\"\" parse the geojson files and create custom classes based upon\n",
    "    unique variatons of the RarePlanes attributes.\n",
    "        -all_annotations_geojson (str): The path to the\n",
    "        `RarePlanes_Public_All_Annotations.geojson` file.\n",
    "        - geojson_dir (str): directory containing the geojson files\n",
    "        for individual images or tiles\n",
    "        - output_path (str): directory to output the customized geojsons. Need to provide the absolute path.\n",
    "        - category_attributes (list): A list of attributes to combine\n",
    "        to create a custom class.  Choose any combintaion of the following:\n",
    "        ['role','num_engines', 'propulsion', 'canards', 'num_tail_fins',\n",
    "       'wing_position', 'wing_type', 'faa_wingspan_class']\n",
    "    :returns\n",
    "        - new geojsons with a custom_id for each combination of unique\n",
    "        attributes.\n",
    "        -A lookup table for each classes custom_id.\n",
    "    \"\"\"\n",
    "    os.makedirs(output_path, exist_ok=True)\n",
    "    gdf = gpd.read_file(all_annotations_geojson)\n",
    "    lookup_gdf = count_unique_index(gdf, category_attributes)\n",
    "    lookup_gdf['custom_id'] = list(range(1, len(lookup_gdf) + 1))\n",
    "    lookup_gdf.drop(columns=['count'], inplace=True)\n",
    "    lookup_gdf.to_csv(os.path.join(output_path, \"custom_class_lookup.csv\"))\n",
    "    os.chdir(geojson_dir)\n",
    "    geojsons = glob.glob(\"*.geojson\")\n",
    "    for geojson in tqdm(geojsons):\n",
    "        gdf = gpd.read_file(geojson)\n",
    "        gdf = pd.merge(gdf, lookup_gdf, on=category_attributes, how='left')\n",
    "        gdf.to_file(os.path.join(output_path, geojson), driver=\"GeoJSON\", encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5815/5815 [02:32<00:00, 38.15it/s]\n",
      "100%|██████████| 2710/2710 [01:10<00:00, 38.37it/s]\n"
     ]
    }
   ],
   "source": [
    "## We recommend you leave these paths static for training/testing as yolo requires files to be in specific locations.\n",
    "## Feel free to customize classes using any combination of using any combination of 'role','num_engines', 'propulsion', 'canards', 'num_tail_fins','wing_position', 'wing_type', or 'faa_wingspan_class'\n",
    "\n",
    "#geojsons\n",
    "all_annotations_geojson = '/home/ubuntu/src/yolo_planes/wdata/RarePlanes_Public_All_Annotations.geojson'\n",
    "geojson_dir_train = '/home/ubuntu/src/yolo_planes/wdata/train/geojson_aircraft_tiled/'\n",
    "geojson_dir_test = '/home/ubuntu/src/yolo_planes/wdata/test/geojson_aircraft_tiled/'\n",
    "\n",
    "#output paths\n",
    "output_path_train_one = '/home/ubuntu/src/yolo_planes/geojsons_train/yolo_class_one'\n",
    "output_path_test_one = '/home/ubuntu/src/yolo_planes/geojsons_test/yolo_class_one'\n",
    "\n",
    "#classes to include in custom class (edit therese with any others you would like to add/subtract)\n",
    "class_one = ['num_engines', 'propulsion', 'role']\n",
    "\n",
    "#train\n",
    "create_custom_classes(all_annotations_geojson, geojson_dir_train, output_path_train_one, class_one)\n",
    "\n",
    "#test\n",
    "create_custom_classes(all_annotations_geojson, geojson_dir_test, output_path_test_one, class_one)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating YOLO `.yaml` Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the data `.yaml`, you edit the paths to the images at the top, both train and val (test) images as well as the number of classes and the class list. Save the new classes to edit the `./data/class_one.yaml` file, we print the class list and number of classes to edit your yaml files below. We've already taken care of this for the number of engines and propulsion type custom class, but if you would like to use a different one it should look something like the example yaml below\n",
    "\n",
    "```\n",
    "train: ../class_one/images/train/\n",
    "val: ../class_one/images/val/\n",
    "\n",
    "# number of classes\n",
    "nc: 8\n",
    "\n",
    "# class names\n",
    "names: ['0,0,unpowered,1','1,1,jet,2','2,1,propeller,3','3,2,jet,4','4,2,propeller,5','5,3,jet,6','6,4,jet,7','7,4,propeller,8']\n",
    "```\n",
    "\n",
    "Additionally, change the number of classes in the `./models/yolov5l.yaml`. In this case, the number after `nc:` at the top of the `yolov5l.yaml` file to the number of classes you print in the next block. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "output_type": "error",
     "ename": "NameError",
     "evalue": "name 'pd' is not defined",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-c96168cf674e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput_path_train_one\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'custom_class_lookup.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mclasses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrow\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miterrows\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0mstring\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mitem\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrow\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'pd' is not defined"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv(os.path.join(output_path_train_one,'custom_class_lookup.csv'))\n",
    "classes = []\n",
    "for _, row in df[df.columns[1:-1]].iterrows():\n",
    "    string = \"\"\n",
    "    for item in row:\n",
    "        string += str(item) + \",\"\n",
    "    classes.append(string[:-1])\n",
    "print(classes)\n",
    "print(\"number of classes\", len(classes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've run this block, you can find the custom geojsons in the directory `yolo_class_one` as well as the class lookup table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating YOLO Labels\n",
    "\n",
    "The following solaris function is used to convert the tiled georeferenced images to YOLOv5 lables whcich contain space delimted class id and location information of the objects of interst in each image (in this case planes). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Training Labels to YOLO\n",
    "def gdf_to_yolo(geodataframe, image, output_dir, column='single_id', im_size=(512, 512), min_overlap=0):\n",
    "    \"\"\"\n",
    "    Convert a geodataframe containing polygons to yolo/yolt format.\n",
    "    Arguments\n",
    "    ---------\n",
    "    geodataframe : str\n",
    "        Path to a :class:`geopandas.GeoDataFrame` with a column named\n",
    "        ``'geometry'``.  Can be created from a geojson with labels for unique\n",
    "        objects. Can be converted to this format with\n",
    "        ``geodataframe=gpd.read_file(\"./xView_30.geojson\")``.\n",
    "    image : str\n",
    "        Path to a georeferenced image (ie a GeoTIFF or png created with GDAL)\n",
    "        that geolocates to the same geography as the `geojson`(s). This function will also\n",
    "        accept a :class:`osgeo.gdal.Dataset` or :class:`rasterio.DatasetReader`\n",
    "        with georeferencing information in this argument.\n",
    "    output_dir : str\n",
    "        Path to an output directory where all of the yolo readable text files\n",
    "        will be placed.\n",
    "    column : str, optional\n",
    "        The column name that contians an unique integer id for each of object\n",
    "        class.\n",
    "    im_size : tuple, optional\n",
    "        A tuple specifying the x and y heighth of a an image.  If specified as\n",
    "        ``(0,0)`` (the default,) then the size is determined automatically.\n",
    "    min_overlap : float, optional\n",
    "        A float value ranging from 0 to 1.  This is a percantage.  If a polygon\n",
    "        does not overlap the image by at least min_overlap, the polygon is\n",
    "        discarded.  i.e. 0.66 = 66%. Default value of 0.66.\n",
    "    Returns\n",
    "    -------\n",
    "    gdf : :class:`geopandas.GeoDataFrame`.\n",
    "        The txt file will be written to the output_dir, however the the output\n",
    "        gdf itself is returned.\n",
    "    \"\"\"\n",
    "    if im_size == (0, 0):\n",
    "        imsize_extract = rasterio.open(image).read()\n",
    "        if len(imsize_extract.shape) == 3:\n",
    "            im_size = (imsize_extract.shape[1], imsize_extract.shape[2])\n",
    "        else:\n",
    "            im_size = (imsize_extract.shape[0], imsize_extract.shape[1])\n",
    "    [x0, y0, x1, y1] = [0, 0, im_size[0], im_size[1]]\n",
    "    out_coords = [[x0, y0], [x0, y1], [x1, y1], [x1, y0]]\n",
    "    points = [shapely.geometry.Point(coord) for coord in out_coords]\n",
    "    pix_poly = shapely.geometry.Polygon([[p.x, p.y] for p in points])\n",
    "    dw = 1. / im_size[0]\n",
    "    dh = 1. / im_size[1]\n",
    "    header = [\"column_sub_1\", \"x\", \"y\", \"w\", \"h\"]\n",
    "    if os.path.isdir(output_dir) is False:\n",
    "        os.mkdir(output_dir)    \n",
    "    img_name = image.split('/')[-1]\n",
    "    output = os.path.join(output_dir, img_name.split('.png')[0] + \".txt\")\n",
    "    gdf = geojson_to_px_gdf(geodataframe, image, precision=None)\n",
    "    gdf['column_sub_1'] = gdf[column] - 1\n",
    "    gdf['area'] = gdf['geometry'].area\n",
    "    gdf['intersection'] = (\n",
    "        gdf['geometry'].intersection(pix_poly).area / gdf['area'])\n",
    "    gdf = gdf[gdf['area'] != 0]\n",
    "    gdf = gdf[gdf['intersection'] >= min_overlap]\n",
    "    if not gdf.empty:\n",
    "        boxy = gdf['geometry'].bounds\n",
    "        for _,row in boxy.iterrows():\n",
    "            if row['maxx'] > im_size[0]:\n",
    "                row['maxx'] = im_size[0]\n",
    "            if row['minx'] < 0:\n",
    "                row['minx'] = 0\n",
    "            if row['maxy'] > im_size[1]:\n",
    "                row['maxy'] = im_size[1]\n",
    "            if row['miny'] < 0:\n",
    "                row['miny'] = 0\n",
    "        boxy['xmid'] = (boxy['minx'] + boxy['maxx']) / 2.0\n",
    "        boxy['ymid'] = (boxy['miny'] + boxy['maxy']) / 2.0\n",
    "        boxy['w0'] = (boxy['maxx'] - boxy['minx'])\n",
    "        boxy['h0'] = (boxy['maxy'] - boxy['miny'])\n",
    "        boxy['x'] = boxy['xmid'] * dw\n",
    "        boxy['y'] = boxy['ymid'] * dh\n",
    "        boxy['w'] = boxy['w0'] * dw\n",
    "        boxy['h'] = boxy['h0'] * dh\n",
    "        if not boxy.empty:\n",
    "            gdf = gdf.join(boxy)\n",
    "        gdf.to_csv(path_or_buf=output, sep=' ', columns=header, index=False, header=False)\n",
    "    return gdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "error",
     "ename": "OSError",
     "evalue": "[Errno 45] Operation not supported: '/home/ubuntu'",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-2d004330c101>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0moutput_dir_train_one\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'/home/ubuntu/src/yolo_planes/class_one/labels/train'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmakedirs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput_dir_train_one\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexist_ok\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0moutput_dir_test_one\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'/home/ubuntu/src/yolo_planes/class_one/labels/val'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py\u001b[0m in \u001b[0;36mmakedirs\u001b[0;34m(name, mode, exist_ok)\u001b[0m\n\u001b[1;32m    211\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhead\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mtail\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexists\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    212\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 213\u001b[0;31m             \u001b[0mmakedirs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexist_ok\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mexist_ok\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    214\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mFileExistsError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    215\u001b[0m             \u001b[0;31m# Defeats race condition when another thread created the path\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py\u001b[0m in \u001b[0;36mmakedirs\u001b[0;34m(name, mode, exist_ok)\u001b[0m\n\u001b[1;32m    211\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhead\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mtail\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexists\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    212\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 213\u001b[0;31m             \u001b[0mmakedirs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexist_ok\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mexist_ok\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    214\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mFileExistsError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    215\u001b[0m             \u001b[0;31m# Defeats race condition when another thread created the path\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py\u001b[0m in \u001b[0;36mmakedirs\u001b[0;34m(name, mode, exist_ok)\u001b[0m\n\u001b[1;32m    211\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhead\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mtail\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexists\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    212\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 213\u001b[0;31m             \u001b[0mmakedirs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexist_ok\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mexist_ok\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    214\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mFileExistsError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    215\u001b[0m             \u001b[0;31m# Defeats race condition when another thread created the path\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py\u001b[0m in \u001b[0;36mmakedirs\u001b[0;34m(name, mode, exist_ok)\u001b[0m\n\u001b[1;32m    211\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhead\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mtail\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexists\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    212\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 213\u001b[0;31m             \u001b[0mmakedirs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexist_ok\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mexist_ok\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    214\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mFileExistsError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    215\u001b[0m             \u001b[0;31m# Defeats race condition when another thread created the path\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py\u001b[0m in \u001b[0;36mmakedirs\u001b[0;34m(name, mode, exist_ok)\u001b[0m\n\u001b[1;32m    211\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhead\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mtail\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexists\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    212\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 213\u001b[0;31m             \u001b[0mmakedirs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexist_ok\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mexist_ok\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    214\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mFileExistsError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    215\u001b[0m             \u001b[0;31m# Defeats race condition when another thread created the path\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py\u001b[0m in \u001b[0;36mmakedirs\u001b[0;34m(name, mode, exist_ok)\u001b[0m\n\u001b[1;32m    221\u001b[0m             \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    222\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 223\u001b[0;31m         \u001b[0mmkdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    224\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    225\u001b[0m         \u001b[0;31m# Cannot rely on checking for EEXIST, since the operating system\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mOSError\u001b[0m: [Errno 45] Operation not supported: '/home/ubuntu'"
     ]
    }
   ],
   "source": [
    "truth_dir_train_name = '/home/ubuntu/src/yolo_planes/class_one/images/train'\n",
    "truth_dir_test_name = '/home/ubuntu/src/yolo_planes/class_one/images/val'\n",
    "truth_dir_train = os.fsencode(truth_dir_train_name)\n",
    "truth_dir_test = os.fsencode(truth_dir_test_name)\n",
    "\n",
    "geo_dir_train_one_name = '/home/ubuntu/src/yolo_planes/geojsons_train/yolo_class_one'\n",
    "geo_dir_train_one = os.fsencode(geo_dir_train_one_name)\n",
    "\n",
    "geo_dir_test_one_name = '/home/ubuntu/src/yolo_planes/geojsons_test/yolo_class_one'\n",
    "geo_dir_test_one = os.fsencode(geo_dir_test_one_name)\n",
    "\n",
    "output_dir_train_one = '/home/ubuntu/src/yolo_planes/class_one/labels/train'\n",
    "os.makedirs(output_dir_train_one, exist_ok = True)\n",
    "\n",
    "output_dir_test_one = '/home/ubuntu/src/yolo_planes/class_one/labels/val'\n",
    "os.makedirs(output_dir_test_one, exist_ok = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "output_type": "error",
     "ename": "NameError",
     "evalue": "name 'tqdm' is not defined",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-c1b46d3df667>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0moutput_dir\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput_dir_train_one\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mfile\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtqdm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlistdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgeo_dir\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m     \u001b[0mfilename\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfsdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfile\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfilename\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mendswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\".geojson\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'tqdm' is not defined"
     ]
    }
   ],
   "source": [
    "geo_dir_name = geo_dir_train_one_name\n",
    "geo_dir = geo_dir_train_one\n",
    "truth_dir_name = truth_dir_train_name\n",
    "truth_dir = truth_dir_train\n",
    "output_dir = output_dir_train_one\n",
    "\n",
    "for file in tqdm(os.listdir(geo_dir)):\n",
    "    filename = os.fsdecode(file)\n",
    "    if filename.endswith(\".geojson\"):\n",
    "        geodataframe = gpd.read_file(geo_dir_name + '/' + filename)\n",
    "        image_name = filename.replace('.geojson', '')     \n",
    "        \n",
    "        for truth in os.listdir(truth_dir):\n",
    "            truth_name = os.fsdecode(truth)\n",
    "            if truth_name.endswith(\".png\"):\n",
    "                check_name = truth_name.replace('.png', '')\n",
    "                if (check_name == image_name):\n",
    "                    image = truth_dir_name + '/' + truth_name \n",
    "                    gdf_to_yolo(geodataframe, image, output_dir=output_dir, column='custom_id', im_size=(512, 512), min_overlap=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "output_type": "error",
     "ename": "NameError",
     "evalue": "name 'output_dir_test_one' is not defined",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-43233c8a1eab>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mtruth_dir_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtruth_dir_test_name\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mtruth_dir\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtruth_dir_test\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0moutput_dir\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput_dir_test_one\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mfile\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtqdm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlistdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgeo_dir\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'output_dir_test_one' is not defined"
     ]
    }
   ],
   "source": [
    "geo_dir_name = geo_dir_test_one_name\n",
    "geo_dir = geo_dir_test_one\n",
    "truth_dir_name = truth_dir_test_name\n",
    "truth_dir = truth_dir_test\n",
    "output_dir = output_dir_test_one\n",
    "\n",
    "for file in tqdm(os.listdir(geo_dir)):\n",
    "    filename = os.fsdecode(file)\n",
    "    if filename.endswith(\".geojson\"):\n",
    "        geodataframe = gpd.read_file(geo_dir_name + '/' + filename)\n",
    "        image_name = filename.replace('.geojson', '')     \n",
    "        \n",
    "        for truth in os.listdir(truth_dir):\n",
    "            truth_name = os.fsdecode(truth)\n",
    "            if truth_name.endswith(\".png\"):\n",
    "                check_name = truth_name.replace('.png', '')\n",
    "                if (check_name == image_name):\n",
    "                    image = truth_dir_name + '/' + truth_name \n",
    "                    gdf_to_yolo(geodataframe, image, output_dir=output_dir, column='custom_id', im_size=(512, 512), min_overlap=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've run these three blocks, you can find the labels in the class_one labels directories—val refers to testing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, please head to the notebook titled `2_yolo_train.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.7 64-bit",
   "language": "python",
   "name": "python_defaultSpec_1597352847495"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}