4_Post-process.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Post-process\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During this stage we want to keep the best quality predictions. So we can apply:\n",
    "\n",
    "   * A threshold to the probability, to preserve the results with high values.\n",
    "   * A minimum area filter, to keep the predictions that correspond to large areas and eliminate false positives (FP) due to small fluctuations.\n",
    "\n",
    "We can process the results in different ways, for example we can first polygon the prediction result while applying a threshold to the probability. Then work with a vector file to filter the predictions from small areas.\n",
    "\n",
    "In this example we work as follows\n",
    "\n",
    "1. first polygonize each result while applying a threshold to the classes\n",
    "2. we join all the results to have a single vector file (but one could work with each result independently if they wanted to)\n",
    "3. Before applying the minimum area filter, it may be useful first to apply a buffer -> dissolve -> multiparts to single parts, this generates that the predictions that are capable are small but are together with others are combined forming a single larger area and avoiding then the minimum area filter removes them\n",
    "4. We apply the minimum area filter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Apply a threshold to de probability and poligonize**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os \n",
    "\n",
    "from os import walk\n",
    "from tqdm import tqdm\n",
    "\n",
    "PATH = os.path.join('./data_result_cba/','onedate_FW')\n",
    "\n",
    "results_folder = os.path.join(PATH,'160_160/')\n",
    "\n",
    "filter_results_folder         = os.path.join(PATH,'filtered/') \n",
    "vector_results_folder         = os.path.join(PATH,'vector_geojson/')\n",
    "\n",
    "!mkdir -p $filter_results_folder\n",
    "!mkdir -p $vector_results_folder\n",
    "\n",
    "input_files = next(walk(results_folder), (None, None, []))[2]  # [] if no file\n",
    "print(input_files[:1])\n",
    "DEBUG=True \n",
    "\n",
    "for raster in tqdm(input_files):\n",
    "  \n",
    "    if \"tif\" in (raster):\n",
    "        \n",
    "        src = results_folder + raster \n",
    "        dst = filter_results_folder + raster  \n",
    "\n",
    "\n",
    "        !gdal_calc.py -A $src --A_band=1 -B $src --B_band=2 \\\n",
    "        --calc=\"(A>140)*(B<240)*100+(A>0)*(B>240)*200\" \\\n",
    "        --outfile=$dst\n",
    "\n",
    "        if DEBUG: print(\"convert into a vector\")\n",
    "        #convert into a vector\n",
    "        vec_file_name = raster[:-4] + '.geojson'\n",
    "        vec_sdt = vector_results_folder + vec_file_name\n",
    "        \n",
    "        !gdal_polygonize.py $dst $vec_sdt\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Merge all de the results**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glob import glob\n",
    "import subprocess\n",
    "\n",
    "files = glob('./data_result_cba/onedate_FW/vector_geojson/*.geojson')\n",
    "\n",
    "for i, f in tqdm(enumerate(files)):\n",
    "    if i==0:\n",
    "        cmd = f'ogrmerge.py -f GPKG -o ./data_result_cba/onedate_FW/merged.gpkg -nln out {f}'\n",
    "        subprocess.run(cmd, shell=True)\n",
    "    cmd = f'ogrmerge.py -append -o ./data_result_cba/onedate_FW/merged.gpkg -nln  out {f}'\n",
    "    subprocess.run(cmd, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Change to utm and create a file for each category**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter\n",
    "    \n",
    "# change to utm\n",
    "name = './data_result_bar/onedate_FW/merged.gpkg'\n",
    "name_utm = './data_result_bar/onedate_FW/merged_utm.gpkg'\n",
    "\n",
    "!ogr2ogr -s_srs EPSG:4326 -t_srs EPSG:32720 -f 'GPKG' $name_utm $name\n",
    "        \n",
    "# buffer disolve\n",
    "name_fuego = './data_result_bar/onedate_FW/merged_filter_buff_fire.gpkg'\n",
    "!rm  $name_fuego   \n",
    "!ogr2ogr \\\n",
    "    -t_srs EPSG:32720 \\\n",
    "    -f \"GPKG\" \\\n",
    "    -sql \"select ST_union(ST_buffer(geom,50)) as geometry FROM out WHERE DN='100' \" \\\n",
    "    -dialect SQLITE \\\n",
    "    $name_fuego \\\n",
    "    $name_utm\n",
    "\n",
    "# buffer disolve\n",
    "name_agua = './data_result_bar/onedate_FW/merged_filter_buff_water.gpkg'\n",
    "!rm  $name_agua   \n",
    "!ogr2ogr \\\n",
    "    -t_srs EPSG:32720 \\\n",
    "    -f \"GPKG\" \\\n",
    "    -sql \"select ST_union(ST_buffer(geom,50)) as geometry FROM out WHERE DN='200' \" \\\n",
    "    -dialect SQLITE \\\n",
    "    $name_agua \\\n",
    "    $name_utm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Filter results of small areas**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_area = 100000\n",
    "\n",
    "input_path = './data_result_cba/onedate_FW/cba2.gpkg' \n",
    "output_path = './data_result_cba/onedate_FW/cba2__minarea100000.gpkg'\n",
    "\n",
    "\n",
    "!ogr2ogr \\\n",
    "    -t_srs EPSG:32720 \\\n",
    "    -f \"GPKG\" \\\n",
    "    -sql \"SELECT * FROM out m WHERE ST_Area(geom) > $min_area\" \\\n",
    "    -dialect SQLITE \\\n",
    "    -nln results \\\n",
    "    $output_path \\\n",
    "    $input_path"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyNKlws8+6PcuvJrSWdSmTxJ",
   "include_colab_link": true,
   "name": "4 - Post-procesamiento",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}