training/evaluate-f2-for-yolox-and-norfair-tracking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why this notebook?\n",
    "* Thanks to everyone who upvoted my previous notebook which is now gold medal status: https://www.kaggle.com/alexchwong/stop-guessing-conf-systematically-evaluate-f2\n",
    "* The popularity of this and its cousin notebook by @locbaop: https://www.kaggle.com/locbaop/systematic-evaluate-f2-yolov5 suggest it is useful.\n",
    "* The highest scoring public notebooks use yolov5 with Norfair tracking\n",
    "* So is Norfair actually increasing F2 score, or is it just changing the F2/CONF curve?\n",
    "\n",
    "# Why evaluate F2 score systematically?\n",
    "* Each model is different in how the CONF parameter decides PRECISION and RECALL\n",
    "* Finding which CONF level is often a matter of guesswork, and you are limited by 5 submissions per day.\n",
    "* By systematically evaluating F2 score at each level of CONF, you can have a good idea which CONF will give you the best competition F2 metric.\n",
    "\n",
    "* This assumes:\n",
    "  * Your validation dataset is similar to the hidden test set\n",
    "  * Your model has not seen your validation dataset\n",
    "\n",
    "### Example model used in this notebook\n",
    "\n",
    "* This notebook uses the YOLOX model kindly provided by @remekkinas (https://www.kaggle.com/remekkinas/yolox-cots-models)\n",
    "  * And assumes it is trained on FOLD=4 of 5-fold split performed in: https://www.kaggle.com/remekkinas/yolox-training-pipeline-cots-dataset-lb-0-507\n",
    "* Norfair tracking is implemented and code modified from @parapapapam (https://www.kaggle.com/parapapapam/yolox-inference-tracking-on-cots-lb-0-539)\n",
    "\n",
    "### Warning\n",
    "* Note the iteration time as an estimate to how long this notebook takes to run.\n",
    "* Minimise wasting precious GPU minutes running at large resolutions. Test at small resolutions! The CONF peak should theoretically be the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By popular demand, this notebook now tests all IOUs in the Competition metric F2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-03T00:38:06.68586Z",
     "iopub.status.busy": "2022-02-03T00:38:06.68556Z",
     "iopub.status.idle": "2022-02-03T00:38:06.693852Z",
     "shell.execute_reply": "2022-02-03T00:38:06.692889Z",
     "shell.execute_reply.started": "2022-02-03T00:38:06.685832Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "test_IOU = np.linspace(0.3, 0.8, 11)\n",
    "test_IOU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which IOU should be used to evaluate histograms?\n",
    "* NB the F2 scores are now calculated as the means at each tested IOU, for each level of confidence\n",
    "* Histograms of TPs and FPs are only drawn for the given eval_IOU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-03T00:38:06.69665Z",
     "iopub.status.busy": "2022-02-03T00:38:06.695918Z",
     "iopub.status.idle": "2022-02-03T00:38:06.700733Z",
     "shell.execute_reply": "2022-02-03T00:38:06.699735Z",
     "shell.execute_reply.started": "2022-02-03T00:38:06.696611Z"
    }
   },
   "outputs": [],
   "source": [
    "eval_IOU = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Control Panel (change your settings here)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-03T00:38:07.815325Z",
     "iopub.status.busy": "2022-02-03T00:38:07.814775Z",
     "iopub.status.idle": "2022-02-03T00:38:07.819693Z",
     "shell.execute_reply": "2022-02-03T00:38:07.818891Z",
     "shell.execute_reply.started": "2022-02-03T00:38:07.815286Z"
    }
   },
   "outputs": [],
   "source": [
    "# CONTROL PANEL\n",
    "\n",
    "MODELPATH = \"yolox-cots-models\"\n",
    "\n",
    "MODELNAME = \"yx_l_003\"\n",
    "\n",
    "MODEL_TYPE = \"L\" # YOLOX \"S\", M\", \"L\" or \"X\"\n",
    "\n",
    "# YOLOX Inference Size (h, w) - each must be multiple of 32\n",
    "SIZE = (800, 1280)\n",
    "\n",
    "# YOLOX NMS\n",
    "NMS_THRESHOLD = 0.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:38:09.57385Z",
     "iopub.status.busy": "2022-02-03T00:38:09.573031Z",
     "iopub.status.idle": "2022-02-03T00:38:09.578186Z",
     "shell.execute_reply": "2022-02-03T00:38:09.577294Z",
     "shell.execute_reply.started": "2022-02-03T00:38:09.5738Z"
    }
   },
   "outputs": [],
   "source": [
    "CHECKPOINT_FILE = f\"/kaggle/input/{MODELPATH}/{MODELNAME}.pth\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installation and Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:38:10.832956Z",
     "iopub.status.busy": "2022-02-03T00:38:10.83268Z",
     "iopub.status.idle": "2022-02-03T00:38:12.427908Z",
     "shell.execute_reply": "2022-02-03T00:38:12.427166Z",
     "shell.execute_reply.started": "2022-02-03T00:38:10.832923Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "import os\n",
    "import torch\n",
    "import importlib\n",
    "import cv2 \n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import ast\n",
    "import shutil\n",
    "import sys\n",
    "\n",
    "from tqdm.notebook import tqdm\n",
    "tqdm.pandas()\n",
    "\n",
    "from PIL import Image\n",
    "from IPython.display import display\n",
    "\n",
    "import copy # for deepcopy nested lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## YOLOX Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "_kg_hide-output": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:38:12.831427Z",
     "iopub.status.busy": "2022-02-03T00:38:12.829563Z",
     "iopub.status.idle": "2022-02-03T00:38:33.727115Z",
     "shell.execute_reply": "2022-02-03T00:38:33.725918Z",
     "shell.execute_reply.started": "2022-02-03T00:38:12.831393Z"
    }
   },
   "outputs": [],
   "source": [
    "# Copy YOLOX and required modules from local repository (Kaggle dataset -> https://www.kaggle.com/remekkinas/yolox-cots-models)\n",
    "%cp -r /kaggle/input/yolox-cots-models /kaggle/working/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "_kg_hide-output": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:38:33.72952Z",
     "iopub.status.busy": "2022-02-03T00:38:33.729301Z",
     "iopub.status.idle": "2022-02-03T00:38:57.836409Z",
     "shell.execute_reply": "2022-02-03T00:38:57.835568Z",
     "shell.execute_reply.started": "2022-02-03T00:38:33.729494Z"
    }
   },
   "outputs": [],
   "source": [
    "# Install YOLOX required modules (those already installed by default Kaggle are commented out)\n",
    "%cd /kaggle/working/yolox-cots-models/yolox-dep\n",
    "\n",
    "# !pip install pip-21.3.1-py3-none-any.whl -f ./ --no-index\n",
    "!pip install loguru-0.5.3-py3-none-any.whl -f ./ --no-index\n",
    "!pip install ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl -f ./ --no-index\n",
    "# !pip install onnx-1.8.1-cp37-cp37m-manylinux2010_x86_64.whl -f ./ --no-index\n",
    "# !pip install onnxruntime-1.8.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl -f ./ --no-index\n",
    "# !pip install onnxoptimizer-0.2.6-cp37-cp37m-manylinux2014_x86_64.whl -f ./ --no-index\n",
    "!pip install thop-0.0.31.post2005241907-py3-none-any.whl -f ./ --no-index\n",
    "# !pip install tabulate-0.8.9-py3-none-any.whl -f ./ --no-index\n",
    "#!pip install onnx-simplifier-0.3.6.tar.gz -f ./ --no-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "_kg_hide-output": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:38:57.840021Z",
     "iopub.status.busy": "2022-02-03T00:38:57.839788Z",
     "iopub.status.idle": "2022-02-03T00:39:16.586217Z",
     "shell.execute_reply": "2022-02-03T00:39:16.585401Z",
     "shell.execute_reply.started": "2022-02-03T00:38:57.839993Z"
    }
   },
   "outputs": [],
   "source": [
    "# Install CocoAPI tool\n",
    "%cd /kaggle/working/yolox-cots-models/yolox-dep/cocoapi/PythonAPI\n",
    "\n",
    "!make\n",
    "!make install\n",
    "!python setup.py install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:39:16.589535Z",
     "iopub.status.busy": "2022-02-03T00:39:16.589082Z",
     "iopub.status.idle": "2022-02-03T00:39:16.59493Z",
     "shell.execute_reply": "2022-02-03T00:39:16.594247Z",
     "shell.execute_reply.started": "2022-02-03T00:39:16.589495Z"
    }
   },
   "outputs": [],
   "source": [
    "import pycocotools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the model here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:39:16.596949Z",
     "iopub.status.busy": "2022-02-03T00:39:16.59642Z",
     "iopub.status.idle": "2022-02-03T00:39:16.612412Z",
     "shell.execute_reply": "2022-02-03T00:39:16.611479Z",
     "shell.execute_reply.started": "2022-02-03T00:39:16.596878Z"
    }
   },
   "outputs": [],
   "source": [
    "%cd /kaggle/working/yolox-cots-models/YOLOX\n",
    "\n",
    "from string import Template\n",
    "\n",
    "config_file_template = '''\n",
    "\n",
    "#!/usr/bin/env python3\n",
    "# -*- coding:utf-8 -*-\n",
    "# Copyright (c) Megvii, Inc. and its affiliates.\n",
    "\n",
    "import os\n",
    "\n",
    "from yolox.exp import Exp as MyExp\n",
    "\n",
    "\n",
    "class Exp(MyExp):\n",
    "    def __init__(self):\n",
    "        super(Exp, self).__init__()\n",
    "        self.depth = $model_depth\n",
    "        self.width = $model_width\n",
    "        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n",
    "        self.num_classes = 1\n",
    "\n",
    "'''\n",
    "if MODEL_TYPE == \"X\":\n",
    "    pipeline = Template(config_file_template).substitute(\n",
    "        model_depth = 1.33,\n",
    "        model_width = 1.25\n",
    "    )\n",
    "elif MODEL_TYPE == \"L\":\n",
    "    pipeline = Template(config_file_template).substitute(\n",
    "        model_depth = 1.0,\n",
    "        model_width = 1.0\n",
    "    )\n",
    "elif MODEL_TYPE == \"M\":\n",
    "    pipeline = Template(config_file_template).substitute(\n",
    "        model_depth = 0.67,\n",
    "        model_width = 0.75\n",
    "    )\n",
    "elif MODEL_TYPE == \"S\":\n",
    "    pipeline = Template(config_file_template).substitute(\n",
    "        model_depth = 0.67,\n",
    "        model_width = 0.75\n",
    "    )\n",
    "\n",
    "with open('cots_config.py', 'w') as f:\n",
    "    f.write(pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:39:16.614395Z",
     "iopub.status.busy": "2022-02-03T00:39:16.613815Z",
     "iopub.status.idle": "2022-02-03T00:39:20.810098Z",
     "shell.execute_reply": "2022-02-03T00:39:20.80936Z",
     "shell.execute_reply.started": "2022-02-03T00:39:16.614357Z"
    }
   },
   "outputs": [],
   "source": [
    "%cd /kaggle/working/yolox-cots-models/YOLOX\n",
    "\n",
    "from yolox.utils import postprocess\n",
    "from yolox.data.data_augment import ValTransform\n",
    "\n",
    "COCO_CLASSES = (\n",
    "  \"starfish\",\n",
    ")\n",
    "\n",
    "# get YOLOX experiment\n",
    "current_exp = importlib.import_module('cots_config')\n",
    "exp = current_exp.Exp()\n",
    "\n",
    "num_classes = 1\n",
    "\n",
    "nmsthre = NMS_THRESHOLD\n",
    "\n",
    "# get YOLOX model\n",
    "model = exp.get_model()\n",
    "model.cuda()\n",
    "model.eval()\n",
    "\n",
    "# get custom trained checkpoint\n",
    "ckpt_file = CHECKPOINT_FILE\n",
    "ckpt = torch.load(ckpt_file, map_location=\"cpu\")\n",
    "model.load_state_dict(ckpt[\"model\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:39:20.811713Z",
     "iopub.status.busy": "2022-02-03T00:39:20.811464Z",
     "iopub.status.idle": "2022-02-03T00:39:20.823213Z",
     "shell.execute_reply": "2022-02-03T00:39:20.822175Z",
     "shell.execute_reply.started": "2022-02-03T00:39:20.81168Z"
    }
   },
   "outputs": [],
   "source": [
    "# Modified from https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507\n",
    "# Additions: \n",
    "#     auto converts to xywh format\n",
    "#     converts tensors to list of floats\n",
    "\n",
    "def yolox_inference(img, model, test_size, conf_threshold = 0.4):\n",
    "    bboxes = []\n",
    "    bbclasses = []\n",
    "    scores = []\n",
    "    \n",
    "    preproc = ValTransform(legacy = False)\n",
    "    \n",
    "    tensor_img, _ = preproc(img, None, test_size)\n",
    "    tensor_img = torch.from_numpy(tensor_img).unsqueeze(0)\n",
    "    tensor_img = tensor_img.float()\n",
    "    tensor_img = tensor_img.cuda()\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        outputs = model(tensor_img)\n",
    "        outputs = postprocess(\n",
    "                    outputs, num_classes, conf_threshold,\n",
    "                    nmsthre, class_agnostic=True\n",
    "                )\n",
    "    \n",
    "    if outputs[0] is None:\n",
    "        return [], [], []\n",
    "    \n",
    "    outputs = outputs[0].cpu()\n",
    "    bboxes = outputs[:, 0:4]\n",
    "    \n",
    "    bboxes /= min(test_size[0] / img.shape[0], test_size[1] / img.shape[1])\n",
    "    bbclasses = outputs[:, 6]\n",
    "    scores = outputs[:, 4] * outputs[:, 5]\n",
    "    \n",
    "    if len(bboxes) == 0:\n",
    "        return [], [], []\n",
    "    \n",
    "    bboxes = bboxes.numpy()\n",
    "    \n",
    "    # format to coco\n",
    "    bboxes[:, 2] = bboxes[:, 2] - bboxes[:, 0]\n",
    "    bboxes[:, 3] = bboxes[:, 3] - bboxes[:, 1]    \n",
    "    \n",
    "    # Converts tensors to lists\n",
    "    return bboxes, bbclasses.tolist(), scores.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IOU Calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:39:20.824701Z",
     "iopub.status.busy": "2022-02-03T00:39:20.824378Z",
     "iopub.status.idle": "2022-02-03T00:39:20.83781Z",
     "shell.execute_reply": "2022-02-03T00:39:20.837153Z",
     "shell.execute_reply.started": "2022-02-03T00:39:20.824657Z"
    }
   },
   "outputs": [],
   "source": [
    "def IOU_coco(bbox1, bbox2):\n",
    "    '''\n",
    "        adapted from https://stackoverflow.com/questions/25349178/calculating-percentage-of-bounding-box-overlap-for-image-detector-evaluation\n",
    "    '''\n",
    "    x_left = max(bbox1[0], bbox2[0])\n",
    "    y_top = max(bbox1[1], bbox2[1])\n",
    "    x_right = min(bbox1[0] + bbox1[2], bbox2[0] + bbox2[2])\n",
    "    y_bottom = min(bbox1[1] + bbox1[3], bbox2[1] + bbox2[3])\n",
    "    if x_right < x_left or y_bottom < y_top:\n",
    "        return 0.0\n",
    "    intersection_area = (x_right - x_left) * (y_bottom - y_top)\n",
    "    bb1_area = bbox1[2] * bbox1[3]\n",
    "    bb2_area = bbox2[2] * bbox2[3]\n",
    "    iou = intersection_area / float(bb1_area + bb2_area - intersection_area)\n",
    "\n",
    "    assert iou >= 0.0\n",
    "    assert iou <= 1.0\n",
    "    return iou"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:39:20.839584Z",
     "iopub.status.busy": "2022-02-03T00:39:20.839238Z",
     "iopub.status.idle": "2022-02-03T00:39:20.850727Z",
     "shell.execute_reply": "2022-02-03T00:39:20.849869Z",
     "shell.execute_reply.started": "2022-02-03T00:39:20.839549Z"
    }
   },
   "outputs": [],
   "source": [
    "# Modified from https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507\n",
    "\n",
    "def draw_yolox_predictions(img, bboxes, scores, bbclasses, classes_dict, boxcolor = (0,0,255)):\n",
    "    outimg = img.copy()\n",
    "    for i in range(len(bboxes)):\n",
    "        box = bboxes[i]\n",
    "        cls_id = int(bbclasses[i])\n",
    "        score = scores[i]\n",
    "        x0 = int(box[0])\n",
    "        y0 = int(box[1])\n",
    "        x1 = x0 + int(box[2])\n",
    "        y1 = y0 + int(box[3])\n",
    "\n",
    "        cv2.rectangle(outimg, (x0, y0), (x1, y1), boxcolor, 2)\n",
    "        cv2.putText(outimg, '{}:{:.1f}%'.format(classes_dict[cls_id], score * 100), (x0, y0 - 3), cv2.FONT_HERSHEY_PLAIN, 0.8, boxcolor, thickness = 1)\n",
    "    return outimg\n",
    "\n",
    "COCO_CLASSES = (\"starfish\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Norfair"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "_kg_hide-output": false,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:39:20.854705Z",
     "iopub.status.busy": "2022-02-03T00:39:20.854274Z",
     "iopub.status.idle": "2022-02-03T00:40:30.542542Z",
     "shell.execute_reply": "2022-02-03T00:40:30.541646Z",
     "shell.execute_reply.started": "2022-02-03T00:39:20.854661Z"
    }
   },
   "outputs": [],
   "source": [
    "# norfair dependencies\n",
    "%cd /kaggle/input/norfair031py3/\n",
    "!pip install commonmark-0.9.1-py2.py3-none-any.whl -f ./ --no-index\n",
    "!pip install rich-9.13.0-py3-none-any.whl\n",
    "\n",
    "!mkdir /kaggle/working/tmp\n",
    "!cp -r /kaggle/input/norfair031py3/filterpy-1.4.5/filterpy-1.4.5/ /kaggle/working/tmp/\n",
    "%cd /kaggle/working/tmp/filterpy-1.4.5/\n",
    "!pip install .\n",
    "!rm -rf /kaggle/working/tmp\n",
    "\n",
    "# norfair\n",
    "%cd /kaggle/input/norfair031py3/\n",
    "!pip install norfair-0.3.1-py3-none-any.whl -f ./ --no-index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:40:30.54606Z",
     "iopub.status.busy": "2022-02-03T00:40:30.545812Z",
     "iopub.status.idle": "2022-02-03T00:40:31.208706Z",
     "shell.execute_reply": "2022-02-03T00:40:31.207958Z",
     "shell.execute_reply.started": "2022-02-03T00:40:30.546031Z"
    }
   },
   "outputs": [],
   "source": [
    "from norfair import Detection, Tracker\n",
    "\n",
    "# Helper to convert bbox in format [x_min, y_min, w, h] and scores to norfair.Detection class\n",
    "def to_norfair(bboxes, scores, frame_id):\n",
    "    result = []\n",
    "    for bb, score in zip(bboxes, scores):\n",
    "        xc, yc = bb[0] + int(round(bb[2] / 2.0)), bb[1] + int(round(bb[3] / 2.0))\n",
    "        w, h = bb[2], bb[3]\n",
    "        result.append(Detection(points=np.array([xc, yc]), scores=np.array([score]), data=np.array([w, h, frame_id])))\n",
    "        \n",
    "    return result\n",
    "\n",
    "# Euclidean distance function to match detections on this frame with tracked_objects from previous frames\n",
    "def euclidean_distance(detection, tracked_object):\n",
    "    return np.linalg.norm(detection.points - tracked_object.estimate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tracker list initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:40:31.210331Z",
     "iopub.status.busy": "2022-02-03T00:40:31.210075Z",
     "iopub.status.idle": "2022-02-03T00:40:31.215319Z",
     "shell.execute_reply": "2022-02-03T00:40:31.214571Z",
     "shell.execute_reply.started": "2022-02-03T00:40:31.210297Z"
    }
   },
   "outputs": [],
   "source": [
    "tracker = Tracker(\n",
    "    distance_function=euclidean_distance, \n",
    "    distance_threshold=30,\n",
    "    hit_inertia_min=3,\n",
    "    hit_inertia_max=6,\n",
    "    initialization_delay=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function to parse YOLOX output to tracker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:40:31.218034Z",
     "iopub.status.busy": "2022-02-03T00:40:31.216545Z",
     "iopub.status.idle": "2022-02-03T00:40:31.227645Z",
     "shell.execute_reply": "2022-02-03T00:40:31.226826Z",
     "shell.execute_reply.started": "2022-02-03T00:40:31.217995Z"
    }
   },
   "outputs": [],
   "source": [
    "def norfair_track(tracker, frame_id, bboxes, bbclasses, scores):\n",
    "    \"\"\"\n",
    "        inputs:\n",
    "        * tracker: the tracker object to process\n",
    "        * bboxes, bbclasses, scores: processed model output\n",
    "    \"\"\"\n",
    "    bboxes_out = []\n",
    "    bbclasses_out = []\n",
    "    scores_out = []\n",
    "    \n",
    "    if len(bboxes):\n",
    "        bboxes_out = copy.deepcopy(bboxes)\n",
    "        bbclasses_out = copy.deepcopy(bbclasses)\n",
    "        scores_out = copy.deepcopy(scores)\n",
    "    \n",
    "    tracked_objects = tracker.update(detections=to_norfair(bboxes, scores, frame_id))\n",
    "    \n",
    "    for tobj in tracked_objects:\n",
    "        bbox_width, bbox_height, last_detected_frame_id = tobj.last_detection.data\n",
    "        if last_detected_frame_id == frame_id:  # Skip objects that were detected on current frame\n",
    "            continue\n",
    "\n",
    "        # Add objects that have no detections on current frame to predictions\n",
    "        xc, yc = tobj.estimate[0]\n",
    "        x_min, y_min = int(round(xc - bbox_width / 2)), int(round(yc - bbox_height / 2))\n",
    "        score = tobj.last_detection.scores[0]\n",
    "        \n",
    "        bboxes_out.append([x_min, y_min, bbox_width, bbox_height])\n",
    "        bbclasses_out.append(0)\n",
    "        scores_out.append(score)\n",
    "    \n",
    "    return bboxes_out, bbclasses_out, scores_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrieve video frames not used in training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:40:31.229201Z",
     "iopub.status.busy": "2022-02-03T00:40:31.228833Z",
     "iopub.status.idle": "2022-02-03T00:40:47.216652Z",
     "shell.execute_reply": "2022-02-03T00:40:47.215914Z",
     "shell.execute_reply.started": "2022-02-03T00:40:31.229168Z"
    }
   },
   "outputs": [],
   "source": [
    "%cd /kaggle/working\n",
    "\n",
    "from sklearn.model_selection import GroupKFold\n",
    "\n",
    "def get_bbox(annots):\n",
    "    bboxes = [list(annot.values()) for annot in annots]\n",
    "    return bboxes\n",
    "\n",
    "def get_path(row):\n",
    "    row['image_path'] = f'{ROOT_DIR}/train_images/video_{row.video_id}/{row.video_frame}.jpg'\n",
    "    return row\n",
    "\n",
    "ROOT_DIR  = '/kaggle/input/tensorflow-great-barrier-reef/'\n",
    "\n",
    "df = pd.read_csv(\"/kaggle/input/tensorflow-great-barrier-reef/train.csv\")\n",
    "\n",
    "# Don't filter for annotated frames. Include frames with no bboxes as well!\n",
    "df[\"num_bbox\"] = df['annotations'].apply(lambda x: str.count(x, 'x'))\n",
    "df_train = df\n",
    "\n",
    "# Annotations \n",
    "df_train['annotations'] = df_train['annotations'].progress_apply(lambda x: ast.literal_eval(x))\n",
    "df_train['bboxes'] = df_train.annotations.progress_apply(get_bbox)\n",
    "\n",
    "df_train = df_train.progress_apply(get_path, axis=1)\n",
    "\n",
    "kf = GroupKFold(n_splits = 5) \n",
    "df_train = df_train.reset_index(drop=True)\n",
    "df_train['fold'] = -1\n",
    "for fold, (train_idx, val_idx) in enumerate(kf.split(df_train, y = df_train.video_id.tolist(), groups=df_train.sequence)):\n",
    "    df_train.loc[val_idx, 'fold'] = fold\n",
    "\n",
    "df_train.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select the dataset that your model hasn't seen!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": false,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:40:47.228868Z",
     "iopub.status.busy": "2022-02-03T00:40:47.226078Z",
     "iopub.status.idle": "2022-02-03T00:40:47.242611Z",
     "shell.execute_reply": "2022-02-03T00:40:47.241813Z",
     "shell.execute_reply.started": "2022-02-03T00:40:47.228829Z"
    }
   },
   "outputs": [],
   "source": [
    "df_test = df_train[df_train.fold == 4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get image paths and ground truth BB's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:40:47.247024Z",
     "iopub.status.busy": "2022-02-03T00:40:47.246621Z",
     "iopub.status.idle": "2022-02-03T00:40:47.515706Z",
     "shell.execute_reply": "2022-02-03T00:40:47.514972Z",
     "shell.execute_reply.started": "2022-02-03T00:40:47.246987Z"
    }
   },
   "outputs": [],
   "source": [
    "# deepcopy is required to avoid contaminating input data\n",
    "\n",
    "df_sample = df_test\n",
    "image_paths = df_sample.image_path.tolist()\n",
    "gt = copy.deepcopy(df_sample.bboxes.tolist())\n",
    "gtmem = copy.deepcopy(df_sample.bboxes.tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test your model is working"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:40:47.51744Z",
     "iopub.status.busy": "2022-02-03T00:40:47.517048Z",
     "iopub.status.idle": "2022-02-03T00:40:53.045013Z",
     "shell.execute_reply": "2022-02-03T00:40:53.044324Z",
     "shell.execute_reply.started": "2022-02-03T00:40:47.517404Z"
    }
   },
   "outputs": [],
   "source": [
    "i = 1350\n",
    "TEST_IMAGE_PATH = image_paths[i]\n",
    "img = cv2.imread(TEST_IMAGE_PATH)\n",
    "\n",
    "# Insert your inference code here:\n",
    "bboxes, bbclasses, scores = yolox_inference(img, model, SIZE, 0.01)\n",
    "\n",
    "# Draw Green ground truth box\n",
    "out_image = draw_yolox_predictions(img, gt[i], [1.0] * len(gt[i]), [0] * len(gt[i]), COCO_CLASSES, (0,255,0))\n",
    "\n",
    "# Draw Red inference box\n",
    "out_image = draw_yolox_predictions(out_image, bboxes, scores, bbclasses, COCO_CLASSES, (0,0,255))\n",
    "\n",
    "# Convert BGR to RGB\n",
    "out_image = cv2.cvtColor(out_image, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "display(Image.fromarray(out_image))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assess Model Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:43:40.347727Z",
     "iopub.status.busy": "2022-02-03T00:43:40.346794Z",
     "iopub.status.idle": "2022-02-03T00:43:40.358686Z",
     "shell.execute_reply": "2022-02-03T00:43:40.357877Z",
     "shell.execute_reply.started": "2022-02-03T00:43:40.347684Z"
    }
   },
   "outputs": [],
   "source": [
    "def calc_pos_and_negs(bboxes, scores, gt_bboxes, IOU):\n",
    "    TP = []\n",
    "    FP = []\n",
    "    FN = 0\n",
    "    \n",
    "    gt0 = copy.deepcopy(gt_bboxes)\n",
    "    bb = copy.deepcopy(bboxes)\n",
    "    \n",
    "    if len(bboxes) == 0:\n",
    "        # all gt are false negative\n",
    "        FN += len(gt0)\n",
    "    else:\n",
    "        for idx, b in enumerate(bb):\n",
    "            b.append(scores[idx])\n",
    "        bb.sort(key = lambda x: x[4], reverse = True)\n",
    "        \n",
    "        if len(gt0) == 0:\n",
    "            # all bboxes are false positives\n",
    "            for b in bb:\n",
    "                FP.append(b[4])\n",
    "        else:\n",
    "            # match bbox with gt\n",
    "            for b in bb:\n",
    "                matched = False\n",
    "                for g in gt0:\n",
    "                    # check whether gt box is already matched to an inference bb\n",
    "                    if len(g) == 4:\n",
    "                        # g bbox is unmatched\n",
    "                        if IOU_coco(b, g) >= IOU:\n",
    "                            g.append(b[4]) # assign confidence values to g; marks g as matched\n",
    "                            matched = True\n",
    "                            TP.append(b[4])\n",
    "                            break\n",
    "                if not matched:\n",
    "                    FP.append(b[4])\n",
    "            for g in gt0:\n",
    "                if len(g) == 4:\n",
    "                    FN += 1\n",
    "    return TP, FP, FN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": false,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:43:42.820106Z",
     "iopub.status.busy": "2022-02-03T00:43:42.817803Z",
     "iopub.status.idle": "2022-02-03T00:43:55.701149Z",
     "shell.execute_reply": "2022-02-03T00:43:55.700117Z",
     "shell.execute_reply.started": "2022-02-03T00:43:42.820049Z"
    }
   },
   "outputs": [],
   "source": [
    "%cd /kaggle/working\n",
    "\n",
    "# Confidence scores of true positives, false positives and count false negatives\n",
    "\n",
    "# Yolov5\n",
    "TP_base = [[] for r in test_IOU] # Confidence scores of true positives\n",
    "FP_base = [[] for r in test_IOU] # Confidence scores of true positives\n",
    "FN_base = [0 for r in test_IOU]  # Count of false negative boxes\n",
    "\n",
    "# Norfair\n",
    "TP_nf = [[] for r in test_IOU] # Confidence scores of true positives\n",
    "FP_nf = [[] for r in test_IOU] # Confidence scores of true positives\n",
    "FN_nf = [0 for r in test_IOU]  # Count of false negative boxes\n",
    "\n",
    "fid = 0\n",
    "\n",
    "for i in tqdm(range(len(image_paths))):\n",
    "# for i in tqdm(range(1250, 1450)):\n",
    "    TEST_IMAGE_PATH = image_paths[i]\n",
    "    img = cv2.imread(TEST_IMAGE_PATH)\n",
    "    \n",
    "    # Base yolov5\n",
    "    bboxes, bbclasses, scores = yolox_inference(img, model, SIZE, 0.01)\n",
    "    \n",
    "    # Convert numpy array to nested list\n",
    "    bboxes = [b.tolist() for b in bboxes]\n",
    "    \n",
    "    # Base norfair\n",
    "    bboxes_track, bbclasses_track, scores_track = norfair_track(tracker, fid, bboxes, bbclasses, scores)\n",
    "    \n",
    "    # Increment frame ID\n",
    "    fid += 1\n",
    "    \n",
    "    # yolov5 evaluation\n",
    "    for idx, IOU in enumerate(test_IOU):\n",
    "        TP, FP, FN = calc_pos_and_negs(bboxes, scores, gt[i], IOU)\n",
    "        if len(TP):\n",
    "            TP_base[idx].append(TP)\n",
    "        if len(FP):\n",
    "            FP_base[idx].append(FP)\n",
    "        FN_base[idx] += FN\n",
    "\n",
    "        # yolov5 norfair evaluation\n",
    "        TP, FP, FN = calc_pos_and_negs(bboxes_track, scores_track, gt[i], IOU)\n",
    "        if len(TP):\n",
    "            TP_nf[idx].append(TP)\n",
    "        if len(FP):\n",
    "            FP_nf[idx].append(FP)\n",
    "        FN_nf[idx] += FN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:44:12.029443Z",
     "iopub.status.busy": "2022-02-03T00:44:12.029177Z",
     "iopub.status.idle": "2022-02-03T00:44:12.036197Z",
     "shell.execute_reply": "2022-02-03T00:44:12.03543Z",
     "shell.execute_reply.started": "2022-02-03T00:44:12.029414Z"
    }
   },
   "outputs": [],
   "source": [
    "# Flatten nested lists\n",
    "\n",
    "for idx, IOU in enumerate(test_IOU):\n",
    "    TP_base[idx] = [item for sublist in TP_base[idx] for item in sublist]\n",
    "    FP_base[idx] = [item for sublist in FP_base[idx] for item in sublist]\n",
    "    TP_nf[idx] = [item for sublist in TP_nf[idx] for item in sublist]\n",
    "    FP_nf[idx] = [item for sublist in FP_nf[idx] for item in sublist]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Display your model's Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:44:15.867071Z",
     "iopub.status.busy": "2022-02-03T00:44:15.866624Z",
     "iopub.status.idle": "2022-02-03T00:44:15.875928Z",
     "shell.execute_reply": "2022-02-03T00:44:15.874871Z",
     "shell.execute_reply.started": "2022-02-03T00:44:15.867025Z"
    }
   },
   "outputs": [],
   "source": [
    "# Which IOU to evaluate in histograms\n",
    "eval_IOU_idx = np.where(np.isclose(test_IOU,eval_IOU))\n",
    "eval_IOU_idx = eval_IOU_idx[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Base YOLOV5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:44:17.661373Z",
     "iopub.status.busy": "2022-02-03T00:44:17.661102Z",
     "iopub.status.idle": "2022-02-03T00:44:17.667628Z",
     "shell.execute_reply": "2022-02-03T00:44:17.666588Z",
     "shell.execute_reply.started": "2022-02-03T00:44:17.661345Z"
    }
   },
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:44:18.883477Z",
     "iopub.status.busy": "2022-02-03T00:44:18.883031Z",
     "iopub.status.idle": "2022-02-03T00:44:19.260099Z",
     "shell.execute_reply": "2022-02-03T00:44:19.258694Z",
     "shell.execute_reply.started": "2022-02-03T00:44:18.883444Z"
    }
   },
   "outputs": [],
   "source": [
    "plt.hist(TP_base[eval_IOU_idx], 100)\n",
    "plt.title(f\"CONF of true positives, base YOLOX @ IOU {eval_IOU}\")\n",
    "plt.xlabel('CONF')\n",
    "plt.ylabel('TP count')\n",
    "plt.show()\n",
    "\n",
    "print(f'True positives = {len(TP_base[eval_IOU_idx])}')\n",
    "print(f'False negatives = {FN_base[eval_IOU_idx]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:44:20.811503Z",
     "iopub.status.busy": "2022-02-03T00:44:20.811235Z",
     "iopub.status.idle": "2022-02-03T00:44:21.216156Z",
     "shell.execute_reply": "2022-02-03T00:44:21.214749Z",
     "shell.execute_reply.started": "2022-02-03T00:44:20.811474Z"
    }
   },
   "outputs": [],
   "source": [
    "plt.hist(FP_base[eval_IOU_idx], 100)\n",
    "plt.title(f\"CONF of false positives, base YOLOX @ IOU {eval_IOU}\")\n",
    "plt.xlabel('CONF')\n",
    "plt.ylabel('FP count')\n",
    "plt.show()\n",
    "\n",
    "print(f'False positives = {len(FP_base[eval_IOU_idx])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:24:19.451624Z",
     "iopub.status.busy": "2022-02-03T00:24:19.451304Z",
     "iopub.status.idle": "2022-02-03T00:24:19.793515Z",
     "shell.execute_reply": "2022-02-03T00:24:19.792791Z",
     "shell.execute_reply.started": "2022-02-03T00:24:19.451593Z"
    }
   },
   "outputs": [],
   "source": [
    "from statistics import mean\n",
    "\n",
    "F2list = []\n",
    "F2max = 0.0\n",
    "F2maxat = -1.0\n",
    "F2mean = 0.0\n",
    "\n",
    "for c in np.arange(0.0, 1.0, 0.01):\n",
    "    F2temp = []\n",
    "    \n",
    "    for idx, IOU in enumerate(test_IOU):\n",
    "        FNcount = FN_base[idx] + sum(1 for i in TP_base[idx] if i < c)\n",
    "        TPcount = sum(1 for i in TP_base[idx] if i >= c)\n",
    "        FPcount = sum(1 for i in FP_base[idx] if i >= c)\n",
    "        R = TPcount / (TPcount + FNcount + 0.0001)\n",
    "        P = TPcount / (TPcount + FPcount + 0.0001)\n",
    "        F2 = (5 * P * R) / (4 * P + R + 0.0001)\n",
    "        F2temp.append(F2)\n",
    "    \n",
    "    F2mean = mean(F2temp)\n",
    "    F2list.append((c, F2mean))\n",
    "    if F2max < F2mean:\n",
    "        F2max = F2mean\n",
    "        F2maxat = c\n",
    "\n",
    "plt.scatter(*zip(*F2list))\n",
    "plt.title(\"CONF vs F2 score: Base YOLOX\")\n",
    "plt.xlabel('CONF')\n",
    "plt.ylabel('F2')\n",
    "plt.show()\n",
    "\n",
    "print(f'F2 max is {F2max} at CONF = {F2maxat}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### YOLOV5 Norfair"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:24:22.679409Z",
     "iopub.status.busy": "2022-02-03T00:24:22.678612Z",
     "iopub.status.idle": "2022-02-03T00:24:23.093654Z",
     "shell.execute_reply": "2022-02-03T00:24:23.09242Z",
     "shell.execute_reply.started": "2022-02-03T00:24:22.679369Z"
    }
   },
   "outputs": [],
   "source": [
    "plt.hist(TP_nf[eval_IOU_idx], 100)\n",
    "plt.title(f\"CONF of true positives, YOLOX Norfair @ IOU {eval_IOU}\")\n",
    "plt.xlabel('CONF')\n",
    "plt.ylabel('TP count')\n",
    "plt.show()\n",
    "\n",
    "print(f'True positives = {len(TP_nf[eval_IOU_idx])}')\n",
    "print(f'False negatives = {FN_nf[eval_IOU_idx]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:24:25.285884Z",
     "iopub.status.busy": "2022-02-03T00:24:25.285066Z",
     "iopub.status.idle": "2022-02-03T00:24:25.664753Z",
     "shell.execute_reply": "2022-02-03T00:24:25.663421Z",
     "shell.execute_reply.started": "2022-02-03T00:24:25.285828Z"
    }
   },
   "outputs": [],
   "source": [
    "plt.hist(FP_nf[eval_IOU_idx], 100)\n",
    "plt.title(f\"CONF of false positives, YOLOX Norfair @ IOU {eval_IOU}\")\n",
    "plt.xlabel('CONF')\n",
    "plt.ylabel('FP count')\n",
    "plt.show()\n",
    "\n",
    "print(f'False positives = {len(FP_nf[eval_IOU_idx])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-02-03T00:24:27.99548Z",
     "iopub.status.busy": "2022-02-03T00:24:27.994991Z",
     "iopub.status.idle": "2022-02-03T00:24:28.362738Z",
     "shell.execute_reply": "2022-02-03T00:24:28.361956Z",
     "shell.execute_reply.started": "2022-02-03T00:24:27.995438Z"
    }
   },
   "outputs": [],
   "source": [
    "F2list = []\n",
    "F2max = 0.0\n",
    "F2maxat = -1.0\n",
    "F2mean = 0.0\n",
    "\n",
    "for c in np.arange(0.0, 1.0, 0.01):\n",
    "    F2temp = []\n",
    "    \n",
    "    for idx, IOU in enumerate(test_IOU):\n",
    "        FNcount = FN_nf[idx] + sum(1 for i in TP_nf[idx] if i < c)\n",
    "        TPcount = sum(1 for i in TP_nf[idx] if i >= c)\n",
    "        FPcount = sum(1 for i in FP_nf[idx] if i >= c)\n",
    "        R = TPcount / (TPcount + FNcount + 0.0001)\n",
    "        P = TPcount / (TPcount + FPcount + 0.0001)\n",
    "        F2 = (5 * P * R) / (4 * P + R + 0.0001)\n",
    "        F2temp.append(F2)\n",
    "    \n",
    "    F2mean = mean(F2temp)\n",
    "    F2list.append((c, F2mean))\n",
    "    if F2max < F2mean:\n",
    "        F2max = F2mean\n",
    "        F2maxat = c\n",
    "\n",
    "plt.scatter(*zip(*F2list))\n",
    "plt.title(\"CONF vs F2 score: YOLOV5 Norfair\")\n",
    "plt.xlabel('CONF')\n",
    "plt.ylabel('F2')\n",
    "plt.show()\n",
    "\n",
    "print(f'F2 max is {F2max} at CONF = {F2maxat}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true
   },
   "outputs": [],
   "source": [
    "!rm -rf /kaggle/working/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Credits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* https://www.kaggle.com/remekkinas/yolox-training-pipeline-cots-dataset-lb-0-507\n",
    "* https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}