Approach_to_the_Problem_&_Inspecting_and_Cleaning_the_Required_Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&ensp;\n",
    "[Home Page](../Start_Here.ipynb)\n",
    "\n",
    "\n",
    "[Previous Notebook](The_Problem_Statement.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "[1](The_Problem_Statement.ipynb)\n",
    "[2]\n",
    "[3](Manipulation_of_Image_Data_and_Category_Determination_using_Text_Data.ipynb)\n",
    "[4](Countering_Data_Imbalance.ipynb)\n",
    "[5](Competition.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "[Next Notebook](Manipulation_of_Image_Data_and_Category_Determination_using_Text_Data.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tropical Cyclone Intensity Estimation using a Deep  Convolutional Neural Network - Part 2 \n",
    "\n",
    "**Contents of this Notebook:**\n",
    "\n",
    "- [Approaching the Problem](#Approaching-the-Problem)\n",
    "- [Data](#Data)\n",
    "- [Tasks](#Task)\n",
    "- [Model and Loss](#Model-and-Loss) \n",
    "- [Training and Evaluation](#Training-and-Evaluation) \n",
    "- [Working with the Image Dataset](#Working-with-the-Image-Dataset)\n",
    "- [Working with the Text Dataset](#Working-with-the-Text-Data)\n",
    "\n",
    "\n",
    "**By the end of this Notebook participants will:**\n",
    "\n",
    "- Understand the Dataset\n",
    "- Learn Pre-processing of the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Approaching the Problem\n",
    "\n",
    "During this lab we will be making use of the following buckets to help us understand how a Machine Learning project should be planned and executed: \n",
    "\n",
    "1. **Data**: To start with any ML project we need data which is pre-processed and can be fed into the network.\n",
    "2. **Task**: There are many tasks present in ML, we need to make sure we understand and define the problem statement accurately.\n",
    "3. **Model**: We need to build our model, which is neither too deep or complex, thereby taking a lot of computational power or too small that it could not learn the important features.\n",
    "4. **Loss**: Out of the many _loss functions_ present, we need to carefully choose a _loss function_ which is suitable for the task we are about to carry out.\n",
    "5. **Learning**: As we mentioned in our last notebook, there are a variety of _optimisers_ each with their advantages and disadvantages. So here we choose an _optimiser_ which is suitable for our task and train our model using the set hyperparameters.\n",
    "6. **Evaluation**: This is a crucial step in the process to determine if our model has learnt the features properly by analysing how it performs when unseen data is given to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "##### The Image data will be annotated using the Text Data to create our dataset from which the model will learn to classify the type of cyclone.\n",
    "\n",
    "\n",
    "**Optional** : [Downloading Dataset](Downloading_Images.ipynb)\n",
    "\n",
    "The dataset has already been downloaded for participants, and we will be using it.\n",
    "\n",
    "Example of images that will be fed into our model  : \n",
    "\n",
    "<table><tr>\n",
    "<td><img src=\"images/example.jpg\" alt=\"Drawing\" style=\"width: 320px;\"/></td>\n",
    "<td><img src=\"images/example1.jpg\" alt=\"Drawing\" style=\"width: 320px;\"/></td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "*Source: https://www.nrlmry.navy.mil/*\n",
    "\n",
    "#### Each Image will be annotated to a category of Cyclone Intensity using the text data with the help of the following table :\n",
    "\n",
    "![alt text](images/cat.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task \n",
    "\n",
    "There are a variety of tasks present in DL, and the task we are about to do is called Multi-class Classification. \n",
    "\n",
    "Here, multiple classes are present, and the model needs to classify the image into the correct class. \n",
    "\n",
    "The Classes here are the intensity of the tropical cyclone which will be estimated using the wind speeds of the cyclone: \n",
    "- NC ( No Category         , $\\leq 20$ knots)\n",
    "- TD ( Tropical Depression , $20-33$ knots)\n",
    "- TS ( Topical Storm       , $34-63$ knots)\n",
    "- H1 ( Category One        , $64-82$ knots)\n",
    "- H2 ( Category Two        , $83-95$ knots)\n",
    "- H3 ( Category Three      , $96-112$ knots)\n",
    "- H4 ( Category Four       , $113-136$ knots)\n",
    "- H5 ( Category Five       , $\\geq 137$ knots)\n",
    "\n",
    "##### Example of Multi-Class Classification is : \n",
    "<img src=\"images/multi_class1.png\" alt=\"Drawing\" style=\"width: 520px;\"/></td>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model and Loss \n",
    "\n",
    "#### We will be using the model given in the research paper : \n",
    "\n",
    "The Hyper-parameters in this model ( kernel size, number of hidden layers ) is tailor-made for this project. \n",
    "\n",
    "![alt text](images/model.png)\n",
    "\n",
    "\n",
    "### Loss Function :\n",
    "\n",
    "We have seen about three different Multi-class loss function, they are :\n",
    "- *Multi-Class Cross-Entropy Loss*\n",
    "- Sparse Multi-class Cross-Entropy Loss\n",
    "- Kullback Leibler Divergence Loss\n",
    "\n",
    "We will be using the Multi-class Cross-Entropy loss function for this classification. And one can use any of the three loss function for this task.\n",
    "\n",
    "##### Multi-Class Cross-Entropy loss : \n",
    "\n",
    "Before we understand how the loss value is calculated, let us understand how the outputs are produced.\n",
    "\n",
    "When we train our model , we convert all the categories to a array of 1's and 0's. Let us assume we have a model to predict if a given image is a cat or a dog, we will then start labelling all the outputs in our dataset and assign `cat = [ 1 , 0 ]` and `dog = [ 0 , 1 ] ` and we will then train our model based on it, our model will then predict the probabilities of the output of what it thinks it to be, let's say we give it an image of a cat, we get the output to be `[ 0.87 , 0.13]` which implies that the model is 87% confident that it is a cat,but this is still not good enough we then calculate the error in this case with the following equation. \n",
    "\n",
    "$$ Loss = J(w) = - \\frac{1}{N}\\sum_{n=0}^{N} \\left[ y_n \\log{\\hat{y_n}} + (1 - y_n) \\log{(1-\\hat{y_n})}  \\right] $$\n",
    "\n",
    "So ,if we try calculating the loss in our case from the equation , we will try calculating loss value for different values \n",
    "\n",
    "$ Loss = - \\frac{1}{2}\\left[ \\log{0.87} + (1 - 0) \\log{(1 - 0.13)} \\right] = -1 * \\log{0.87} = 0.060 $\n",
    "\n",
    "If our model trained over time and if the output would be `[ 0.90 , 0.10 ]` , then let's calculate our loss again.\n",
    "\n",
    "$ Loss = - \\frac{1}{2}\\left[ \\log{0.90} + (1 - 0) \\log{(1 - 0.10)} \\right] = -1 * \\log{0.90} = 0.045 $\n",
    "\n",
    "So ,now we notice our loss value to have reduced , so using these equations our model learns to understand how it performs and updates the weights to perform better. \n",
    "\n",
    "### Optimizer : \n",
    "\n",
    "In this model, we will be using SGD Optimizer ( Stochastic Gradient Descent ) Optimizer.\n",
    "\n",
    "##### Gradient Descent : \n",
    "\n",
    "Gradient descent is an optimization algorithm used to minimize some function by iteratively moving in the direction of steepest descent as defined by the negative of the gradient. \n",
    "\n",
    "Let us get an insight to understand how gradient descent works :\n",
    "\n",
    "Starting at the top of the mountain, we take our first step downhill in the direction specified by the negative gradient. Next, we recalculate the negative gradient (passing in the coordinates of our new point) and take another step in the direction it specifies. We continue this process iteratively until we get to the bottom of our graph, or to a point where we can no longer move downhill–a local minimum.\n",
    "\n",
    "<td><img src=\"images/grad.jpg\" alt=\"Drawing\" style=\"width: 420px;\"/></td>\n",
    "\n",
    "*Source: https://towardsdatascience.com/linear-regression-using-gradient-descent-97a6c8700931*\n",
    "\n",
    "GD runs through all the samples in training set to do a single update for a parameter in a particular iteration. In SGD, on the other hand, you use only one or subset of training sample from your training set to do the update for a parameter in a particular iteration. \n",
    "\n",
    "Using SGD will be faster because only one training sample is used and it starts improving itself right away from the first sample.\n",
    "\n",
    "\n",
    "## Training and Evaluation \n",
    "\n",
    "We will split our dataset into three different sets :\n",
    "\n",
    "- Training Set\n",
    "    - 72 % of the Dataset\n",
    "- Test Set \n",
    "    - 18 % of the Dataset\n",
    "- Validation Set \n",
    "    - 10% of the Dataset \n",
    "\n",
    "You are free to play around with these ratios in the next notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summary of our approach : \n",
    "\n",
    "![alt_text](images/now.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with the Image Dataset \n",
    "\n",
    "The images are downloaded in the 'Dataset' folder\n",
    "\n",
    "### To work with the images, we need to understand the hierarchy of the Stored Images : \n",
    "\n",
    "The Hierarchy of this Dataset is similar to that of U.S.Naval Database for storing images, and this has been retained for storing the images as to easily scale / add more features to this.\n",
    "\n",
    "Let's now understand how it is arranged : \n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Dataset                                  --> Root Folder\n",
    "└── tcdat                                --> First Nested Folder ( tcdat = tropical cyclone database )\n",
    "    ├── tc04                             --> Second Nested Folder ( tcxx = xx stand for yeah 20xx , tc04 = 2004) \n",
    "    │   └── ATL                          --> Third Nested Folder ( ATL -> Stands for Atlantic Cyclones)\n",
    "    │       ├── 01L.ALEX                 --> Fourth Nested Folder ( Name of the Cyclone)\n",
    "    │       │   └── ir                   --> Fifth Nested Folder ( Type of Image , We will be working with IR only)\n",
    "    │       │       └── geo              --> Sixth Nested Folder\n",
    "    │       │           └── 1km          --> Seventh Nested Folder ( Range of the Image ) \n",
    "                                         --> Images will be contained Here\n",
    "                                      \n",
    "So, Now we need to make sure we can recurse through this :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tc14-> 06L.EDOUARD --> 1563\n",
      "tc09-> 03L.BILL --> 643\n",
      "tc09-> 07L.FRED --> 1003\n",
      "tc10-> 06L.DANIELLE --> 905\n",
      "tc10-> 11L.IGOR --> 1047\n",
      "tc10-> 13L.KARL --> 553\n",
      "tc10-> 07L.EARL --> 1018\n",
      "tc10-> 12L.JULIA --> 1739\n",
      "tc07-> 04L.DEAN --> 914\n",
      "tc07-> 06L.FELIX --> 701\n",
      "tc05-> 27L.BETA --> 303\n",
      "tc05-> 25L.WILMA --> 503\n",
      "tc05-> 12L.KATRINA --> 421\n",
      "tc05-> 18L.RITA --> 380\n",
      "tc05-> 04L.DENNIS --> 447\n",
      "tc05-> 05L.EMILY --> 870\n",
      "tc06-> 07L.GORDON --> 686\n",
      "tc06-> 08L.HELENE --> 881\n",
      "tc04-> 09L.IVAN --> 1974\n",
      "tc04-> 12L.KARL --> 772\n",
      "tc04-> 01L.ALEX --> 783\n",
      "tc04-> 03L.CHARLEY --> 526\n",
      "tc04-> 06L.FRANCES --> 1670\n",
      "tc04-> 11L.JEANNE --> 1347\n",
      "tc11-> 09L.IRENE --> 1502\n",
      "tc11-> 16L.OPHELIA --> 1370\n",
      "tc11-> 12L.KATIA --> 1361\n",
      "tc08-> 07L.GUSTAV --> 1594\n",
      "tc08-> 09L.IKE --> 1543\n",
      "tc08-> 02L.BERTHA --> 1724\n",
      "tc08-> 15L.OMAR --> 648\n",
      "tc12-> 13L.MICHAEL --> 1220\n",
      "\n",
      "Total number of images present in the Dataset : 32611\n"
     ]
    }
   ],
   "source": [
    "#Import Necessary Libraries \n",
    "import os\n",
    "\n",
    "#List what is present in 'Dataset/' directory\n",
    "dir ='Dataset/'\n",
    "a = os.listdir(dir)\n",
    "try :\n",
    "    a.remove('Aug') # Exclude the augmented images\n",
    "except : \n",
    "    pass\n",
    "#Recurse to find total number of Images \n",
    "total=[]\n",
    "\n",
    "# First Nested Folder\n",
    "for i in a:\n",
    "    b = os.listdir(dir+i)\n",
    "    #Second Nested Folder\n",
    "    for j in b :\n",
    "        c = os.listdir(dir+i+'/'+j)\n",
    "        #Third Nested Folder\n",
    "        for k in c :\n",
    "            d = os.listdir(dir+i+'/'+j+'/'+k)\n",
    "            #Fourth Nested Folder ( Cyclone Name)\n",
    "            for l in d :\n",
    "                # 5th , 6th and 7th are always '/ir/geo/1km' , so we add them to the string.\n",
    "                e = os.listdir(dir+i+'/'+j+'/'+k+'/'+l+'/ir/geo/1km')\n",
    "                print(j+'-> '+l+' --> '+ str(len(e)))\n",
    "                # e contains the list of images of that cyclone\n",
    "                total = total + (e)\n",
    "print('')\n",
    "print('Total number of images present in the Dataset :',len(total)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So , now let us try to understand how the images look like. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['20040729.0645.goes-12.ir.x.99LINVEST.25kts-1009mb-250N-710W.jpg',\n",
       " '20040729.0702.goes-12.ir.x.99LINVEST.25kts-1009mb-250N-710W.jpg',\n",
       " '20040729.0715.goes-12.ir.x.99LINVEST.25kts-1009mb-250N-710W.jpg',\n",
       " '20040729.0732.goes-12.ir.x.99LINVEST.25kts-1009mb-250N-710W.jpg',\n",
       " '20040729.0745.goes-12.ir.x.99LINVEST.25kts-1009mb-250N-710W.jpg']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total.sort()\n",
    "total[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By a quick glance, we can understand the the Image Name has much information stored in it. \n",
    "\n",
    "Now, let us understand the format of the Data stored : \n",
    "\n",
    "*YYYYMMDD.HHMM*.Name of Satellite that captured the image and other relevant Data.jpg\n",
    "\n",
    "YYYYMMDD.HHMM  -> Year Month Date. Hours Minutes \n",
    "\n",
    "*Now we will be using the date time provided in the image name to annotate the type of category using the text data*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with the Text Data \n",
    "\n",
    "We will be using the Best Track Data (HURDAT2) in the Atlantic Region for the Tropical Cyclone. From its [description](http://www.nhc.noaa.gov/data/#hurdat) on the NOAA's data web page:\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p class=\"reg\"><span style=\"font-weight:bold;\">Atlantic hurricane database (HURDAT2) 1851-2018</span> <br>\n",
    "This dataset was provided on 10 May 2019 to include the 2018 update to the best tracks.\n",
    "</p>\n",
    "\n",
    "<p class=\"reg\">\n",
    "This dataset (<a href=\"/data/hurdat/hurdat2-format-atlantic.pdf\">known as Atlantic HURDAT2</a>) has\n",
    "a comma-delimited, text format with six-hourly information on the location,\n",
    "maximum winds, central pressure, and (beginning in 2004) size of all known tropical cyclones and subtropical cyclones.\n",
    "The original HURDAT database has been retired.</p>\n",
    "\n",
    "**( Optional - [Pre-processing the Data](Pre-Processing_Text_Data.ipynb))**\n",
    "\n",
    "This data follows a Modified CSV format because of which Pandas ( Python Library for Data Manipulation and Analysis ) can not get any useable data directly. Hence, we will be building our parser to pre-process this data and make it usable in the upcoming tasks.\n",
    "\n",
    "The steps followed in Notebook [Pre-Processing Text Data](Pre-Processing_Text_Data.ipynb) are :\n",
    "\n",
    "- Understanding the format of the data \n",
    "- Storing the cyclone in a dictionary\n",
    "- Converting the dictionary to a Dataframe\n",
    "- Restructuring the columns and making it readable\n",
    "- Replacing sentinel values and removing empty strings\n",
    "- Removing unwanted spaces and reindexing the Data frame\n",
    "- Save this Dataframe to a CSV File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let us have a look at the text data :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "      <th>date</th>\n",
       "      <th>hours_minutes</th>\n",
       "      <th>record_identifier</th>\n",
       "      <th>status_of_system</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>maximum_sustained_wind_knots</th>\n",
       "      <th>...</th>\n",
       "      <th>34_kt_sw</th>\n",
       "      <th>34_kt_nw</th>\n",
       "      <th>50_kt_ne</th>\n",
       "      <th>50_kt_se</th>\n",
       "      <th>50_kt_sw</th>\n",
       "      <th>50_kt_nw</th>\n",
       "      <th>64_kt_ne</th>\n",
       "      <th>64_kt_se</th>\n",
       "      <th>64_kt_sw</th>\n",
       "      <th>64_kt_nw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>51300</th>\n",
       "      <td>51300</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>20181010</td>\n",
       "      <td>1200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>12.3N</td>\n",
       "      <td>31.4W</td>\n",
       "      <td>55</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51301</th>\n",
       "      <td>51301</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>20181010</td>\n",
       "      <td>1800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>12.8N</td>\n",
       "      <td>32.0W</td>\n",
       "      <td>50</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51302</th>\n",
       "      <td>51302</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>20181011</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>13.3N</td>\n",
       "      <td>32.5W</td>\n",
       "      <td>50</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51303</th>\n",
       "      <td>51303</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>20181011</td>\n",
       "      <td>600</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>13.6N</td>\n",
       "      <td>33.1W</td>\n",
       "      <td>45</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51304</th>\n",
       "      <td>51304</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>20181011</td>\n",
       "      <td>1200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>13.9N</td>\n",
       "      <td>33.8W</td>\n",
       "      <td>45</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51305</th>\n",
       "      <td>51305</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>20181011</td>\n",
       "      <td>1800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>14.3N</td>\n",
       "      <td>34.6W</td>\n",
       "      <td>45</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51306</th>\n",
       "      <td>51306</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>20181012</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>14.8N</td>\n",
       "      <td>35.0W</td>\n",
       "      <td>40</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51307</th>\n",
       "      <td>51307</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>20181012</td>\n",
       "      <td>600</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>15.3N</td>\n",
       "      <td>35.3W</td>\n",
       "      <td>40</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51308</th>\n",
       "      <td>51308</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>20181012</td>\n",
       "      <td>1200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>15.8N</td>\n",
       "      <td>35.7W</td>\n",
       "      <td>35</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51309</th>\n",
       "      <td>51309</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>20181012</td>\n",
       "      <td>1800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TD</td>\n",
       "      <td>16.2N</td>\n",
       "      <td>37.0W</td>\n",
       "      <td>30</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       index        id    name      date  hours_minutes record_identifier  \\\n",
       "51300  51300  AL152018  NADINE  20181010           1200               NaN   \n",
       "51301  51301  AL152018  NADINE  20181010           1800               NaN   \n",
       "51302  51302  AL152018  NADINE  20181011              0               NaN   \n",
       "51303  51303  AL152018  NADINE  20181011            600               NaN   \n",
       "51304  51304  AL152018  NADINE  20181011           1200               NaN   \n",
       "51305  51305  AL152018  NADINE  20181011           1800               NaN   \n",
       "51306  51306  AL152018  NADINE  20181012              0               NaN   \n",
       "51307  51307  AL152018  NADINE  20181012            600               NaN   \n",
       "51308  51308  AL152018  NADINE  20181012           1200               NaN   \n",
       "51309  51309  AL152018  NADINE  20181012           1800               NaN   \n",
       "\n",
       "      status_of_system latitude longitude  maximum_sustained_wind_knots  ...  \\\n",
       "51300               TS    12.3N     31.4W                            55  ...   \n",
       "51301               TS    12.8N     32.0W                            50  ...   \n",
       "51302               TS    13.3N     32.5W                            50  ...   \n",
       "51303               TS    13.6N     33.1W                            45  ...   \n",
       "51304               TS    13.9N     33.8W                            45  ...   \n",
       "51305               TS    14.3N     34.6W                            45  ...   \n",
       "51306               TS    14.8N     35.0W                            40  ...   \n",
       "51307               TS    15.3N     35.3W                            40  ...   \n",
       "51308               TS    15.8N     35.7W                            35  ...   \n",
       "51309               TD    16.2N     37.0W                            30  ...   \n",
       "\n",
       "       34_kt_sw  34_kt_nw  50_kt_ne  50_kt_se  50_kt_sw  50_kt_nw  64_kt_ne  \\\n",
       "51300      30.0      60.0      30.0      20.0      20.0      30.0       0.0   \n",
       "51301      30.0      60.0      30.0      20.0       0.0      20.0       0.0   \n",
       "51302      30.0      60.0      30.0       0.0       0.0      30.0       0.0   \n",
       "51303      30.0      60.0       0.0       0.0       0.0       0.0       0.0   \n",
       "51304      30.0      50.0       0.0       0.0       0.0       0.0       0.0   \n",
       "51305      30.0      50.0       0.0       0.0       0.0       0.0       0.0   \n",
       "51306       0.0      50.0       0.0       0.0       0.0       0.0       0.0   \n",
       "51307       0.0      50.0       0.0       0.0       0.0       0.0       0.0   \n",
       "51308       0.0      50.0       0.0       0.0       0.0       0.0       0.0   \n",
       "51309       0.0       0.0       0.0       0.0       0.0       0.0       0.0   \n",
       "\n",
       "       64_kt_se  64_kt_sw  64_kt_nw  \n",
       "51300       0.0       0.0       0.0  \n",
       "51301       0.0       0.0       0.0  \n",
       "51302       0.0       0.0       0.0  \n",
       "51303       0.0       0.0       0.0  \n",
       "51304       0.0       0.0       0.0  \n",
       "51305       0.0       0.0       0.0  \n",
       "51306       0.0       0.0       0.0  \n",
       "51307       0.0       0.0       0.0  \n",
       "51308       0.0       0.0       0.0  \n",
       "51309       0.0       0.0       0.0  \n",
       "\n",
       "[10 rows x 23 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "atlantic_storms= pd.read_csv('atlantic.csv')\n",
    "atlantic_storms.tail(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "      <th>date</th>\n",
       "      <th>record_identifier</th>\n",
       "      <th>status_of_system</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>maximum_sustained_wind_knots</th>\n",
       "      <th>maximum_pressure</th>\n",
       "      <th>...</th>\n",
       "      <th>34_kt_sw</th>\n",
       "      <th>34_kt_nw</th>\n",
       "      <th>50_kt_ne</th>\n",
       "      <th>50_kt_se</th>\n",
       "      <th>50_kt_sw</th>\n",
       "      <th>50_kt_nw</th>\n",
       "      <th>64_kt_ne</th>\n",
       "      <th>64_kt_se</th>\n",
       "      <th>64_kt_sw</th>\n",
       "      <th>64_kt_nw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>51305</th>\n",
       "      <td>51305</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>2018-10-11 18:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>14.3N</td>\n",
       "      <td>34.6W</td>\n",
       "      <td>45</td>\n",
       "      <td>1002.0</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51306</th>\n",
       "      <td>51306</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>2018-10-12 00:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>14.8N</td>\n",
       "      <td>35.0W</td>\n",
       "      <td>40</td>\n",
       "      <td>1003.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51307</th>\n",
       "      <td>51307</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>2018-10-12 06:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>15.3N</td>\n",
       "      <td>35.3W</td>\n",
       "      <td>40</td>\n",
       "      <td>1004.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51308</th>\n",
       "      <td>51308</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>2018-10-12 12:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TS</td>\n",
       "      <td>15.8N</td>\n",
       "      <td>35.7W</td>\n",
       "      <td>35</td>\n",
       "      <td>1006.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51309</th>\n",
       "      <td>51309</td>\n",
       "      <td>AL152018</td>\n",
       "      <td>NADINE</td>\n",
       "      <td>2018-10-12 18:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TD</td>\n",
       "      <td>16.2N</td>\n",
       "      <td>37.0W</td>\n",
       "      <td>30</td>\n",
       "      <td>1008.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       index        id    name                date record_identifier  \\\n",
       "51305  51305  AL152018  NADINE 2018-10-11 18:00:00               NaN   \n",
       "51306  51306  AL152018  NADINE 2018-10-12 00:00:00               NaN   \n",
       "51307  51307  AL152018  NADINE 2018-10-12 06:00:00               NaN   \n",
       "51308  51308  AL152018  NADINE 2018-10-12 12:00:00               NaN   \n",
       "51309  51309  AL152018  NADINE 2018-10-12 18:00:00               NaN   \n",
       "\n",
       "      status_of_system latitude longitude  maximum_sustained_wind_knots  \\\n",
       "51305               TS    14.3N     34.6W                            45   \n",
       "51306               TS    14.8N     35.0W                            40   \n",
       "51307               TS    15.3N     35.3W                            40   \n",
       "51308               TS    15.8N     35.7W                            35   \n",
       "51309               TD    16.2N     37.0W                            30   \n",
       "\n",
       "       maximum_pressure  ...  34_kt_sw  34_kt_nw  50_kt_ne  50_kt_se  \\\n",
       "51305            1002.0  ...      30.0      50.0       0.0       0.0   \n",
       "51306            1003.0  ...       0.0      50.0       0.0       0.0   \n",
       "51307            1004.0  ...       0.0      50.0       0.0       0.0   \n",
       "51308            1006.0  ...       0.0      50.0       0.0       0.0   \n",
       "51309            1008.0  ...       0.0       0.0       0.0       0.0   \n",
       "\n",
       "       50_kt_sw  50_kt_nw  64_kt_ne  64_kt_se  64_kt_sw  64_kt_nw  \n",
       "51305       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "51306       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "51307       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "51308       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "51309       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "\n",
       "[5 rows x 22 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We'll be using the Date and Time for Finding out the Velocity of the Cyclone \n",
    "# So we'll make it into a readable format\n",
    "\n",
    "atlantic_storms['date'] = pd.to_datetime(atlantic_storms['date'].astype(str))\n",
    "atlantic_storms['date'] = atlantic_storms.apply(lambda srs: srs['date'].replace(hour=int((\"%04d\" % srs['hours_minutes'])[:2]), minute=int((\"%04d\" % srs['hours_minutes'])[2:])), axis='columns')\n",
    "del atlantic_storms['hours_minutes']\n",
    "atlantic_storms.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's now save our Text Data to Use it in the Upcoming Notebooks\n",
    "atlantic_storms.to_csv(\"atlantic_storms.csv\", encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alternate Approach : \n",
    "\n",
    "Above we have seen the flow in which we approach the problem, we can also solve the same problem using an alternate approach : \n",
    "\n",
    "![alt_text](images/alt.png)\n",
    " \n",
    "But, the later approach is not advised and the reason behind the same lies in the data we are working with : \n",
    "\n",
    "Let us take the output of the cell containing the names of the images: \n",
    "\n",
    "```\n",
    "['20040729.0645.goes-12.ir.x.99LINVEST.25kts-1009mb-250N-710W.jpg',\n",
    " '20040729.0702.goes-12.ir.x.99LINVEST.25kts-1009mb-250N-710W.jpg',\n",
    " '20040729.0715.goes-12.ir.x.99LINVEST.25kts-1009mb-250N-710W.jpg',\n",
    " '20040729.0732.goes-12.ir.x.99LINVEST.25kts-1009mb-250N-710W.jpg',\n",
    " '20040729.0745.goes-12.ir.x.99LINVEST.25kts-1009mb-250N-710W.jpg']\n",
    " ```\n",
    "\n",
    "We can see the time of the images above , it is taken approximately every 15 minutes.\n",
    "\n",
    "Let's see the time interval of our text dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51305   2018-10-11 18:00:00\n",
       "51306   2018-10-12 00:00:00\n",
       "51307   2018-10-12 06:00:00\n",
       "51308   2018-10-12 12:00:00\n",
       "51309   2018-10-12 18:00:00\n",
       "Name: date, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "atlantic_storms.tail()['date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we notice the text data to be sampled once every six hours because of which we will be forced to use interpolation techniques to find the velocity at any particular instant. \n",
    "\n",
    "Now determining the velocity at any time instance with this interpolated data is going to be deviated from the truth value, but we know that a class has a range of velocity so the probability that our interpolated class being correct is more realstic as compared to the former.\n",
    "\n",
    "## Licensing\n",
    "This material is released by OpenACC-Standard.org, in collaboration with NVIDIA Corporation, under the Creative Commons Attribution 4.0 International (CC BY 4.0)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Previous Notebook](The_Problem_Statement.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "[1](The_Problem_Statement.ipynb)\n",
    "[2]\n",
    "[3](Manipulation_of_Image_Data_and_Category_Determination_using_Text_Data.ipynb)\n",
    "[4](Countering_Data_Imbalance.ipynb)\n",
    "[5](Competition.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "[Next Notebook](Manipulation_of_Image_Data_and_Category_Determination_using_Text_Data.ipynb)\n",
    "\n",
    "\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&ensp;\n",
    "[Home Page](../Start_Here.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}