14_recurrent_neural_networks_ex_8.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ch 14. Excercise 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, Luis already put his solution to github(https://github.com/luisandresilva/Kaggle/tree/master/Rain-II). He used Keras API and python 2. Here, we will reproduce his model for latest tf.keras API and python 3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, download data of \"How Much Did It Rain? II\" competition and put it on `datasets/rain2` folder. Read the `train.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "rain2 = pd.read_csv('datasets/rain2/train.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's view some rows via `head()` method of pandas's `DataFrame`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>minutes_past</th>\n",
       "      <th>radardist_km</th>\n",
       "      <th>Ref</th>\n",
       "      <th>Ref_5x5_10th</th>\n",
       "      <th>Ref_5x5_50th</th>\n",
       "      <th>Ref_5x5_90th</th>\n",
       "      <th>RefComposite</th>\n",
       "      <th>RefComposite_5x5_10th</th>\n",
       "      <th>RefComposite_5x5_50th</th>\n",
       "      <th>...</th>\n",
       "      <th>RhoHV_5x5_90th</th>\n",
       "      <th>Zdr</th>\n",
       "      <th>Zdr_5x5_10th</th>\n",
       "      <th>Zdr_5x5_50th</th>\n",
       "      <th>Zdr_5x5_90th</th>\n",
       "      <th>Kdp</th>\n",
       "      <th>Kdp_5x5_10th</th>\n",
       "      <th>Kdp_5x5_50th</th>\n",
       "      <th>Kdp_5x5_90th</th>\n",
       "      <th>Expected</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>45</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>55</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>10.5</td>\n",
       "      <td>15.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>16.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>-0.1250</td>\n",
       "      <td>0.3125</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>1.059998</td>\n",
       "      <td>-1.410004</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>1.059998</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>26.5</td>\n",
       "      <td>22.5</td>\n",
       "      <td>25.5</td>\n",
       "      <td>31.5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>28.5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.005000</td>\n",
       "      <td>0.0625</td>\n",
       "      <td>-0.1875</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>0.6875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.409988</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>2.0</td>\n",
       "      <td>21.5</td>\n",
       "      <td>15.5</td>\n",
       "      <td>20.5</td>\n",
       "      <td>25.0</td>\n",
       "      <td>26.5</td>\n",
       "      <td>23.5</td>\n",
       "      <td>25.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.001667</td>\n",
       "      <td>0.3125</td>\n",
       "      <td>-0.0625</td>\n",
       "      <td>0.3125</td>\n",
       "      <td>0.6250</td>\n",
       "      <td>0.349991</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>1.759994</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2</td>\n",
       "      <td>16</td>\n",
       "      <td>2.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>17.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>20.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>20.5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.001667</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>0.1250</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>0.6875</td>\n",
       "      <td>0.349991</td>\n",
       "      <td>-1.059998</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.059998</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2</td>\n",
       "      <td>21</td>\n",
       "      <td>2.0</td>\n",
       "      <td>24.5</td>\n",
       "      <td>16.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>24.5</td>\n",
       "      <td>24.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>0.0625</td>\n",
       "      <td>0.1875</td>\n",
       "      <td>0.5625</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>-1.059998</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>1.759994</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2</td>\n",
       "      <td>26</td>\n",
       "      <td>2.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>16.5</td>\n",
       "      <td>17.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.5625</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>0.4375</td>\n",
       "      <td>0.6875</td>\n",
       "      <td>-1.760010</td>\n",
       "      <td>-1.760010</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>0.709991</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2</td>\n",
       "      <td>31</td>\n",
       "      <td>2.0</td>\n",
       "      <td>22.5</td>\n",
       "      <td>19.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>23.5</td>\n",
       "      <td>25.5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.001667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.1875</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>0.6250</td>\n",
       "      <td>-1.059998</td>\n",
       "      <td>-2.120010</td>\n",
       "      <td>-0.710007</td>\n",
       "      <td>0.349991</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2</td>\n",
       "      <td>37</td>\n",
       "      <td>2.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>18.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>19.5</td>\n",
       "      <td>20.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.1875</td>\n",
       "      <td>0.4375</td>\n",
       "      <td>0.8125</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.760010</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>1.059998</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2</td>\n",
       "      <td>42</td>\n",
       "      <td>2.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.5</td>\n",
       "      <td>17.0</td>\n",
       "      <td>19.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.6250</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>0.6250</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.349991</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2</td>\n",
       "      <td>47</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>3.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>16.5</td>\n",
       "      <td>18.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>0.1875</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.6875</td>\n",
       "      <td>0.349991</td>\n",
       "      <td>-2.110001</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>1.059998</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>14.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>23.5</td>\n",
       "      <td>28.0</td>\n",
       "      <td>23.5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>0.6250</td>\n",
       "      <td>0.9375</td>\n",
       "      <td>1.3750</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>-1.410004</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>2.119995</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>2</td>\n",
       "      <td>58</td>\n",
       "      <td>2.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>16.5</td>\n",
       "      <td>22.5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>31.5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>29.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.001667</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>0.1875</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>-1.410004</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>0.699997</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>3</td>\n",
       "      <td>14</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>3</td>\n",
       "      <td>18</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>3</td>\n",
       "      <td>23</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>3</td>\n",
       "      <td>28</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>3</td>\n",
       "      <td>33</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>3</td>\n",
       "      <td>38</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>3</td>\n",
       "      <td>43</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>3</td>\n",
       "      <td>48</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>3</td>\n",
       "      <td>53</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0.801667</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0625</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>3</td>\n",
       "      <td>58</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>30 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Id  minutes_past  radardist_km   Ref  Ref_5x5_10th  Ref_5x5_50th  \\\n",
       "0    1             3          10.0   NaN           NaN           NaN   \n",
       "1    1            16          10.0   NaN           NaN           NaN   \n",
       "2    1            25          10.0   NaN           NaN           NaN   \n",
       "3    1            35          10.0   NaN           NaN           NaN   \n",
       "4    1            45          10.0   NaN           NaN           NaN   \n",
       "5    1            55          10.0   NaN           NaN           NaN   \n",
       "6    2             1           2.0   9.0           5.0           7.5   \n",
       "7    2             6           2.0  26.5          22.5          25.5   \n",
       "8    2            11           2.0  21.5          15.5          20.5   \n",
       "9    2            16           2.0  18.0          14.0          17.5   \n",
       "10   2            21           2.0  24.5          16.5          21.0   \n",
       "11   2            26           2.0  12.0          12.0          16.0   \n",
       "12   2            31           2.0  22.5          19.0          22.0   \n",
       "13   2            37           2.0  14.0          14.0          18.5   \n",
       "14   2            42           2.0  12.0          11.0          12.5   \n",
       "15   2            47           2.0   1.5           3.5           7.0   \n",
       "16   2            53           2.0  16.0          14.5          18.0   \n",
       "17   2            58           2.0  22.0          16.5          22.5   \n",
       "18   3             4          10.0   NaN           NaN           NaN   \n",
       "19   3             9          10.0   NaN           NaN           NaN   \n",
       "20   3            14          10.0   NaN           NaN           NaN   \n",
       "21   3            18          10.0   NaN           NaN           NaN   \n",
       "22   3            23          10.0   NaN           NaN           NaN   \n",
       "23   3            28          10.0   NaN           NaN           NaN   \n",
       "24   3            33          10.0   NaN           NaN           NaN   \n",
       "25   3            38          10.0   NaN           NaN           NaN   \n",
       "26   3            43          10.0   NaN           NaN           NaN   \n",
       "27   3            48          10.0   NaN           NaN           NaN   \n",
       "28   3            53          10.0   NaN           NaN           NaN   \n",
       "29   3            58          10.0   NaN           NaN           NaN   \n",
       "\n",
       "    Ref_5x5_90th  RefComposite  RefComposite_5x5_10th  RefComposite_5x5_50th  \\\n",
       "0            NaN           NaN                    NaN                    NaN   \n",
       "1            NaN           NaN                    NaN                    NaN   \n",
       "2            NaN           NaN                    NaN                    NaN   \n",
       "3            NaN           NaN                    NaN                    NaN   \n",
       "4            NaN           NaN                    NaN                    NaN   \n",
       "5            NaN           NaN                    NaN                    NaN   \n",
       "6           10.5          15.0                   10.5                   16.5   \n",
       "7           31.5          26.5                   26.5                   28.5   \n",
       "8           25.0          26.5                   23.5                   25.0   \n",
       "9           21.0          20.5                   18.0                   20.5   \n",
       "10          24.5          24.5                   21.0                   24.0   \n",
       "11          20.0          16.5                   17.0                   19.0   \n",
       "12          25.0          26.0                   23.5                   25.5   \n",
       "13          21.0          19.5                   20.0                   21.0   \n",
       "14          17.0          19.5                   18.0                   21.0   \n",
       "15          10.5          18.0                   16.5                   18.5   \n",
       "16          23.5          28.0                   23.5                   26.5   \n",
       "17          26.5          31.5                   26.5                   29.0   \n",
       "18           NaN           NaN                    NaN                    NaN   \n",
       "19           NaN           NaN                    NaN                    NaN   \n",
       "20           NaN           NaN                    NaN                    NaN   \n",
       "21           NaN           NaN                    NaN                    NaN   \n",
       "22           NaN           NaN                    NaN                    NaN   \n",
       "23           NaN           NaN                    NaN                    NaN   \n",
       "24           NaN           NaN                    NaN                    NaN   \n",
       "25           NaN           NaN                    NaN                    NaN   \n",
       "26           8.5           NaN                    NaN                    NaN   \n",
       "27           NaN           NaN                    NaN                    NaN   \n",
       "28           NaN           NaN                    NaN                    NaN   \n",
       "29           NaN           NaN                    NaN                    NaN   \n",
       "\n",
       "      ...      RhoHV_5x5_90th     Zdr  Zdr_5x5_10th  Zdr_5x5_50th  \\\n",
       "0     ...                 NaN     NaN           NaN           NaN   \n",
       "1     ...                 NaN     NaN           NaN           NaN   \n",
       "2     ...                 NaN     NaN           NaN           NaN   \n",
       "3     ...                 NaN     NaN           NaN           NaN   \n",
       "4     ...                 NaN     NaN           NaN           NaN   \n",
       "5     ...                 NaN     NaN           NaN           NaN   \n",
       "6     ...            0.998333  0.3750       -0.1250        0.3125   \n",
       "7     ...            1.005000  0.0625       -0.1875        0.2500   \n",
       "8     ...            1.001667  0.3125       -0.0625        0.3125   \n",
       "9     ...            1.001667  0.2500        0.1250        0.3750   \n",
       "10    ...            0.998333  0.2500        0.0625        0.1875   \n",
       "11    ...            0.998333  0.5625        0.2500        0.4375   \n",
       "12    ...            1.001667  0.0000       -0.1875        0.2500   \n",
       "13    ...            0.998333  0.5000        0.1875        0.4375   \n",
       "14    ...            0.998333  0.6250        0.3750        0.6250   \n",
       "15    ...            0.998333  0.3750        0.1875        0.5000   \n",
       "16    ...            0.998333  0.8750        0.6250        0.9375   \n",
       "17    ...            1.001667  0.3750        0.1875        0.3750   \n",
       "18    ...                 NaN     NaN           NaN           NaN   \n",
       "19    ...                 NaN     NaN           NaN           NaN   \n",
       "20    ...                 NaN     NaN           NaN           NaN   \n",
       "21    ...                 NaN     NaN           NaN           NaN   \n",
       "22    ...                 NaN     NaN           NaN           NaN   \n",
       "23    ...                 NaN     NaN           NaN           NaN   \n",
       "24    ...                 NaN     NaN           NaN           NaN   \n",
       "25    ...                 NaN     NaN           NaN           NaN   \n",
       "26    ...                 NaN     NaN           NaN           NaN   \n",
       "27    ...                 NaN     NaN           NaN           NaN   \n",
       "28    ...            0.801667     NaN           NaN           NaN   \n",
       "29    ...                 NaN     NaN           NaN           NaN   \n",
       "\n",
       "    Zdr_5x5_90th       Kdp  Kdp_5x5_10th  Kdp_5x5_50th  Kdp_5x5_90th  \\\n",
       "0            NaN       NaN           NaN           NaN           NaN   \n",
       "1            NaN       NaN           NaN           NaN           NaN   \n",
       "2            NaN       NaN           NaN           NaN           NaN   \n",
       "3            NaN       NaN           NaN           NaN           NaN   \n",
       "4            NaN       NaN           NaN           NaN           NaN   \n",
       "5            NaN       NaN           NaN           NaN           NaN   \n",
       "6         0.8750  1.059998     -1.410004     -0.350006      1.059998   \n",
       "7         0.6875       NaN           NaN           NaN      1.409988   \n",
       "8         0.6250  0.349991           NaN     -0.350006      1.759994   \n",
       "9         0.6875  0.349991     -1.059998      0.000000      1.059998   \n",
       "10        0.5625 -0.350006     -1.059998     -0.350006      1.759994   \n",
       "11        0.6875 -1.760010     -1.760010     -0.350006      0.709991   \n",
       "12        0.6250 -1.059998     -2.120010     -0.710007      0.349991   \n",
       "13        0.8125  0.000000     -1.760010     -0.350006      1.059998   \n",
       "14        0.8750 -0.350006     -0.350006      0.000000      0.349991   \n",
       "15        0.6875  0.349991     -2.110001     -0.350006      1.059998   \n",
       "16        1.3750 -0.350006     -1.410004     -0.350006      2.119995   \n",
       "17        0.8750 -1.410004           NaN     -0.350006      0.699997   \n",
       "18           NaN       NaN           NaN           NaN           NaN   \n",
       "19           NaN       NaN           NaN           NaN           NaN   \n",
       "20           NaN       NaN           NaN           NaN           NaN   \n",
       "21           NaN       NaN           NaN           NaN           NaN   \n",
       "22           NaN       NaN           NaN           NaN           NaN   \n",
       "23           NaN       NaN           NaN           NaN           NaN   \n",
       "24           NaN       NaN           NaN           NaN           NaN   \n",
       "25           NaN       NaN           NaN           NaN           NaN   \n",
       "26           NaN       NaN           NaN           NaN           NaN   \n",
       "27           NaN       NaN           NaN           NaN           NaN   \n",
       "28        2.0625       NaN           NaN           NaN           NaN   \n",
       "29           NaN       NaN           NaN           NaN           NaN   \n",
       "\n",
       "     Expected  \n",
       "0    0.254000  \n",
       "1    0.254000  \n",
       "2    0.254000  \n",
       "3    0.254000  \n",
       "4    0.254000  \n",
       "5    0.254000  \n",
       "6    1.016000  \n",
       "7    1.016000  \n",
       "8    1.016000  \n",
       "9    1.016000  \n",
       "10   1.016000  \n",
       "11   1.016000  \n",
       "12   1.016000  \n",
       "13   1.016000  \n",
       "14   1.016000  \n",
       "15   1.016000  \n",
       "16   1.016000  \n",
       "17   1.016000  \n",
       "18  26.162014  \n",
       "19  26.162014  \n",
       "20  26.162014  \n",
       "21  26.162014  \n",
       "22  26.162014  \n",
       "23  26.162014  \n",
       "24  26.162014  \n",
       "25  26.162014  \n",
       "26  26.162014  \n",
       "27  26.162014  \n",
       "28  26.162014  \n",
       "29  26.162014  \n",
       "\n",
       "[30 rows x 24 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rain2.head(n=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 24 columns. First column `Id` is unique value representing a sequence. Last column `Expected` is target. We simply remove rows that `Ref` column is `NaN`. If `Ref` column is `NaN`, all other columns is `NaN` too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = rain2.dropna(subset=['Ref'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `describe()` method, we get the statistics for dataset. The features excluding `Id` and `Expected` has different scale. But the order of magnitude is not significant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>minutes_past</th>\n",
       "      <th>radardist_km</th>\n",
       "      <th>Ref</th>\n",
       "      <th>Ref_5x5_10th</th>\n",
       "      <th>Ref_5x5_50th</th>\n",
       "      <th>Ref_5x5_90th</th>\n",
       "      <th>RefComposite</th>\n",
       "      <th>RefComposite_5x5_10th</th>\n",
       "      <th>RefComposite_5x5_50th</th>\n",
       "      <th>...</th>\n",
       "      <th>RhoHV_5x5_90th</th>\n",
       "      <th>Zdr</th>\n",
       "      <th>Zdr_5x5_10th</th>\n",
       "      <th>Zdr_5x5_50th</th>\n",
       "      <th>Zdr_5x5_90th</th>\n",
       "      <th>Kdp</th>\n",
       "      <th>Kdp_5x5_10th</th>\n",
       "      <th>Kdp_5x5_50th</th>\n",
       "      <th>Kdp_5x5_90th</th>\n",
       "      <th>Expected</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>6.349375e+06</td>\n",
       "      <td>6.349375e+06</td>\n",
       "      <td>6.349375e+06</td>\n",
       "      <td>6.349375e+06</td>\n",
       "      <td>5.269105e+06</td>\n",
       "      <td>6.156846e+06</td>\n",
       "      <td>6.349288e+06</td>\n",
       "      <td>6.349375e+06</td>\n",
       "      <td>5.550921e+06</td>\n",
       "      <td>6.207490e+06</td>\n",
       "      <td>...</td>\n",
       "      <td>4.600898e+06</td>\n",
       "      <td>4.069799e+06</td>\n",
       "      <td>3.577674e+06</td>\n",
       "      <td>4.095586e+06</td>\n",
       "      <td>4.600898e+06</td>\n",
       "      <td>3.494665e+06</td>\n",
       "      <td>2.978691e+06</td>\n",
       "      <td>3.511354e+06</td>\n",
       "      <td>4.062702e+06</td>\n",
       "      <td>6.349375e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.894813e+05</td>\n",
       "      <td>2.927051e+01</td>\n",
       "      <td>9.286182e+00</td>\n",
       "      <td>2.292666e+01</td>\n",
       "      <td>1.998385e+01</td>\n",
       "      <td>2.302103e+01</td>\n",
       "      <td>2.812518e+01</td>\n",
       "      <td>2.540071e+01</td>\n",
       "      <td>2.256900e+01</td>\n",
       "      <td>2.541082e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>1.013113e+00</td>\n",
       "      <td>5.295568e-01</td>\n",
       "      <td>-6.975932e-01</td>\n",
       "      <td>3.815768e-01</td>\n",
       "      <td>2.057199e+00</td>\n",
       "      <td>3.230033e-02</td>\n",
       "      <td>-3.409001e+00</td>\n",
       "      <td>-3.781228e-01</td>\n",
       "      <td>4.038652e+00</td>\n",
       "      <td>1.714876e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.406124e+05</td>\n",
       "      <td>1.715383e+01</td>\n",
       "      <td>4.068784e+00</td>\n",
       "      <td>1.035516e+01</td>\n",
       "      <td>9.195141e+00</td>\n",
       "      <td>9.882618e+00</td>\n",
       "      <td>1.028228e+01</td>\n",
       "      <td>1.044013e+01</td>\n",
       "      <td>9.547383e+00</td>\n",
       "      <td>1.003713e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>4.179456e-02</td>\n",
       "      <td>1.476643e+00</td>\n",
       "      <td>1.017368e+00</td>\n",
       "      <td>9.260191e-01</td>\n",
       "      <td>1.617217e+00</td>\n",
       "      <td>3.699795e+00</td>\n",
       "      <td>2.719573e+00</td>\n",
       "      <td>2.087361e+00</td>\n",
       "      <td>3.902270e+00</td>\n",
       "      <td>2.025279e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>-3.100000e+01</td>\n",
       "      <td>-3.200000e+01</td>\n",
       "      <td>-3.200000e+01</td>\n",
       "      <td>-2.850000e+01</td>\n",
       "      <td>-2.800000e+01</td>\n",
       "      <td>-3.050000e+01</td>\n",
       "      <td>-2.750000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>2.083333e-01</td>\n",
       "      <td>-7.875000e+00</td>\n",
       "      <td>-7.875000e+00</td>\n",
       "      <td>-7.875000e+00</td>\n",
       "      <td>-7.875000e+00</td>\n",
       "      <td>-9.604000e+01</td>\n",
       "      <td>-8.079000e+01</td>\n",
       "      <td>-7.168000e+01</td>\n",
       "      <td>-1.002000e+02</td>\n",
       "      <td>1.000000e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.923290e+05</td>\n",
       "      <td>1.400000e+01</td>\n",
       "      <td>6.000000e+00</td>\n",
       "      <td>1.600000e+01</td>\n",
       "      <td>1.450000e+01</td>\n",
       "      <td>1.650000e+01</td>\n",
       "      <td>2.150000e+01</td>\n",
       "      <td>1.850000e+01</td>\n",
       "      <td>1.650000e+01</td>\n",
       "      <td>1.850000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>9.983333e-01</td>\n",
       "      <td>-1.875000e-01</td>\n",
       "      <td>-1.062500e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.062500e+00</td>\n",
       "      <td>-1.410004e+00</td>\n",
       "      <td>-4.540008e+00</td>\n",
       "      <td>-7.100067e-01</td>\n",
       "      <td>2.059998e+00</td>\n",
       "      <td>5.080003e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.906240e+05</td>\n",
       "      <td>2.900000e+01</td>\n",
       "      <td>1.000000e+01</td>\n",
       "      <td>2.250000e+01</td>\n",
       "      <td>2.000000e+01</td>\n",
       "      <td>2.300000e+01</td>\n",
       "      <td>2.750000e+01</td>\n",
       "      <td>2.500000e+01</td>\n",
       "      <td>2.250000e+01</td>\n",
       "      <td>2.500000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>1.005000e+00</td>\n",
       "      <td>3.750000e-01</td>\n",
       "      <td>-6.250000e-01</td>\n",
       "      <td>3.125000e-01</td>\n",
       "      <td>1.687500e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>-2.820007e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>3.509994e+00</td>\n",
       "      <td>1.425001e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>8.883170e+05</td>\n",
       "      <td>4.400000e+01</td>\n",
       "      <td>1.200000e+01</td>\n",
       "      <td>2.950000e+01</td>\n",
       "      <td>2.600000e+01</td>\n",
       "      <td>2.950000e+01</td>\n",
       "      <td>3.500000e+01</td>\n",
       "      <td>3.200000e+01</td>\n",
       "      <td>2.900000e+01</td>\n",
       "      <td>3.200000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>1.051667e+00</td>\n",
       "      <td>1.062500e+00</td>\n",
       "      <td>-1.875000e-01</td>\n",
       "      <td>6.875000e-01</td>\n",
       "      <td>2.562500e+00</td>\n",
       "      <td>1.409988e+00</td>\n",
       "      <td>-1.740006e+00</td>\n",
       "      <td>3.499908e-01</td>\n",
       "      <td>5.629990e+00</td>\n",
       "      <td>4.064002e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.180945e+06</td>\n",
       "      <td>5.900000e+01</td>\n",
       "      <td>2.100000e+01</td>\n",
       "      <td>7.100000e+01</td>\n",
       "      <td>6.250000e+01</td>\n",
       "      <td>6.900000e+01</td>\n",
       "      <td>7.250000e+01</td>\n",
       "      <td>9.250000e+01</td>\n",
       "      <td>6.600000e+01</td>\n",
       "      <td>7.100000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>1.051667e+00</td>\n",
       "      <td>7.937500e+00</td>\n",
       "      <td>7.937500e+00</td>\n",
       "      <td>7.937500e+00</td>\n",
       "      <td>7.937500e+00</td>\n",
       "      <td>1.797500e+02</td>\n",
       "      <td>3.169998e+00</td>\n",
       "      <td>1.280000e+01</td>\n",
       "      <td>1.446000e+02</td>\n",
       "      <td>3.301773e+04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Id  minutes_past  radardist_km           Ref  Ref_5x5_10th  \\\n",
       "count  6.349375e+06  6.349375e+06  6.349375e+06  6.349375e+06  5.269105e+06   \n",
       "mean   5.894813e+05  2.927051e+01  9.286182e+00  2.292666e+01  1.998385e+01   \n",
       "std    3.406124e+05  1.715383e+01  4.068784e+00  1.035516e+01  9.195141e+00   \n",
       "min    2.000000e+00  0.000000e+00  0.000000e+00 -3.100000e+01 -3.200000e+01   \n",
       "25%    2.923290e+05  1.400000e+01  6.000000e+00  1.600000e+01  1.450000e+01   \n",
       "50%    5.906240e+05  2.900000e+01  1.000000e+01  2.250000e+01  2.000000e+01   \n",
       "75%    8.883170e+05  4.400000e+01  1.200000e+01  2.950000e+01  2.600000e+01   \n",
       "max    1.180945e+06  5.900000e+01  2.100000e+01  7.100000e+01  6.250000e+01   \n",
       "\n",
       "       Ref_5x5_50th  Ref_5x5_90th  RefComposite  RefComposite_5x5_10th  \\\n",
       "count  6.156846e+06  6.349288e+06  6.349375e+06           5.550921e+06   \n",
       "mean   2.302103e+01  2.812518e+01  2.540071e+01           2.256900e+01   \n",
       "std    9.882618e+00  1.028228e+01  1.044013e+01           9.547383e+00   \n",
       "min   -3.200000e+01 -2.850000e+01 -2.800000e+01          -3.050000e+01   \n",
       "25%    1.650000e+01  2.150000e+01  1.850000e+01           1.650000e+01   \n",
       "50%    2.300000e+01  2.750000e+01  2.500000e+01           2.250000e+01   \n",
       "75%    2.950000e+01  3.500000e+01  3.200000e+01           2.900000e+01   \n",
       "max    6.900000e+01  7.250000e+01  9.250000e+01           6.600000e+01   \n",
       "\n",
       "       RefComposite_5x5_50th      ...       RhoHV_5x5_90th           Zdr  \\\n",
       "count           6.207490e+06      ...         4.600898e+06  4.069799e+06   \n",
       "mean            2.541082e+01      ...         1.013113e+00  5.295568e-01   \n",
       "std             1.003713e+01      ...         4.179456e-02  1.476643e+00   \n",
       "min            -2.750000e+01      ...         2.083333e-01 -7.875000e+00   \n",
       "25%             1.850000e+01      ...         9.983333e-01 -1.875000e-01   \n",
       "50%             2.500000e+01      ...         1.005000e+00  3.750000e-01   \n",
       "75%             3.200000e+01      ...         1.051667e+00  1.062500e+00   \n",
       "max             7.100000e+01      ...         1.051667e+00  7.937500e+00   \n",
       "\n",
       "       Zdr_5x5_10th  Zdr_5x5_50th  Zdr_5x5_90th           Kdp  Kdp_5x5_10th  \\\n",
       "count  3.577674e+06  4.095586e+06  4.600898e+06  3.494665e+06  2.978691e+06   \n",
       "mean  -6.975932e-01  3.815768e-01  2.057199e+00  3.230033e-02 -3.409001e+00   \n",
       "std    1.017368e+00  9.260191e-01  1.617217e+00  3.699795e+00  2.719573e+00   \n",
       "min   -7.875000e+00 -7.875000e+00 -7.875000e+00 -9.604000e+01 -8.079000e+01   \n",
       "25%   -1.062500e+00  0.000000e+00  1.062500e+00 -1.410004e+00 -4.540008e+00   \n",
       "50%   -6.250000e-01  3.125000e-01  1.687500e+00  0.000000e+00 -2.820007e+00   \n",
       "75%   -1.875000e-01  6.875000e-01  2.562500e+00  1.409988e+00 -1.740006e+00   \n",
       "max    7.937500e+00  7.937500e+00  7.937500e+00  1.797500e+02  3.169998e+00   \n",
       "\n",
       "       Kdp_5x5_50th  Kdp_5x5_90th      Expected  \n",
       "count  3.511354e+06  4.062702e+06  6.349375e+06  \n",
       "mean  -3.781228e-01  4.038652e+00  1.714876e+01  \n",
       "std    2.087361e+00  3.902270e+00  2.025279e+02  \n",
       "min   -7.168000e+01 -1.002000e+02  1.000000e-02  \n",
       "25%   -7.100067e-01  2.059998e+00  5.080003e-01  \n",
       "50%    0.000000e+00  3.509994e+00  1.425001e+00  \n",
       "75%    3.499908e-01  5.629990e+00  4.064002e+00  \n",
       "max    1.280000e+01  1.446000e+02  3.301773e+04  \n",
       "\n",
       "[8 rows x 24 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We removed the rows that `Ref` column is `NaN`. But there are still `NaN` values. Let's check How many `NaN` are in the `train_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                             0\n",
       "minutes_past                   0\n",
       "radardist_km                   0\n",
       "Ref                            0\n",
       "Ref_5x5_10th             1080270\n",
       "Ref_5x5_50th              192529\n",
       "Ref_5x5_90th                  87\n",
       "RefComposite                   0\n",
       "RefComposite_5x5_10th     798454\n",
       "RefComposite_5x5_50th     141885\n",
       "RefComposite_5x5_90th         67\n",
       "RhoHV                    2279576\n",
       "RhoHV_5x5_10th           2771701\n",
       "RhoHV_5x5_50th           2253789\n",
       "RhoHV_5x5_90th           1748477\n",
       "Zdr                      2279576\n",
       "Zdr_5x5_10th             2771701\n",
       "Zdr_5x5_50th             2253789\n",
       "Zdr_5x5_90th             1748477\n",
       "Kdp                      2854710\n",
       "Kdp_5x5_10th             3370684\n",
       "Kdp_5x5_50th             2838021\n",
       "Kdp_5x5_90th             2286673\n",
       "Expected                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. For simplicity, just fill `NaN` cell with 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                       0\n",
       "minutes_past             0\n",
       "radardist_km             0\n",
       "Ref                      0\n",
       "Ref_5x5_10th             0\n",
       "Ref_5x5_50th             0\n",
       "Ref_5x5_90th             0\n",
       "RefComposite             0\n",
       "RefComposite_5x5_10th    0\n",
       "RefComposite_5x5_50th    0\n",
       "RefComposite_5x5_90th    0\n",
       "RhoHV                    0\n",
       "RhoHV_5x5_10th           0\n",
       "RhoHV_5x5_50th           0\n",
       "RhoHV_5x5_90th           0\n",
       "Zdr                      0\n",
       "Zdr_5x5_10th             0\n",
       "Zdr_5x5_50th             0\n",
       "Zdr_5x5_90th             0\n",
       "Kdp                      0\n",
       "Kdp_5x5_10th             0\n",
       "Kdp_5x5_50th             0\n",
       "Kdp_5x5_90th             0\n",
       "Expected                 0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df = train_df.fillna(0)\n",
    "train_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can group by `Id` column via `groupby` method. This is the sequence we feed into the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_seq = train_df.groupby(['Id'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's check the number of unique `Id` and maximum length of sequence. `size()` method return the length of each group. It's pandas's `Series` object that has `count()` and `max()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(731556, 19)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_seq_size = train_seq.size()\n",
    "train_seq_size.count(), train_seq_size.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we know number of sequence and max length of each sequence. So we can make numpy array `X` and `y`. `X` has 731,556 sequences. Each sequence has up to 19 rows. Each row has 22 features. `y` has just one column for `Expected`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.zeros((731556, 19, 22))\n",
    "y = np.zeros((731556, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luis applied some preprocessing and feature extraction. Here, we just try to apply raw dataset to LSTM without any preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "731556\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "for name, group in train_seq:\n",
    "    # d.shape is (seq_length, 24)\n",
    "    d = group.values\n",
    "    # column 1~22 are features.\n",
    "    # column 0 is Id and column 23 is target.\n",
    "    # save 1~22 features to 0~21 index of dataset up to d.shape[0].\n",
    "    X[i, :d.shape[0], 0:22] = d[:, 1:23]\n",
    "    y[i, 0] = d[0, 23]\n",
    "    i += 1;\n",
    "print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We make a python generator function to feed batch data to `fit_generator()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def feed_gen(X, y, batch_size=1024):\n",
    "    shuffled_index = np.random.permutation(len(X)) # shuffle index\n",
    "    start = 0\n",
    "    while 1:\n",
    "        end = start + batch_size\n",
    "        if end > len(X):  # cannot exceed X's length\n",
    "            end = len(X)\n",
    "        yield X[shuffled_index[start:end]], y[shuffled_index[start:end]]\n",
    "        start = end\n",
    "        if end >= len(X): # if arrive at the end, shuffle again.\n",
    "            shuffled_index = np.random.permutation(len(X))\n",
    "            start = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split dataset to train set and test set. In fact test set is validation set for this task. Total dataset size is 731,556. Test set is sufficient 10% of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate `steps_per_epoch` and `validation_steps` for `fit_generator()` method. We set `batch_size` is 1024."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(643.0, 72.0)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 1024\n",
    "steps_per_epoch = np.ceil(X_train.shape[0] / batch_size)\n",
    "validation_steps = np.ceil(X_test.shape[0] / batch_size)\n",
    "steps_per_epoch, validation_steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can make two generator for training and testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_gen = feed_gen(X_train, y_train, batch_size)\n",
    "val_gen = feed_gen(X_test, y_test, batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model has two layers. `LSTM` layer has 35 units. It is just Luis choice. Final `Dense` layer has 1 unit without activation for `Expected` target. All other parameters of `LSTM` and `Dense` layer use default value. Loss is `'mae'` for regression task. We choose `'rmsprop'` optimizer with default values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, LSTM\n",
    "\n",
    "model = Sequential()\n",
    "model.add(LSTM(35, input_shape=(None, 22)))\n",
    "model.add(Dense(1))\n",
    "\n",
    "model.compile(loss='mae', optimizer='rmsprop')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model training is simple. We just put the two generators and steps for epoch and validation. Let's try 100 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "643/643 [==============================] - 15s 23ms/step - loss: 23.3592 - val_loss: 22.4884\n",
      "Epoch 2/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.2784 - val_loss: 22.8763\n",
      "Epoch 3/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.2461 - val_loss: 22.5458\n",
      "Epoch 4/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.2264 - val_loss: 22.2134\n",
      "Epoch 5/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.2092 - val_loss: 22.8454\n",
      "Epoch 6/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1981 - val_loss: 23.1129\n",
      "Epoch 7/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1891 - val_loss: 22.3145\n",
      "Epoch 8/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1821 - val_loss: 21.9953\n",
      "Epoch 9/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1786 - val_loss: 22.3672\n",
      "Epoch 10/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1720 - val_loss: 23.1699\n",
      "Epoch 11/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1685 - val_loss: 22.4761\n",
      "Epoch 12/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1650 - val_loss: 21.7791\n",
      "Epoch 13/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1613 - val_loss: 22.2748\n",
      "Epoch 14/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1596 - val_loss: 23.2295\n",
      "Epoch 15/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1558 - val_loss: 22.7003\n",
      "Epoch 16/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1543 - val_loss: 22.4209\n",
      "Epoch 17/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1514 - val_loss: 21.8673\n",
      "Epoch 18/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1495 - val_loss: 22.3021\n",
      "Epoch 19/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1477 - val_loss: 22.4935\n",
      "Epoch 20/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1462 - val_loss: 23.3478\n",
      "Epoch 21/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1431 - val_loss: 21.3769\n",
      "Epoch 22/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1428 - val_loss: 21.9813\n",
      "Epoch 23/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1396 - val_loss: 22.3207\n",
      "Epoch 24/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1373 - val_loss: 22.6694\n",
      "Epoch 25/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1369 - val_loss: 22.1238\n",
      "Epoch 26/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1365 - val_loss: 21.3457\n",
      "Epoch 27/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1358 - val_loss: 22.6521\n",
      "Epoch 28/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1337 - val_loss: 21.9777\n",
      "Epoch 29/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1312 - val_loss: 22.9756\n",
      "Epoch 30/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1305 - val_loss: 22.3396\n",
      "Epoch 31/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1287 - val_loss: 22.1856\n",
      "Epoch 32/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1278 - val_loss: 22.3289\n",
      "Epoch 33/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1268 - val_loss: 21.7444\n",
      "Epoch 34/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1245 - val_loss: 23.2074\n",
      "Epoch 35/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1238 - val_loss: 22.0729\n",
      "Epoch 36/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1228 - val_loss: 21.4301\n",
      "Epoch 37/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1223 - val_loss: 22.5052\n",
      "Epoch 38/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1210 - val_loss: 21.5444\n",
      "Epoch 39/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1195 - val_loss: 22.3118\n",
      "Epoch 40/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1193 - val_loss: 22.7369\n",
      "Epoch 41/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1195 - val_loss: 22.1068\n",
      "Epoch 42/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1191 - val_loss: 21.6861\n",
      "Epoch 43/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1169 - val_loss: 22.4820\n",
      "Epoch 44/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1169 - val_loss: 22.3503\n",
      "Epoch 45/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1156 - val_loss: 22.2530\n",
      "Epoch 46/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1145 - val_loss: 21.5551\n",
      "Epoch 47/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1153 - val_loss: 22.6081\n",
      "Epoch 48/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1135 - val_loss: 21.7304\n",
      "Epoch 49/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1125 - val_loss: 22.7773\n",
      "Epoch 50/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1113 - val_loss: 22.8420\n",
      "Epoch 51/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1113 - val_loss: 22.8178\n",
      "Epoch 52/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1106 - val_loss: 22.4296\n",
      "Epoch 53/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1100 - val_loss: 21.7362\n",
      "Epoch 54/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1084 - val_loss: 22.7183\n",
      "Epoch 55/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1081 - val_loss: 22.3555\n",
      "Epoch 56/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1084 - val_loss: 22.4259\n",
      "Epoch 57/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1069 - val_loss: 23.2153\n",
      "Epoch 58/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1061 - val_loss: 22.9800\n",
      "Epoch 59/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1052 - val_loss: 22.7757\n",
      "Epoch 60/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1048 - val_loss: 21.8380\n",
      "Epoch 61/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1039 - val_loss: 21.5761\n",
      "Epoch 62/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1042 - val_loss: 22.3047\n",
      "Epoch 63/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1023 - val_loss: 23.1328\n",
      "Epoch 64/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1026 - val_loss: 22.4232\n",
      "Epoch 65/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1021 - val_loss: 22.2933\n",
      "Epoch 66/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1015 - val_loss: 21.1154\n",
      "Epoch 67/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1006 - val_loss: 22.1923\n",
      "Epoch 68/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.1005 - val_loss: 21.9716\n",
      "Epoch 69/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0997 - val_loss: 22.3442\n",
      "Epoch 70/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0983 - val_loss: 22.2724\n",
      "Epoch 71/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0991 - val_loss: 22.5911\n",
      "Epoch 72/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0983 - val_loss: 21.6359\n",
      "Epoch 73/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0982 - val_loss: 22.7203\n",
      "Epoch 74/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0974 - val_loss: 22.4947\n",
      "Epoch 75/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0953 - val_loss: 20.7540\n",
      "Epoch 76/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0951 - val_loss: 22.6389\n",
      "Epoch 77/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0967 - val_loss: 22.2818\n",
      "Epoch 78/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0939 - val_loss: 22.0626\n",
      "Epoch 79/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0944 - val_loss: 22.2057\n",
      "Epoch 80/100\n",
      "643/643 [==============================] - 12s 19ms/step - loss: 23.0934 - val_loss: 23.2137\n",
      "Epoch 81/100\n",
      "643/643 [==============================] - 13s 21ms/step - loss: 23.0940 - val_loss: 21.4464\n",
      "Epoch 82/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0915 - val_loss: 22.6907\n",
      "Epoch 83/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0922 - val_loss: 22.5729\n",
      "Epoch 84/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0912 - val_loss: 22.4931\n",
      "Epoch 85/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0902 - val_loss: 22.3457\n",
      "Epoch 86/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0901 - val_loss: 21.6814\n",
      "Epoch 87/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0899 - val_loss: 21.8171\n",
      "Epoch 88/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0895 - val_loss: 22.9623\n",
      "Epoch 89/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0900 - val_loss: 23.2738\n",
      "Epoch 90/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0888 - val_loss: 21.2127\n",
      "Epoch 91/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0879 - val_loss: 22.2649\n",
      "Epoch 92/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0868 - val_loss: 22.1007\n",
      "Epoch 93/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0862 - val_loss: 22.3556\n",
      "Epoch 94/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0863 - val_loss: 22.7753\n",
      "Epoch 95/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0865 - val_loss: 22.3068\n",
      "Epoch 96/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0860 - val_loss: 21.9186\n",
      "Epoch 97/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0861 - val_loss: 21.5697\n",
      "Epoch 98/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0845 - val_loss: 22.2280\n",
      "Epoch 99/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0857 - val_loss: 22.9889\n",
      "Epoch 100/100\n",
      "643/643 [==============================] - 13s 20ms/step - loss: 23.0853 - val_loss: 21.9630\n"
     ]
    }
   ],
   "source": [
    "history = model.fit_generator(train_gen,\n",
    "                              steps_per_epoch=steps_per_epoch,\n",
    "                              epochs=100,\n",
    "                              validation_data=val_gen,\n",
    "                              validation_steps=validation_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can make prediction for test data. Read the test.csv file and fill `NaN` with 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "rain2_test = pd.read_csv('datasets/rain2/test.csv')\n",
    "test_df = rain2_test.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test data has 23 columns not 24 columns. It doesn't have `Expected` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>minutes_past</th>\n",
       "      <th>radardist_km</th>\n",
       "      <th>Ref</th>\n",
       "      <th>Ref_5x5_10th</th>\n",
       "      <th>Ref_5x5_50th</th>\n",
       "      <th>Ref_5x5_90th</th>\n",
       "      <th>RefComposite</th>\n",
       "      <th>RefComposite_5x5_10th</th>\n",
       "      <th>RefComposite_5x5_50th</th>\n",
       "      <th>...</th>\n",
       "      <th>RhoHV_5x5_50th</th>\n",
       "      <th>RhoHV_5x5_90th</th>\n",
       "      <th>Zdr</th>\n",
       "      <th>Zdr_5x5_10th</th>\n",
       "      <th>Zdr_5x5_50th</th>\n",
       "      <th>Zdr_5x5_90th</th>\n",
       "      <th>Kdp</th>\n",
       "      <th>Kdp_5x5_10th</th>\n",
       "      <th>Kdp_5x5_50th</th>\n",
       "      <th>Kdp_5x5_90th</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>11.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>14.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>8.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>15.5</td>\n",
       "      <td>13.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  minutes_past  radardist_km   Ref  Ref_5x5_10th  Ref_5x5_50th  \\\n",
       "0   1             1           8.0   0.0           0.0           0.0   \n",
       "1   1             5           8.0  10.0           0.0          10.0   \n",
       "2   1             8           8.0   0.0           0.0           7.0   \n",
       "3   1            12           8.0  14.0           0.0           9.0   \n",
       "4   1            15           8.0  10.5           0.0           9.0   \n",
       "\n",
       "   Ref_5x5_90th  RefComposite  RefComposite_5x5_10th  RefComposite_5x5_50th  \\\n",
       "0          14.0           0.0                    0.0                    0.0   \n",
       "1          18.0          11.5                    0.0                   11.0   \n",
       "2          14.5           0.0                    0.0                    7.0   \n",
       "3          16.0          14.0                    0.0                    9.0   \n",
       "4          15.5          13.5                    0.0                    9.0   \n",
       "\n",
       "       ...       RhoHV_5x5_50th  RhoHV_5x5_90th  Zdr  Zdr_5x5_10th  \\\n",
       "0      ...                  0.0             0.0  0.0           0.0   \n",
       "1      ...                  0.0             0.0  0.0           0.0   \n",
       "2      ...                  0.0             0.0  0.0           0.0   \n",
       "3      ...                  0.0             0.0  0.0           0.0   \n",
       "4      ...                  0.0             0.0  0.0           0.0   \n",
       "\n",
       "   Zdr_5x5_50th  Zdr_5x5_90th  Kdp  Kdp_5x5_10th  Kdp_5x5_50th  Kdp_5x5_90th  \n",
       "0           0.0           0.0  0.0           0.0           0.0           0.0  \n",
       "1           0.0           0.0  0.0           0.0           0.0           0.0  \n",
       "2           0.0           0.0  0.0           0.0           0.0           0.0  \n",
       "3           0.0           0.0  0.0           0.0           0.0           0.0  \n",
       "4           0.0           0.0  0.0           0.0           0.0           0.0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. Let's check the number of sequences and maximum length just like above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(717625, 19)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_seq = test_df.groupby(['Id'])\n",
    "test_seq_size = test_seq.size()\n",
    "test_seq_size.count(), test_seq_size.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create `X_text` numpy array. Its size is (717625, 19, 22)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = np.zeros((717625, 19, 22))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like train data, copy `test_seq` to `X_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "717625\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "for name, group in test_seq:\n",
    "    # d.shape is (seq_length, 23)\n",
    "    d = group.values\n",
    "    # column 1~22 are features.\n",
    "    # save 1~22 features to 0~21 index of dataset up to d.shape[0].\n",
    "    X_test[i, :d.shape[0], 0:22] = d[:, 1:23]\n",
    "    i += 1;\n",
    "print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can feed `X_test` to our model and get the predictions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making submission file, we stack `pred` array and its index horizontally. Using `numpy.savetxt`, we can make csv file with format(\"int, float\") and title(\"Id,Expected\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_with_index = np.hstack((np.arange(1, pred.shape[0]+1).reshape(-1,1), pred))\n",
    "np.savetxt(\"datasets/rain2/test_prediction.csv\", pred_with_index, \"%d,%f\", \n",
    "           delimiter=\",\", header=\"Id,Expected\", comments=\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Id,Expected\r\n",
      "1,0.611155\r\n",
      "2,1.193402\r\n",
      "3,3.511984\r\n",
      "4,7.424457\r\n",
      "5,0.799749\r\n",
      "6,0.733051\r\n",
      "7,2.819939\r\n",
      "8,0.949913\r\n",
      "9,0.663545\r\n"
     ]
    }
   ],
   "source": [
    "!head \"datasets/rain2/test_prediction.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you submit this file to Kaggle competition(https://www.kaggle.com/c/how-much-did-it-rain-ii/submit), you will get about 23.76 score. It may be ranked before 100. Cool!\n",
    "\n",
    "You can stack another `LSTM` layer and change weight initializer or optimizer. If you want to search more parameters, it's better to use `keras.wrappers.scikit_learn.KerasClassifier` with scikit-learn's `GridSearchCV`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}