14_recurrent_neural_networks_ex_8-2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ch 14. Excercise 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, Luis already put his solution to github(https://github.com/luisandresilva/Kaggle/tree/master/Rain-II). He used Keras API and python 2. Here, we will reproduce his model for latest tf.keras API and python 3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, download data of \"How Much Did It Rain? II\" competition and put it on `datasets/rain2` folder. Read the `train.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "rain2 = pd.read_csv('datasets/rain2/train.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's view some rows via `head()` method of pandas's `DataFrame`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>minutes_past</th>\n",
       "      <th>radardist_km</th>\n",
       "      <th>Ref</th>\n",
       "      <th>Ref_5x5_10th</th>\n",
       "      <th>Ref_5x5_50th</th>\n",
       "      <th>Ref_5x5_90th</th>\n",
       "      <th>RefComposite</th>\n",
       "      <th>RefComposite_5x5_10th</th>\n",
       "      <th>RefComposite_5x5_50th</th>\n",
       "      <th>...</th>\n",
       "      <th>RhoHV_5x5_90th</th>\n",
       "      <th>Zdr</th>\n",
       "      <th>Zdr_5x5_10th</th>\n",
       "      <th>Zdr_5x5_50th</th>\n",
       "      <th>Zdr_5x5_90th</th>\n",
       "      <th>Kdp</th>\n",
       "      <th>Kdp_5x5_10th</th>\n",
       "      <th>Kdp_5x5_50th</th>\n",
       "      <th>Kdp_5x5_90th</th>\n",
       "      <th>Expected</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>45</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>55</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.254000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>10.5</td>\n",
       "      <td>15.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>16.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>-0.1250</td>\n",
       "      <td>0.3125</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>1.059998</td>\n",
       "      <td>-1.410004</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>1.059998</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>26.5</td>\n",
       "      <td>22.5</td>\n",
       "      <td>25.5</td>\n",
       "      <td>31.5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>28.5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.005000</td>\n",
       "      <td>0.0625</td>\n",
       "      <td>-0.1875</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>0.6875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.409988</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>2.0</td>\n",
       "      <td>21.5</td>\n",
       "      <td>15.5</td>\n",
       "      <td>20.5</td>\n",
       "      <td>25.0</td>\n",
       "      <td>26.5</td>\n",
       "      <td>23.5</td>\n",
       "      <td>25.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.001667</td>\n",
       "      <td>0.3125</td>\n",
       "      <td>-0.0625</td>\n",
       "      <td>0.3125</td>\n",
       "      <td>0.6250</td>\n",
       "      <td>0.349991</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>1.759994</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2</td>\n",
       "      <td>16</td>\n",
       "      <td>2.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>17.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>20.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>20.5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.001667</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>0.1250</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>0.6875</td>\n",
       "      <td>0.349991</td>\n",
       "      <td>-1.059998</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.059998</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2</td>\n",
       "      <td>21</td>\n",
       "      <td>2.0</td>\n",
       "      <td>24.5</td>\n",
       "      <td>16.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>24.5</td>\n",
       "      <td>24.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>0.0625</td>\n",
       "      <td>0.1875</td>\n",
       "      <td>0.5625</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>-1.059998</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>1.759994</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2</td>\n",
       "      <td>26</td>\n",
       "      <td>2.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>16.5</td>\n",
       "      <td>17.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.5625</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>0.4375</td>\n",
       "      <td>0.6875</td>\n",
       "      <td>-1.760010</td>\n",
       "      <td>-1.760010</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>0.709991</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2</td>\n",
       "      <td>31</td>\n",
       "      <td>2.0</td>\n",
       "      <td>22.5</td>\n",
       "      <td>19.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>23.5</td>\n",
       "      <td>25.5</td>\n",
       "      <td>...</td>\n",
       "      <td>1.001667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.1875</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>0.6250</td>\n",
       "      <td>-1.059998</td>\n",
       "      <td>-2.120010</td>\n",
       "      <td>-0.710007</td>\n",
       "      <td>0.349991</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2</td>\n",
       "      <td>37</td>\n",
       "      <td>2.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>18.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>19.5</td>\n",
       "      <td>20.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.1875</td>\n",
       "      <td>0.4375</td>\n",
       "      <td>0.8125</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.760010</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>1.059998</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2</td>\n",
       "      <td>42</td>\n",
       "      <td>2.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.5</td>\n",
       "      <td>17.0</td>\n",
       "      <td>19.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.6250</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>0.6250</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.349991</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2</td>\n",
       "      <td>47</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>3.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>16.5</td>\n",
       "      <td>18.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>0.1875</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.6875</td>\n",
       "      <td>0.349991</td>\n",
       "      <td>-2.110001</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>1.059998</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>14.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>23.5</td>\n",
       "      <td>28.0</td>\n",
       "      <td>23.5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998333</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>0.6250</td>\n",
       "      <td>0.9375</td>\n",
       "      <td>1.3750</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>-1.410004</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>2.119995</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>2</td>\n",
       "      <td>58</td>\n",
       "      <td>2.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>16.5</td>\n",
       "      <td>22.5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>31.5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>29.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.001667</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>0.1875</td>\n",
       "      <td>0.3750</td>\n",
       "      <td>0.8750</td>\n",
       "      <td>-1.410004</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.350006</td>\n",
       "      <td>0.699997</td>\n",
       "      <td>1.016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>3</td>\n",
       "      <td>14</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>3</td>\n",
       "      <td>18</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>3</td>\n",
       "      <td>23</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>3</td>\n",
       "      <td>28</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>3</td>\n",
       "      <td>33</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>3</td>\n",
       "      <td>38</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>3</td>\n",
       "      <td>43</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>3</td>\n",
       "      <td>48</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>3</td>\n",
       "      <td>53</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0.801667</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0625</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>3</td>\n",
       "      <td>58</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.162014</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>30 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Id  minutes_past  radardist_km   Ref  Ref_5x5_10th  Ref_5x5_50th  \\\n",
       "0    1             3          10.0   NaN           NaN           NaN   \n",
       "1    1            16          10.0   NaN           NaN           NaN   \n",
       "2    1            25          10.0   NaN           NaN           NaN   \n",
       "3    1            35          10.0   NaN           NaN           NaN   \n",
       "4    1            45          10.0   NaN           NaN           NaN   \n",
       "5    1            55          10.0   NaN           NaN           NaN   \n",
       "6    2             1           2.0   9.0           5.0           7.5   \n",
       "7    2             6           2.0  26.5          22.5          25.5   \n",
       "8    2            11           2.0  21.5          15.5          20.5   \n",
       "9    2            16           2.0  18.0          14.0          17.5   \n",
       "10   2            21           2.0  24.5          16.5          21.0   \n",
       "11   2            26           2.0  12.0          12.0          16.0   \n",
       "12   2            31           2.0  22.5          19.0          22.0   \n",
       "13   2            37           2.0  14.0          14.0          18.5   \n",
       "14   2            42           2.0  12.0          11.0          12.5   \n",
       "15   2            47           2.0   1.5           3.5           7.0   \n",
       "16   2            53           2.0  16.0          14.5          18.0   \n",
       "17   2            58           2.0  22.0          16.5          22.5   \n",
       "18   3             4          10.0   NaN           NaN           NaN   \n",
       "19   3             9          10.0   NaN           NaN           NaN   \n",
       "20   3            14          10.0   NaN           NaN           NaN   \n",
       "21   3            18          10.0   NaN           NaN           NaN   \n",
       "22   3            23          10.0   NaN           NaN           NaN   \n",
       "23   3            28          10.0   NaN           NaN           NaN   \n",
       "24   3            33          10.0   NaN           NaN           NaN   \n",
       "25   3            38          10.0   NaN           NaN           NaN   \n",
       "26   3            43          10.0   NaN           NaN           NaN   \n",
       "27   3            48          10.0   NaN           NaN           NaN   \n",
       "28   3            53          10.0   NaN           NaN           NaN   \n",
       "29   3            58          10.0   NaN           NaN           NaN   \n",
       "\n",
       "    Ref_5x5_90th  RefComposite  RefComposite_5x5_10th  RefComposite_5x5_50th  \\\n",
       "0            NaN           NaN                    NaN                    NaN   \n",
       "1            NaN           NaN                    NaN                    NaN   \n",
       "2            NaN           NaN                    NaN                    NaN   \n",
       "3            NaN           NaN                    NaN                    NaN   \n",
       "4            NaN           NaN                    NaN                    NaN   \n",
       "5            NaN           NaN                    NaN                    NaN   \n",
       "6           10.5          15.0                   10.5                   16.5   \n",
       "7           31.5          26.5                   26.5                   28.5   \n",
       "8           25.0          26.5                   23.5                   25.0   \n",
       "9           21.0          20.5                   18.0                   20.5   \n",
       "10          24.5          24.5                   21.0                   24.0   \n",
       "11          20.0          16.5                   17.0                   19.0   \n",
       "12          25.0          26.0                   23.5                   25.5   \n",
       "13          21.0          19.5                   20.0                   21.0   \n",
       "14          17.0          19.5                   18.0                   21.0   \n",
       "15          10.5          18.0                   16.5                   18.5   \n",
       "16          23.5          28.0                   23.5                   26.5   \n",
       "17          26.5          31.5                   26.5                   29.0   \n",
       "18           NaN           NaN                    NaN                    NaN   \n",
       "19           NaN           NaN                    NaN                    NaN   \n",
       "20           NaN           NaN                    NaN                    NaN   \n",
       "21           NaN           NaN                    NaN                    NaN   \n",
       "22           NaN           NaN                    NaN                    NaN   \n",
       "23           NaN           NaN                    NaN                    NaN   \n",
       "24           NaN           NaN                    NaN                    NaN   \n",
       "25           NaN           NaN                    NaN                    NaN   \n",
       "26           8.5           NaN                    NaN                    NaN   \n",
       "27           NaN           NaN                    NaN                    NaN   \n",
       "28           NaN           NaN                    NaN                    NaN   \n",
       "29           NaN           NaN                    NaN                    NaN   \n",
       "\n",
       "      ...      RhoHV_5x5_90th     Zdr  Zdr_5x5_10th  Zdr_5x5_50th  \\\n",
       "0     ...                 NaN     NaN           NaN           NaN   \n",
       "1     ...                 NaN     NaN           NaN           NaN   \n",
       "2     ...                 NaN     NaN           NaN           NaN   \n",
       "3     ...                 NaN     NaN           NaN           NaN   \n",
       "4     ...                 NaN     NaN           NaN           NaN   \n",
       "5     ...                 NaN     NaN           NaN           NaN   \n",
       "6     ...            0.998333  0.3750       -0.1250        0.3125   \n",
       "7     ...            1.005000  0.0625       -0.1875        0.2500   \n",
       "8     ...            1.001667  0.3125       -0.0625        0.3125   \n",
       "9     ...            1.001667  0.2500        0.1250        0.3750   \n",
       "10    ...            0.998333  0.2500        0.0625        0.1875   \n",
       "11    ...            0.998333  0.5625        0.2500        0.4375   \n",
       "12    ...            1.001667  0.0000       -0.1875        0.2500   \n",
       "13    ...            0.998333  0.5000        0.1875        0.4375   \n",
       "14    ...            0.998333  0.6250        0.3750        0.6250   \n",
       "15    ...            0.998333  0.3750        0.1875        0.5000   \n",
       "16    ...            0.998333  0.8750        0.6250        0.9375   \n",
       "17    ...            1.001667  0.3750        0.1875        0.3750   \n",
       "18    ...                 NaN     NaN           NaN           NaN   \n",
       "19    ...                 NaN     NaN           NaN           NaN   \n",
       "20    ...                 NaN     NaN           NaN           NaN   \n",
       "21    ...                 NaN     NaN           NaN           NaN   \n",
       "22    ...                 NaN     NaN           NaN           NaN   \n",
       "23    ...                 NaN     NaN           NaN           NaN   \n",
       "24    ...                 NaN     NaN           NaN           NaN   \n",
       "25    ...                 NaN     NaN           NaN           NaN   \n",
       "26    ...                 NaN     NaN           NaN           NaN   \n",
       "27    ...                 NaN     NaN           NaN           NaN   \n",
       "28    ...            0.801667     NaN           NaN           NaN   \n",
       "29    ...                 NaN     NaN           NaN           NaN   \n",
       "\n",
       "    Zdr_5x5_90th       Kdp  Kdp_5x5_10th  Kdp_5x5_50th  Kdp_5x5_90th  \\\n",
       "0            NaN       NaN           NaN           NaN           NaN   \n",
       "1            NaN       NaN           NaN           NaN           NaN   \n",
       "2            NaN       NaN           NaN           NaN           NaN   \n",
       "3            NaN       NaN           NaN           NaN           NaN   \n",
       "4            NaN       NaN           NaN           NaN           NaN   \n",
       "5            NaN       NaN           NaN           NaN           NaN   \n",
       "6         0.8750  1.059998     -1.410004     -0.350006      1.059998   \n",
       "7         0.6875       NaN           NaN           NaN      1.409988   \n",
       "8         0.6250  0.349991           NaN     -0.350006      1.759994   \n",
       "9         0.6875  0.349991     -1.059998      0.000000      1.059998   \n",
       "10        0.5625 -0.350006     -1.059998     -0.350006      1.759994   \n",
       "11        0.6875 -1.760010     -1.760010     -0.350006      0.709991   \n",
       "12        0.6250 -1.059998     -2.120010     -0.710007      0.349991   \n",
       "13        0.8125  0.000000     -1.760010     -0.350006      1.059998   \n",
       "14        0.8750 -0.350006     -0.350006      0.000000      0.349991   \n",
       "15        0.6875  0.349991     -2.110001     -0.350006      1.059998   \n",
       "16        1.3750 -0.350006     -1.410004     -0.350006      2.119995   \n",
       "17        0.8750 -1.410004           NaN     -0.350006      0.699997   \n",
       "18           NaN       NaN           NaN           NaN           NaN   \n",
       "19           NaN       NaN           NaN           NaN           NaN   \n",
       "20           NaN       NaN           NaN           NaN           NaN   \n",
       "21           NaN       NaN           NaN           NaN           NaN   \n",
       "22           NaN       NaN           NaN           NaN           NaN   \n",
       "23           NaN       NaN           NaN           NaN           NaN   \n",
       "24           NaN       NaN           NaN           NaN           NaN   \n",
       "25           NaN       NaN           NaN           NaN           NaN   \n",
       "26           NaN       NaN           NaN           NaN           NaN   \n",
       "27           NaN       NaN           NaN           NaN           NaN   \n",
       "28        2.0625       NaN           NaN           NaN           NaN   \n",
       "29           NaN       NaN           NaN           NaN           NaN   \n",
       "\n",
       "     Expected  \n",
       "0    0.254000  \n",
       "1    0.254000  \n",
       "2    0.254000  \n",
       "3    0.254000  \n",
       "4    0.254000  \n",
       "5    0.254000  \n",
       "6    1.016000  \n",
       "7    1.016000  \n",
       "8    1.016000  \n",
       "9    1.016000  \n",
       "10   1.016000  \n",
       "11   1.016000  \n",
       "12   1.016000  \n",
       "13   1.016000  \n",
       "14   1.016000  \n",
       "15   1.016000  \n",
       "16   1.016000  \n",
       "17   1.016000  \n",
       "18  26.162014  \n",
       "19  26.162014  \n",
       "20  26.162014  \n",
       "21  26.162014  \n",
       "22  26.162014  \n",
       "23  26.162014  \n",
       "24  26.162014  \n",
       "25  26.162014  \n",
       "26  26.162014  \n",
       "27  26.162014  \n",
       "28  26.162014  \n",
       "29  26.162014  \n",
       "\n",
       "[30 rows x 24 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rain2.head(n=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 24 columns. First column `Id` is unique value representing a sequence. Last column `Expected` is target. We simply remove rows that `Ref` column is `NaN`. If `Ref` column is `NaN`, all other columns is `NaN` too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = rain2.dropna(subset=['Ref'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `describe()` method, we get the statistics for dataset. The features excluding `Id` and `Expected` has different scale. But the order of magnitude is not significant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>minutes_past</th>\n",
       "      <th>radardist_km</th>\n",
       "      <th>Ref</th>\n",
       "      <th>Ref_5x5_10th</th>\n",
       "      <th>Ref_5x5_50th</th>\n",
       "      <th>Ref_5x5_90th</th>\n",
       "      <th>RefComposite</th>\n",
       "      <th>RefComposite_5x5_10th</th>\n",
       "      <th>RefComposite_5x5_50th</th>\n",
       "      <th>...</th>\n",
       "      <th>RhoHV_5x5_90th</th>\n",
       "      <th>Zdr</th>\n",
       "      <th>Zdr_5x5_10th</th>\n",
       "      <th>Zdr_5x5_50th</th>\n",
       "      <th>Zdr_5x5_90th</th>\n",
       "      <th>Kdp</th>\n",
       "      <th>Kdp_5x5_10th</th>\n",
       "      <th>Kdp_5x5_50th</th>\n",
       "      <th>Kdp_5x5_90th</th>\n",
       "      <th>Expected</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>6.349375e+06</td>\n",
       "      <td>6.349375e+06</td>\n",
       "      <td>6.349375e+06</td>\n",
       "      <td>6.349375e+06</td>\n",
       "      <td>5.269105e+06</td>\n",
       "      <td>6.156846e+06</td>\n",
       "      <td>6.349288e+06</td>\n",
       "      <td>6.349375e+06</td>\n",
       "      <td>5.550921e+06</td>\n",
       "      <td>6.207490e+06</td>\n",
       "      <td>...</td>\n",
       "      <td>4.600898e+06</td>\n",
       "      <td>4.069799e+06</td>\n",
       "      <td>3.577674e+06</td>\n",
       "      <td>4.095586e+06</td>\n",
       "      <td>4.600898e+06</td>\n",
       "      <td>3.494665e+06</td>\n",
       "      <td>2.978691e+06</td>\n",
       "      <td>3.511354e+06</td>\n",
       "      <td>4.062702e+06</td>\n",
       "      <td>6.349375e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.894813e+05</td>\n",
       "      <td>2.927051e+01</td>\n",
       "      <td>9.286182e+00</td>\n",
       "      <td>2.292666e+01</td>\n",
       "      <td>1.998385e+01</td>\n",
       "      <td>2.302103e+01</td>\n",
       "      <td>2.812518e+01</td>\n",
       "      <td>2.540071e+01</td>\n",
       "      <td>2.256900e+01</td>\n",
       "      <td>2.541082e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>1.013113e+00</td>\n",
       "      <td>5.295568e-01</td>\n",
       "      <td>-6.975932e-01</td>\n",
       "      <td>3.815768e-01</td>\n",
       "      <td>2.057199e+00</td>\n",
       "      <td>3.230033e-02</td>\n",
       "      <td>-3.409001e+00</td>\n",
       "      <td>-3.781228e-01</td>\n",
       "      <td>4.038652e+00</td>\n",
       "      <td>1.714876e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.406124e+05</td>\n",
       "      <td>1.715383e+01</td>\n",
       "      <td>4.068784e+00</td>\n",
       "      <td>1.035516e+01</td>\n",
       "      <td>9.195141e+00</td>\n",
       "      <td>9.882618e+00</td>\n",
       "      <td>1.028228e+01</td>\n",
       "      <td>1.044013e+01</td>\n",
       "      <td>9.547383e+00</td>\n",
       "      <td>1.003713e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>4.179456e-02</td>\n",
       "      <td>1.476643e+00</td>\n",
       "      <td>1.017368e+00</td>\n",
       "      <td>9.260191e-01</td>\n",
       "      <td>1.617217e+00</td>\n",
       "      <td>3.699795e+00</td>\n",
       "      <td>2.719573e+00</td>\n",
       "      <td>2.087361e+00</td>\n",
       "      <td>3.902270e+00</td>\n",
       "      <td>2.025279e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>-3.100000e+01</td>\n",
       "      <td>-3.200000e+01</td>\n",
       "      <td>-3.200000e+01</td>\n",
       "      <td>-2.850000e+01</td>\n",
       "      <td>-2.800000e+01</td>\n",
       "      <td>-3.050000e+01</td>\n",
       "      <td>-2.750000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>2.083333e-01</td>\n",
       "      <td>-7.875000e+00</td>\n",
       "      <td>-7.875000e+00</td>\n",
       "      <td>-7.875000e+00</td>\n",
       "      <td>-7.875000e+00</td>\n",
       "      <td>-9.604000e+01</td>\n",
       "      <td>-8.079000e+01</td>\n",
       "      <td>-7.168000e+01</td>\n",
       "      <td>-1.002000e+02</td>\n",
       "      <td>1.000000e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.923290e+05</td>\n",
       "      <td>1.400000e+01</td>\n",
       "      <td>6.000000e+00</td>\n",
       "      <td>1.600000e+01</td>\n",
       "      <td>1.450000e+01</td>\n",
       "      <td>1.650000e+01</td>\n",
       "      <td>2.150000e+01</td>\n",
       "      <td>1.850000e+01</td>\n",
       "      <td>1.650000e+01</td>\n",
       "      <td>1.850000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>9.983333e-01</td>\n",
       "      <td>-1.875000e-01</td>\n",
       "      <td>-1.062500e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.062500e+00</td>\n",
       "      <td>-1.410004e+00</td>\n",
       "      <td>-4.540008e+00</td>\n",
       "      <td>-7.100067e-01</td>\n",
       "      <td>2.059998e+00</td>\n",
       "      <td>5.080003e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.906240e+05</td>\n",
       "      <td>2.900000e+01</td>\n",
       "      <td>1.000000e+01</td>\n",
       "      <td>2.250000e+01</td>\n",
       "      <td>2.000000e+01</td>\n",
       "      <td>2.300000e+01</td>\n",
       "      <td>2.750000e+01</td>\n",
       "      <td>2.500000e+01</td>\n",
       "      <td>2.250000e+01</td>\n",
       "      <td>2.500000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>1.005000e+00</td>\n",
       "      <td>3.750000e-01</td>\n",
       "      <td>-6.250000e-01</td>\n",
       "      <td>3.125000e-01</td>\n",
       "      <td>1.687500e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>-2.820007e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>3.509994e+00</td>\n",
       "      <td>1.425001e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>8.883170e+05</td>\n",
       "      <td>4.400000e+01</td>\n",
       "      <td>1.200000e+01</td>\n",
       "      <td>2.950000e+01</td>\n",
       "      <td>2.600000e+01</td>\n",
       "      <td>2.950000e+01</td>\n",
       "      <td>3.500000e+01</td>\n",
       "      <td>3.200000e+01</td>\n",
       "      <td>2.900000e+01</td>\n",
       "      <td>3.200000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>1.051667e+00</td>\n",
       "      <td>1.062500e+00</td>\n",
       "      <td>-1.875000e-01</td>\n",
       "      <td>6.875000e-01</td>\n",
       "      <td>2.562500e+00</td>\n",
       "      <td>1.409988e+00</td>\n",
       "      <td>-1.740006e+00</td>\n",
       "      <td>3.499908e-01</td>\n",
       "      <td>5.629990e+00</td>\n",
       "      <td>4.064002e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.180945e+06</td>\n",
       "      <td>5.900000e+01</td>\n",
       "      <td>2.100000e+01</td>\n",
       "      <td>7.100000e+01</td>\n",
       "      <td>6.250000e+01</td>\n",
       "      <td>6.900000e+01</td>\n",
       "      <td>7.250000e+01</td>\n",
       "      <td>9.250000e+01</td>\n",
       "      <td>6.600000e+01</td>\n",
       "      <td>7.100000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>1.051667e+00</td>\n",
       "      <td>7.937500e+00</td>\n",
       "      <td>7.937500e+00</td>\n",
       "      <td>7.937500e+00</td>\n",
       "      <td>7.937500e+00</td>\n",
       "      <td>1.797500e+02</td>\n",
       "      <td>3.169998e+00</td>\n",
       "      <td>1.280000e+01</td>\n",
       "      <td>1.446000e+02</td>\n",
       "      <td>3.301773e+04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Id  minutes_past  radardist_km           Ref  Ref_5x5_10th  \\\n",
       "count  6.349375e+06  6.349375e+06  6.349375e+06  6.349375e+06  5.269105e+06   \n",
       "mean   5.894813e+05  2.927051e+01  9.286182e+00  2.292666e+01  1.998385e+01   \n",
       "std    3.406124e+05  1.715383e+01  4.068784e+00  1.035516e+01  9.195141e+00   \n",
       "min    2.000000e+00  0.000000e+00  0.000000e+00 -3.100000e+01 -3.200000e+01   \n",
       "25%    2.923290e+05  1.400000e+01  6.000000e+00  1.600000e+01  1.450000e+01   \n",
       "50%    5.906240e+05  2.900000e+01  1.000000e+01  2.250000e+01  2.000000e+01   \n",
       "75%    8.883170e+05  4.400000e+01  1.200000e+01  2.950000e+01  2.600000e+01   \n",
       "max    1.180945e+06  5.900000e+01  2.100000e+01  7.100000e+01  6.250000e+01   \n",
       "\n",
       "       Ref_5x5_50th  Ref_5x5_90th  RefComposite  RefComposite_5x5_10th  \\\n",
       "count  6.156846e+06  6.349288e+06  6.349375e+06           5.550921e+06   \n",
       "mean   2.302103e+01  2.812518e+01  2.540071e+01           2.256900e+01   \n",
       "std    9.882618e+00  1.028228e+01  1.044013e+01           9.547383e+00   \n",
       "min   -3.200000e+01 -2.850000e+01 -2.800000e+01          -3.050000e+01   \n",
       "25%    1.650000e+01  2.150000e+01  1.850000e+01           1.650000e+01   \n",
       "50%    2.300000e+01  2.750000e+01  2.500000e+01           2.250000e+01   \n",
       "75%    2.950000e+01  3.500000e+01  3.200000e+01           2.900000e+01   \n",
       "max    6.900000e+01  7.250000e+01  9.250000e+01           6.600000e+01   \n",
       "\n",
       "       RefComposite_5x5_50th      ...       RhoHV_5x5_90th           Zdr  \\\n",
       "count           6.207490e+06      ...         4.600898e+06  4.069799e+06   \n",
       "mean            2.541082e+01      ...         1.013113e+00  5.295568e-01   \n",
       "std             1.003713e+01      ...         4.179456e-02  1.476643e+00   \n",
       "min            -2.750000e+01      ...         2.083333e-01 -7.875000e+00   \n",
       "25%             1.850000e+01      ...         9.983333e-01 -1.875000e-01   \n",
       "50%             2.500000e+01      ...         1.005000e+00  3.750000e-01   \n",
       "75%             3.200000e+01      ...         1.051667e+00  1.062500e+00   \n",
       "max             7.100000e+01      ...         1.051667e+00  7.937500e+00   \n",
       "\n",
       "       Zdr_5x5_10th  Zdr_5x5_50th  Zdr_5x5_90th           Kdp  Kdp_5x5_10th  \\\n",
       "count  3.577674e+06  4.095586e+06  4.600898e+06  3.494665e+06  2.978691e+06   \n",
       "mean  -6.975932e-01  3.815768e-01  2.057199e+00  3.230033e-02 -3.409001e+00   \n",
       "std    1.017368e+00  9.260191e-01  1.617217e+00  3.699795e+00  2.719573e+00   \n",
       "min   -7.875000e+00 -7.875000e+00 -7.875000e+00 -9.604000e+01 -8.079000e+01   \n",
       "25%   -1.062500e+00  0.000000e+00  1.062500e+00 -1.410004e+00 -4.540008e+00   \n",
       "50%   -6.250000e-01  3.125000e-01  1.687500e+00  0.000000e+00 -2.820007e+00   \n",
       "75%   -1.875000e-01  6.875000e-01  2.562500e+00  1.409988e+00 -1.740006e+00   \n",
       "max    7.937500e+00  7.937500e+00  7.937500e+00  1.797500e+02  3.169998e+00   \n",
       "\n",
       "       Kdp_5x5_50th  Kdp_5x5_90th      Expected  \n",
       "count  3.511354e+06  4.062702e+06  6.349375e+06  \n",
       "mean  -3.781228e-01  4.038652e+00  1.714876e+01  \n",
       "std    2.087361e+00  3.902270e+00  2.025279e+02  \n",
       "min   -7.168000e+01 -1.002000e+02  1.000000e-02  \n",
       "25%   -7.100067e-01  2.059998e+00  5.080003e-01  \n",
       "50%    0.000000e+00  3.509994e+00  1.425001e+00  \n",
       "75%    3.499908e-01  5.629990e+00  4.064002e+00  \n",
       "max    1.280000e+01  1.446000e+02  3.301773e+04  \n",
       "\n",
       "[8 rows x 24 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We removed the rows that `Ref` column is `NaN`. But there are still `NaN` values. Let's check How many `NaN` are in the `train_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                             0\n",
       "minutes_past                   0\n",
       "radardist_km                   0\n",
       "Ref                            0\n",
       "Ref_5x5_10th             1080270\n",
       "Ref_5x5_50th              192529\n",
       "Ref_5x5_90th                  87\n",
       "RefComposite                   0\n",
       "RefComposite_5x5_10th     798454\n",
       "RefComposite_5x5_50th     141885\n",
       "RefComposite_5x5_90th         67\n",
       "RhoHV                    2279576\n",
       "RhoHV_5x5_10th           2771701\n",
       "RhoHV_5x5_50th           2253789\n",
       "RhoHV_5x5_90th           1748477\n",
       "Zdr                      2279576\n",
       "Zdr_5x5_10th             2771701\n",
       "Zdr_5x5_50th             2253789\n",
       "Zdr_5x5_90th             1748477\n",
       "Kdp                      2854710\n",
       "Kdp_5x5_10th             3370684\n",
       "Kdp_5x5_50th             2838021\n",
       "Kdp_5x5_90th             2286673\n",
       "Expected                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. For simplicity, just fill `NaN` cell with 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                       0\n",
       "minutes_past             0\n",
       "radardist_km             0\n",
       "Ref                      0\n",
       "Ref_5x5_10th             0\n",
       "Ref_5x5_50th             0\n",
       "Ref_5x5_90th             0\n",
       "RefComposite             0\n",
       "RefComposite_5x5_10th    0\n",
       "RefComposite_5x5_50th    0\n",
       "RefComposite_5x5_90th    0\n",
       "RhoHV                    0\n",
       "RhoHV_5x5_10th           0\n",
       "RhoHV_5x5_50th           0\n",
       "RhoHV_5x5_90th           0\n",
       "Zdr                      0\n",
       "Zdr_5x5_10th             0\n",
       "Zdr_5x5_50th             0\n",
       "Zdr_5x5_90th             0\n",
       "Kdp                      0\n",
       "Kdp_5x5_10th             0\n",
       "Kdp_5x5_50th             0\n",
       "Kdp_5x5_90th             0\n",
       "Expected                 0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df = train_df.fillna(0)\n",
    "train_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can group by `Id` column via `groupby` method. This is the sequence we feed into the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_seq = train_df.groupby(['Id'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's check the number of unique `Id` and maximum length of sequence. `size()` method return the length of each group. It's pandas's `Series` object that has `count()` and `max()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(731556, 19)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_seq_size = train_seq.size()\n",
    "train_seq_size.count(), train_seq_size.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we know number of sequence and max length of each sequence. So we can make numpy array `X` and `y`. `X` has 731,556 sequences. Each sequence has up to 19 rows. Each row has 22 features. `y` has just one column for `Expected`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.zeros((731556, 19, 22))\n",
    "y = np.zeros((731556, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luis applied some preprocessing and feature extraction. Here, we just try to apply raw dataset to LSTM without any preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "731556\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "for name, group in train_seq:\n",
    "    # d.shape is (seq_length, 24)\n",
    "    d = group.values\n",
    "    # column 1~22 are features.\n",
    "    # column 0 is Id and column 23 is target.\n",
    "    # save 1~22 features to 0~21 index of dataset up to d.shape[0].\n",
    "    X[i, :d.shape[0], 0:22] = d[:, 1:23]\n",
    "    y[i, 0] = d[0, 23]\n",
    "    i += 1;\n",
    "print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split dataset to train set and test set. In fact test set is validation set for this task. Total dataset size is 731,556. Test set is sufficient 10% of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate `steps_per_epoch` and `validation_steps` for `fit_generator()` method. We set `batch_size` is 1024."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(643.0, 72.0)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 1024\n",
    "steps_per_epoch = np.ceil(X_train.shape[0] / batch_size)\n",
    "validation_steps = np.ceil(X_test.shape[0] / batch_size)\n",
    "steps_per_epoch, validation_steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model has three layers. `LSTM` layer has 50 units. It is just Luis choice. Final `Dense` layer has 1 unit without activation for `Expected` target. All other parameters of `LSTM` and `Dense` layer use default value. Loss is `'mae'` for regression task. We choose `'adam'` optimizer with default values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, LSTM, Bidirectional\n",
    "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "\n",
    "model = Sequential()\n",
    "model.add(LSTM(50, input_shape=(None, 22), return_sequences=True))\n",
    "model.add(LSTM(50))\n",
    "model.add(Dense(1))\n",
    "\n",
    "model.compile(loss='mae', optimizer='adam')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model training is simple. We just put the two generators and steps for epoch and validation. Let's try 150 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 493800 samples, validate on 164600 samples\n",
      "Epoch 1/150\n",
      "493800/493800 [==============================] - 42s 85us/step - loss: 23.2963 - val_loss: 23.1770\n",
      "Epoch 2/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.2118 - val_loss: 23.1109\n",
      "Epoch 3/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1849 - val_loss: 23.1646\n",
      "Epoch 4/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1757 - val_loss: 23.1062\n",
      "Epoch 5/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1713 - val_loss: 23.1158\n",
      "Epoch 6/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1618 - val_loss: 23.0918\n",
      "Epoch 7/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1564 - val_loss: 23.0897\n",
      "Epoch 8/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1573 - val_loss: 23.0747\n",
      "Epoch 9/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1489 - val_loss: 23.0772\n",
      "Epoch 10/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1484 - val_loss: 23.0792\n",
      "Epoch 11/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1462 - val_loss: 23.0741\n",
      "Epoch 12/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1386 - val_loss: 23.0949\n",
      "Epoch 13/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1352 - val_loss: 23.0888\n",
      "Epoch 14/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1343 - val_loss: 23.0835\n",
      "Epoch 15/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1309 - val_loss: 23.0896\n",
      "Epoch 16/150\n",
      "493800/493800 [==============================] - 40s 81us/step - loss: 23.1316 - val_loss: 23.1009\n"
     ]
    }
   ],
   "source": [
    "callbacks_list = [EarlyStopping(monitor='val_loss', patience=5), \n",
    "                  ModelCheckpoint(filepath='14_ex_8.h5', monitor='val_loss', save_best_only=True)]\n",
    "history = model.fit(X_train, y_train,\n",
    "                    epochs=150, batch_size=512,\n",
    "                    validation_split=0.25,\n",
    "                    callbacks=callbacks_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd81fXZ//HXpSzZCLiAEqwWGYYQI4Ioy1HUukAFRHFFiqtabW9tsdZxc9f1c6GtUhUHURy4intgcVAwIAQQEQdqACVQQRQQA9fvj88JhHBCzknOyTlJ3s/H4zzOOd/zHddhnOv72ebuiIiI7JLqAEREJD0oIYiICKCEICIiEUoIIiICKCGIiEiEEoKIiABKCCIiEqGEICIigBKCiIhE1Et1APFo06aNZ2RkpDoMEZEaZfbs2avcvW1F+9WohJCRkUF+fn6qwxARqVHM7MtY9lOVkYiIADEkBDPrYGbTzGyRmS00s0sj228wswIzm2tmr5nZPlGO7WhmsyP7LDSzMaU+O8jM5pvZp2Z2l5lZYr+aiIjEI5YSQjFwhbt3AXoDF5lZV+AWd8909yxgKnBNlGNXAIdG9jkEuKpU4vgHMBrYP/IYXLWvIiIiVVFhG4K7ryD8sOPu68xsEdDO3T8qtVsTYId5tN19U6m3DYkkIDPbG2ju7jMi7x8BTgJeruT3EJEk+PnnnyksLGTjxo2pDkVi0KhRI9q3b0/9+vUrdXxcjcpmlgH0BGZG3o8DRgFrgYHlHNMBeBHYD/ijuy83sxygsNRuhUC7OGMXkSQrLCykWbNmZGRkoFrd9OburF69msLCQjp16lSpc8TcqGxmTYEpwGXu/n0kgLHu3gHIAy4uJ8iv3T2TkBDOMrM9gWj/sqKu1GNmo80s38zyi4qKYg13q7w8yMiAXXYJz3l5cZ9CpM7auHEjrVu3VjKoAcyM1q1bV6k0F1NCMLP6hGSQ5+7PRNnlMWDozs7h7suBhcDhhBJB+1IftweWl3PcBHfPcfectm0r7Ea7nbw8GD0avvwS3MPz6NFKCiLxUDKoOar6dxVLLyMDHgAWufttpbbvX2q3E4CPoxzb3sx2i7xuBfQFFkfaJdaZWe/I+UcBz1fpm0QxdiysX7/9tvXrw3YREdleLCWEvsCZwKBI99G5ZnYscKOZLTCzAuBooKQ7ao6Z3R85tgsw08zmAf8GbnX3+ZHPLgDuBz4FPiMJDcpffRXfdhFJL6tXryYrK4usrCz22msv2rVrt/X9pk2bKj4BcM4557B48eKd7nPPPfeQl6Cqg8MOO4y5c+cm5FzVLZZeRu8Svc7/pXL2zwdyI69fBzJ3sl/3mCOthF/8IlQTRdsuIomXlxdK4F99Ff6fjRsHI0dW/nytW7fe+uN67bXX0rRpU/7whz9st4+74+7sskv0+9uJEydWeJ2LLrqo8kHWIrV6pPK4cdC48fbbGjcO20Uksaqzze7TTz+le/fujBkzhuzsbFasWMHo0aPJycmhW7duXH/99Vv3LbljLy4upmXLllx11VX06NGDPn36sHLlSgCuvvpq7rjjjq37X3XVVfTq1YvOnTvz/vvvA/Djjz8ydOhQevTowYgRI8jJyamwJDBp0iQOPPBAunfvzp///GcAiouLOfPMM7duv+uuuwC4/fbb6dq1Kz169OCMM85I+J9ZLGp1Qhg5EiZMgI4dwSw8T5hQtTsWEYmuutvsPvroI8477zw+/PBD2rVrx4033kh+fj7z5s3j9ddf56OPPtrhmLVr19K/f3/mzZtHnz59ePDBB6Oe292ZNWsWt9xyy9bkMn78ePbaay/mzZvHVVddxYcffrjT+AoLC7n66quZNm0aH374Ie+99x5Tp05l9uzZrFq1ivnz57NgwQJGjRoFwM0338zcuXOZN28ed999dxX/dCqnVicECD/+S5fCli3hWclAJDmqu83ul7/8JQcffPDW948//jjZ2dlkZ2ezaNGiqAlht91245hjjgHgoIMOYunSpVHPPWTIkB32effddxk+fDgAPXr0oFu3bjuNb+bMmQwaNIg2bdpQv359Tj/9dKZPn85+++3H4sWLufTSS3n11Vdp0aIFAN26deOMM84gLy+v0gPLqqrWJwQRqR7ltc0lq82uSZMmW18vWbKEO++8k7feeouCggIGDx4ctT9+gwYNtr7eddddKS4ujnruhg0b7rCPe9ShUuUqb//WrVtTUFDAYYcdxl133cVvf/tbAF599VXGjBnDrFmzyMnJYfPmzXFdLxGUEEQkIVLZZvf999/TrFkzmjdvzooVK3j11VcTfo3DDjuMJ598EoD58+dHLYGU1rt3b6ZNm8bq1aspLi5m8uTJ9O/fn6KiItydU089leuuu445c+awefNmCgsLGTRoELfccgtFRUWsL1v/Vg1q1HoIIpK+SqpjE9nLKFbZ2dl07dqV7t27s++++9K3b9+EX+OSSy5h1KhRZGZmkp2dTffu3bdW90TTvn17rr/+egYMGIC7c/zxx3PccccxZ84czjvvPNwdM+Omm26iuLiY008/nXXr1rFlyxauvPJKmjVrlvDvUBGLtxiUSjk5Oa4FckSqz6JFi+jSpUuqw0gLxcXFFBcX06hRI5YsWcLRRx/NkiVLqFcvve6ro/2dmdlsd8+p6Nj0+iYiImnqhx9+4IgjjqC4uBh357777ku7ZFBVtevbiIgkScuWLZk9e3aqw0gqNSqLiAighCAiIhFKCCIiAighiIhIhBKCiKStAQMG7DDI7I477uDCCy/c6XFNmzYFYPny5Zxyyinlnruibux33HHHdgPEjj32WNasWRNL6Dt17bXXcuutt1b5PImmhCAiaWvEiBFMnjx5u22TJ09mxIgRMR2/zz778PTTT1f6+mUTwksvvUTLli0rfb50p4QgImnrlFNOYerUqfz0008ALF26lOXLl3PYYYdtHReQnZ3NgQceyPPP77jo4tKlS+nePSy7smHDBoYPH05mZibDhg1jw4YNW/e74IILtk6d/de//hWAu+66i+XLlzNw4EAGDhwIQEZGBqtWrQLgtttuo3v37nTv3n3r1NlLly6lS5cunH/++XTr1o2jjz56u+tEM3fuXHr37k1mZiYnn3wy33333dbrd+3alczMzK2T6v373//eukBQz549WbduXaX/bKPROAQRic1ll0GiVwLLyoLIj2k0rVu3plevXrzyyiuceOKJTJ48mWHDhmFmNGrUiGeffZbmzZuzatUqevfuzQknnFDuusL/+Mc/aNy4MQUFBRQUFJCdnb31s3HjxrH77ruzefNmjjjiCAoKCvjd737HbbfdxrRp02jTps1255o9ezYTJ05k5syZuDuHHHII/fv3p1WrVixZsoTHH3+cf/7zn5x22mlMmTJlp+sbjBo1ivHjx9O/f3+uueYarrvuOu644w5uvPFGvvjiCxo2bLi1murWW2/lnnvuoW/fvvzwww80atQonj/tCqmEICJprXS1UenqInfnz3/+M5mZmRx55JEsW7aMb7/9ttzzTJ8+fesPc2ZmJpmZ2xZzfPLJJ8nOzqZnz54sXLiwwonr3n33XU4++WSaNGlC06ZNGTJkCO+88w4AnTp1IisrC9j5FNsQ1mdYs2YN/fv3B+Css85i+vTpW2McOXIkkyZN2joium/fvlx++eXcddddrFmzJuEjpVVCEJHY7OROPplOOukkLr/8cubMmcOGDRu23tnn5eVRVFTE7NmzqV+/PhkZGVGnvC4tWunhiy++4NZbb+WDDz6gVatWnH322RWeZ2dzwJVMnQ1h+uyKqozK8+KLLzJ9+nReeOEFbrjhBhYuXMhVV13Fcccdx0svvUTv3r154403OOCAAyp1/mhUQhCRtNa0aVMGDBjAueeeu11j8tq1a9ljjz2oX78+06ZN48toC6iX0q9fP/Ii63kuWLCAgoICIEyd3aRJE1q0aMG3337Lyy+/vPWYZs2aRa2n79evH8899xzr16/nxx9/5Nlnn+Xwww+P+7u1aNGCVq1abS1dPProo/Tv358tW7bw9ddfM3DgQG6++WbWrFnDDz/8wGeffcaBBx7IlVdeSU5ODh9//HHc19wZlRBEJO2NGDGCIUOGbNfjaOTIkRx//PHk5OSQlZVV4Z3yBRdcwDnnnENmZiZZWVn06tULCKuf9ezZk27duu0wdfbo0aM55phj2HvvvZk2bdrW7dnZ2Zx99tlbz5Gbm0vPnj13Wj1UnocffpgxY8awfv169t13XyZOnMjmzZs544wzWLt2Le7O73//e1q2bMlf/vIXpk2bxq677krXrl23rv6WKJr+WkTKpemva56qTH+tKiMREQGUEEREJEIJQUR2qiZVK9d1Vf27UkIQkXI1atSI1atXKynUAO7O6tWrqzRYTb2MRKRc7du3p7CwkKKiolSHIjFo1KgR7du3r/TxSggiUq769evTqVOnVIch1URVRiIiAighiIhIhBKCiIgASggiIhKhhCAiIoASgoiIRCghiIgIoIQgIiIRFSYEM+tgZtPMbJGZLTSzSyPbbzCzAjOba2avmdk+UY7NMrMZkeMKzGxYqc8eMrMvIsfPNbOsxH41ERGJRywlhGLgCnfvAvQGLjKzrsAt7p7p7lnAVOCaKMeuB0a5ezdgMHCHmbUs9fkf3T0r8kjw6t0iIhKPCqeucPcVwIrI63Vmtgho5+6lV6FuAuww+5W7f1Lq9XIzWwm0BdZUNXAREUmsuNoQzCwD6AnMjLwfZ2ZfAyOJXkIofWwvoAHwWanN4yJVSbebWcNyjhttZvlmlq8JtkREkifmhGBmTYEpwGXu/j2Au4919w5AHnDxTo7dG3gUOMfdt0Q2/wk4ADgY2B24Mtqx7j7B3XPcPadt27axhisiInGKKSGYWX1CMshz92ei7PIYMLScY5sDLwJXu/t/Sra7+woPfgImAr3iDV5ERBInll5GBjwALHL320pt37/UbicAH0c5tgHwLPCIuz9V5rO9S53/JGBBZb6AiIgkRizrIfQFzgTmm1lJT6A/A+eZWWdgC/AlMAbAzHKAMe6eC5wG9ANam9nZkWPPjvQoyjOztoABc0uOFxGR1LCatDReTk6O5+fnpzoMEZEaxcxmu3tORftppLKIiABKCCIiEqGEICIigBKCiIhEKCGIiAighCAiIhFKCCIiAighiIhIhBKCiIgASggiIhKhhCAiIoASgoiIRCghiIgIoIQgIiIRSggiIgIoIYiISIQSgoiIAEoIIiISoYQgIiKAEoKIiEQoIYiICKCEICIiEUoIIiICKCGIiEiEEoKIiABKCHHLy4OMDNhll/Ccl5fqiEREEqNeqgOoSfLyYPRoWL8+vP/yy/AeYOTI1MUlIpIIKiHEYezYbcmgxPr1YbuISE2nhBCHr76Kb7uISE2ihBCHX/wivu0iIjWJEkIcxo2Dxo2339a4cdguIlLTKSHEYeRImDABOnYEs/A8YYIalEWkdlAvoziNHKkEICK1k0oIIiICKCGIiEhEhQnBzDqY2TQzW2RmC83s0sj2G8yswMzmmtlrZrZPlGOzzGxG5LgCMxtW6rNOZjbTzJaY2RNm1iCxX01EROIRSwmhGLjC3bsAvYGLzKwrcIu7Z7p7FjAVuCbKseuBUe7eDRgM3GFmLSOf3QTc7u77A98B51Xxu4iISBVUmBDcfYW7z4m8XgcsAtq5+/eldmsCeJRjP3H3JZHXy4GVQFszM2AQ8HRk14eBk6ryRUREpGri6mVkZhlAT2Bm5P04YBSwFhhYwbG9gAbAZ0BrYI27F0c+LgTaxROLiIgkVsyNymbWFJgCXFZSOnD3se7eAcgDLt7JsXsDjwLnuPsWwKLstkMJI3LsaDPLN7P8oqKiWMMVEZE4xZQQzKw+IRnkufszUXZ5DBhazrHNgReBq939P5HNq4CWZlZSQmkPLI92vLtPcPccd89p27ZtLOGKiEglxNLLyIAHgEXuflup7fuX2u0E4OMoxzYAngUecfenSra7uwPTgFMim84Cnq/MFxARkcSIpYTQFzgTGBTpYjrXzI4FbjSzBWZWABwNlHRHzTGz+yPHngb0A84udWxW5LMrgcvN7FNCm8IDCfxeIiISJws36zVDTk6O5+fnpzoMEZEaxcxmu3tORftppLKIiABKCCIiEqGEICIigBKCiIhEKCGIiAighJAW8vIgIwN22SU85+WlOiIRqYu0YlqK5eXB6NGwfn14/+WX4T1oZTYRqV4qIaTY2LHbkkGJ9evDdhGR6qSEkGJffRXfdhGRZFFCSLFf/CK+7SIiyaKEkGLjxkHjxttva9w4bBcRqU5KCCk2ciRMmAAdO4JZeJ4wQQ3KIlL91MsoDYwcqQQgIqmnEoKIiAB1JSHMnQuvvprqKKqNBrqJSGXUjSqjK6+ETz8ND4u2nHPtoYFuIlJZdaOEMHw4fP45fPBBqiNJOg10E5HKqhsJ4eSToUEDmDw51ZEkXTIGuqkKSqRuqBsJoWVLOOYYeOIJ2LIl1dEkVaIHupVUQX35Jbhvq4JSUhCpfepGQoBQbbR8ObzzTqojSapED3RTFZRI3VF3EsLxx4dfxlpebZTogW6aa0mk7qg7CaFJEzjhBHj6afj551RHk1QjR8LSpaF2bOnSqvUu0lxLInVH3UkIEKqNVq2CN99MdSQ1huZaEqk76lZCGDwYWrSo9dVGiaS5lkTqDnP3VMcQs5ycHM/Pz6/aSc49F6ZMgW+/hUaNEhOYiEgaM7PZ7p5T0X51q4QAodro++/h5ZdTHcn2Zs2q9W0bIpLe6l5CGDQI2rZNr2qjGTPgkEPgzjtTHUm10EA3kfRU9xJCvXpw6qnwr3/BDz+kOprgrrvC8z//GUZ/1WIa6CaSvupeQoBQbbRhA7zwQqojgRUrQlfYffeFTz6Bd99NdURJpYFuIumrbiaEvn2hffv0qDa67z4oLoZnnoHmzeH++1MdUVJpoJtI+qqbCWGXXWDYMHjlFfjvf1MXx6ZNISEccwz06AGnnw5PPQVr1qQupiSrKQPd1M4hdVHdTAgAI0aEXj3PPpu6GKZMgW++gUsuCe9zc0NV1mOPpS6mJEvWQLdE/oCrnUPqqro3DqGEO/zqV+HX4/XXE3POeB16KBQVweLF4ZfMHbKzwwiwOXNSE1M1yMsLbQZffRVKBuPGVW2gW9lFgSAkmcoOoMvICEmgrI4dw1QgIjWNxiFUxCw0Lr/1VrhLr26zZ4fuphddFJJBSUy5ufDhh7U6ISRyriVIfEO12jmkrqq7CQFCtdGWLaGXT3UbPz5MuHfOOdtvP/30MIK6ljcuJ1Kif8BrSjuHSKLV7YTQtSsceCA8/nj1XreoKPRwGjUqzK1UWqtWYZxEXt6Ot70SVaJ/wDWhn9RVFSYEM+tgZtPMbJGZLTSzSyPbbzCzAjOba2avmdk+5Rz/ipmtMbOpZbY/ZGZfRI6fa2ZZiflKcRo+HN5/P3qlcbLcfz/89FOoLoomNzdMr5GKkksNlOgfcE3oJ3WWu+/0AewNZEdeNwM+AboCzUvt8zvg3nKOPwI4HphaZvtDwCkVXb/046CDDvKE++wzd3C/+ebEnzuan39279DBfdCg8vfZssV9//3dDz+8emKqBSZNcu/Y0d0sPE+alOqItpfu8UntBuR7DL+xFZYQ3H2Fu8+JvF4HLALaufv3pXZrAkTtruTubwLr4sxT1WfffaFXr+qrNnr+efj6621dTaMpaVx+553QA0kqlOiG6kRKVjdWjZWQRIurDcHMMoCewMzI+3Fm9jUwErimEtcfF6l2ut3MGpZzzdFmlm9m+UVFRZW4RAxGjAg9e6rjx/fuu0Pl9vHH73y/UaPCvEsPPJD8mCSpkjFdh8ZKSDLEnBDMrCkwBbispHTg7mPdvQOQB1wc57X/BBwAHAzsDlwZbSd3n+DuOe6e07Zt2zgvEaNTTw135U88kZzzl5g/H95+Gy68EHbddef77rVXSBoPPxxGNEuNlYxurMlKMipx1G0xJQQzq09IBnnu/kyUXR4DhsZz4UhVlLv7T8BEoFc8xydUu3bQr1+oNkrmQL277w5dSnNzY9s/NxdWroSpUyveV9JWMrqxJjrJqMQhEFsvIwMeABa5+22ltu9farcTgI/jubCZ7V3q/CcBC+I5PuFGjICPP4aCguSc/7vvYNKkMM6gdevYjvn1r0Oy0piEGi0Z3VgTnWQ0C61AbCWEvsCZwKBSXUSPBW40swVmVgAcDZR0R80xs62/YGb2DvAUcISZFZrZryMf5ZnZfGA+0Ab438R9rUoYOjRU4yRrBtSJE8P/sIvjqFnbddew5Ocrr4SGaKmRktGNNdFJRqOzBeryXEbRHHNMKCV8/nn4n5somzeHeZP23jv+9Q6WLg09oa69Fq6pTLu91FaJnBNK8zfVbprLqDJGjAj/+mfOTOx5X345JJmddTUtT0YGHHlk6G20eXNi45IaLZFdbTU6W0AJYXsnnggNGyZ+TMLdd8M++8CQIZU7Pjc33Aa+8UZi4xKJ0OhsASWE7bVoAcceC08+mbi78cWL4dVXYcwYqF+/cuc48cTQEK3GZUmiRA/uUzfWmkcJoawRI8J02NOnJ+Z899wTEsH551f+HA0bhoFqzz8fuqGKpDl1Y62ZlBDKOu44aNo0MdVG69bBQw/BaaeFgWZVkZsbVnh79NGqxyWSZOrGWjMpIZTVuHGoopkypeojhB95JCSFyjQml9W1a1hh7f77kzt4TiQBktWNNdHVUKrW2p4SQjTDh8N//1u1Rlz30Jh88MFwyCGJiSs3N3SLff/9xJxPJEmSMTo70dVQqtbakRJCNEcfHRaqqUq10RtvhB/veAaiVeTUU6FZMzUuS9pLRjfWRFdDqVprR0oI0TRoEEYuP/ccbNhQuXPcfTe0bQvDhiUurqZNQ6P3k0/C2rWJO69IgiWjG2uiq6GSUa1V46u0Ylk0IV0eSVkgpzxvvBEWznnqqfiP/fzzsBLK2LGJj2vWrBDXvfcm/twiaaxjx/BPv+yjY8f0ON+kSe6NG29/rsaNK78YUiLPR6IWyKmzBgyAPfes3NxGf/97SOljxiQ8LHJyIDNT1UZS5yS6GirR56sNVVpKCOXZddfQXfTFF8P6xrFavz5MM3HyydC+feLjKllNLT8f5s5N/PlF0lSiq6ESfb6aUKVVESWEnRk+HDZuDAPCYvXYY2Gq60R0NS3PyJFhsJpWU5M6JtGjqRN5vkT3rEpGT62KKCHsTJ8+4bYh1mojdxg/PlTpHH548uLafffQ6D1pUuUbvUUkodK9SisWSgg7YxZ6Cb32GqxeXfH+77wTFti55JLETp8dTW4urFkDz0RbwE5Eqlu6V2nFQushVOTDDyE7G+67L4xa2ZlTT4U334TCwh1Te6Jt2RLWWOjQAaZNS+61RKRG03oIiZKVBZ07V1xtVFgIzz4L552X/GQAoRfTeefB22/DkiXJv56I1HpKCBUxC43Lb78Ny5eXv9+994a79gsvrLbQOOus0BvqwQer75oiUmspIcRi+PDQYPzUU9E/37gxVO4dfzx06lR9ce2zT5iddeLEMBOqiEgVKCHE4oADQtVRedVGTz4JRUWJnbcoVuefD99+G8ZLiIhUgRJCrIYPh//8B774YsfP7r47JI0jj6z+uAYPDiUFjVwWkSpSQohVySR1Tzyx/faZM+GDD0LpINldTaOpVw/OOQdefjk0bIuIVJISQqwyMsJAtbLVRuPHhympR41KSVgAnHtuaNB+6KHUxSAiNZ4SQjyGD4d582DRovD+229D+8HZZ4ekkCr77gtHHBGmstiyJXVxiEiNpoQQj9NOC/3/S0oJEyaE3j2paEwuKzc3TMby1luJPa97OOfQodC/f2g8F5FaSQkhHnvtFabFnjw5JIJ774Vf/zqMGE61k04KcxwlqnH5++9DY3m3bqH08fbbMGtW+L5r1iTmGiKSVpQQ4jV8OHzyCfzlL2GgWjJnNY1Ho0Zw5plhtPSqVZU/z8KFcNFF0K5d+G5NmoS2icLCMG/SggVw7LHwww8JC11E0oMSQryGDg09e266KdTdDx6c6oi2Oe882LQpzIIaj59/hqefhoEDoXv30BYxZMi2HlRnnQW77QbHHBNKR7NmwYknhgF5IlJrKCHEa/fdQ7UJhDvpXXdNbTylHXggHHJIqDaKZdLCb76BG24IPahOPTWMsbjxxlAaePhh6NVrx2OGDAkjo996C045JSQgEakVlBAq4+KLw5oH55yT6kh2lJsbqn1mzoz+uTu8+y6MGBFW2rjmmlAqeP55+OwzuPJKaNNm59c480z4xz/C6OgzzoDNmxP/PUSk2ikhVMbgwaH7aatWqY5kR8OGhXr/so3LP/4I//xnmILj8MPDQLaLLoLFi+HVV+GEE+Ir7YwZA7feGuZ3ys1Vd1eRWqBeqgOQBGvWLDR8T54Mt98eqoX+/vdQzbN2bSjZTJgAp58eEkdVXHFFaFy+9tpwrvHjUzNaW0QSQgmhNsrNDQ3DhxwSBtHVqxfq+y+6CPr2TeyP9jXXwLp18P/+HzRtCn/7m5KCSA2lhFAbHXJIeBQWwvXXhxlR99orOdcyg1tuCVVSN90USihjxybnWiKSVEoItZEZvPdeeN6lGpqJzOCee0JSuPrqUH102WXJv66IJFSFvxZm1sHMppnZIjNbaGaXRrbfYGYFZjbXzF4zs33KOf4VM1tjZlPLbO9kZjPNbImZPWFmDRLzlQQIDcTVkQxK7LJLWLltyBD4/e81HbdIDRTLL0YxcIW7dwF6AxeZWVfgFnfPdPcsYCpwTTnH3wKcGWX7TcDt7r4/8B1wXtzRS3qpVw8efzz0who9OrwWkRqjwoTg7ivcfU7k9TpgEdDO3b8vtVsTIOpIKHd/E1hXepuZGTAIeDqy6WHgpLijl/TToAFMmQL9+oXxCs8/n+qIRGqu4uIw88Chh4b5xZIsrjoFM8sAegIzI+/HmdnXwEjKLyFE0xpY4+7FkfeFQLt4YpE01rgx/OtfkJMTZoh9/fVURyRSsxQXwyOPQNeu4cbqhx9g2bKkXzbmhGBmTYEpwGUlpQN3H+vuHYA8IJ45oKP1S4xawjCz0WaWb2b5RZp6ueZo1iwMfjsgbihvAAAL9UlEQVTggDDv0TvvpDoikfRXXBwmk+zSJcwh1qRJmFRy7tywLcliSghmVp+QDPLc/ZkouzwGDI3juquAlmZW0supPbA82o7uPsHdc9w9p23btnFcQlKuVSt47bUwRcZxx0F+fqojEklPP/8cOmV07hymxGnWDJ57DubMgZNPrrYOIrH0MjLgAWCRu99Wavv+pXY7Afg41ou6uwPTgFMim84CVNlcG+25J7zxBrRuHSYFnD8/1RGJpI9Nm0KPvM6dw2zFrVrBCy/A7NmhZF3NgzxjSTt9Cb2EBkW6mM41s2OBG81sgZkVAEcDJd1Rc8xsa59DM3sHeAo4wswKzSwyVShXApeb2aeENoUHEve1JK20bw9vvhnWbDjqqLCehNQNa9fC9OlaP6OsTZvgvvvC4lrnnx8mlJw6NUw3f/zxKRvtbx7LNMlpIicnx/NV7VBzLVoUeh/ttltoU+jYMdURSTJ9+SUcfXS4AahfH/r0CTcERx0VOhyk09Tx1eWnn0LV0N/+Bl9/HWYU+OtfQ1ftJCYBM5vt7jkV7afZTqX6dOkSehytWxeW5VyxItURSbIsXBi6Sn77bZhX6/e/D6WEv/wFevcOd8RDhoRp1D/9NLb1O2qyjRvDaP799oMLLwyl5ldegRkzwsJTaTL/l6aukOqVlRV6Hx15ZFihbdiw0M6w117bPzdtmupIpbLefx9+85tQRfjOO2HhphKrVoXqw9dfD49nnw3bMzLCv4mjjgo3C61bpyT0hNu4MUw7f9NNodvoYYeFmYePOCJtkkBpqjKS1Hj77dCt7uuvo98dNmmyfYKIljRKnhs3rvbwpRwvvhhW32vfPvQwy8gof1/3UDooSQ5vvRUGX5lBdnZIDkceGWbobdSo2r5CQmzYEKaZv+mmUBLu1y9UDQ0cmJJEEGuVkRKCpFZxcbhr/OabUL2ws+fVq6Ofo1mzbQli5MgwbUZ1zuMkwcMPh54yWVnw0kuwxx7xHV9cHLomlySIGTPCtt12C4s6lbQ/HHhgev79/vRTiPn110M7wTffQP/+Yb2QAQNSGpoSgtQ+P/8MK1eWnzAWLw4DeAYMCF35fvnLVEdcd9xyC/zP/4Q7+meeCUm6qtatCz2UShLERx+F7W3bhqqXvn3DIzs7TJlS3dxhwYJt8U2fDuvXh8byQYPCNPD9+1d/XFEoIUjd4x4aMK+4ItxZ/t//wSWXpOfdZG3hHhLBrbeGaUoeeQQaNkzOtZYtC+0Pb74Z1gX//POwvVEjOPjgbQni0ENh992TF8Mbb4QE8MYb4WYEwoj8kjaQAQOgefPkXL+SlBCk7ioshN/+NlRb9O0bkkTnzqmOqvb5+efQh/7hh8NqfHfeWb1dSb/5Jqz78f774XnOnBAThB5tJcmhb1/Yf//K1d2vWwf//ve2UsCiRWH7HnuEBFDy6NAhcd8rCZQQpG5zh0cfhUsvDT09rr8eLr+8bvZ9T4b160OJ4MUX4brrQnfSVPea2bAhDOx6771tieK778JnbdtuSw59+8JBB0UvyRQXh3OUJID//GdbO0a/fttKAenajlEOJQQRCD08LrggTMPdq1fo8te1a6qjqtn++98wmnbGDPj732HMmFRHFN2WLfDxx9sniCVLwmcNG4bBcX37hgFzy5aFBDBt2raeTgcdtC0BHHpozevpVIoSgkgJd3jiCbj44lAF8Ne/wh//GEbPSnwKC8Oo2iVL4LHHYGg8c1qmgZUrt1UxvfdemDNo06bwWadO27q6DhpUe8ZCoIQgsqOVK0NSeOqp0DPlwQehR49UR1VzLF4cpqL47rtQ4ho4MNURVd3GjfDhh6FNoBb3StPUFSJl7bEHPPkkPP10uNPNyQl9xEvuEKV8H3wQunpu3BgaWWtDMoBQDdSnT61OBvFQQpC6Z+jQ0Kd92LDQIHrwwaHqoCbYvDmUdBYuDPXdU6bAvHlhe7K89lpIAM2bh2qWnj2Tdy1JKc1lJHVT69Zhrdphw0Kj6CGHhP7011xTvY2H7qEKpqgo/NCvXLnz16tWRZ/qo2nT0Gjep0949O6dmDrwxx8PU4x07RrmoNp776qfU9KW2hBE1qwJXVInTgz91ydODAmiKrZsCT/gX38dHoWF4XnZsu1/6IuKQrfGaFq1CtVcbduG52ivW7QIpZ0ZM8KjoGBbaeFXv9qWIPr0gW7d4ut2O3586LZ7+OFh0ZYWLar2ZyIpo0ZlkXi98kqYB2nZsjBd8w03hP7nZbmHO/XSP/RlXy9btmPbRMOGsM8+Yd6l8n7gS163aVO5XlA//hjq+2fMCH3oZ8wISQfCdBJlSxHRRvS6h5LS//4vnHRSKCXU4C6XooQgUjnffx+qju67L4xuvfDCHe/0CwtD42pp9euHGT7btw+jVjt02PF127bVP3jLPUzxUFKCKFuK6Nx5+1JE585huo8JEyA3N6xXUE81yzWdEoJIVbz5ZvhBXLo0VLO0a1f+D32HDuGuvqaMXC1diih5rFoVPmvQIJRsxo4NJaRUjz6WhFBCEKmqTZvCD+Wee9buKS/c4bPPQmKYNStUK515ZqqjkgSKNSGoLChSngYNQp1/bWcWlnbcbz8lgjquhpRxRUQk2ZQQREQEUEIQEZEIJQQREQGUEEREJEIJQUREACUEERGJUEIQERGgho1UNrMi4MtUxxHRBliV6iAqkO4xpnt8oBgTId3jg/SPsarxdXT3thXtVKMSQjoxs/xYhoKnUrrHmO7xgWJMhHSPD9I/xuqKT1VGIiICKCGIiEiEEkLlTUh1ADFI9xjTPT5QjImQ7vFB+sdYLfGpDUFERACVEEREJEIJIU5m1sHMppnZIjNbaGaXpjqmaMxsVzP70MympjqWaMyspZk9bWYfR/4s+6Q6prLM7PeRv+MFZva4maV0YWEze9DMVprZglLbdjez181sSeS5VRrGeEvk77nAzJ41s5bpFmOpz/5gZm5mbVIRWySGqPGZ2SVmtjjyb/LmZFxbCSF+xcAV7t4F6A1cZGZdUxxTNJcCi1IdxE7cCbzi7gcAPUizWM2sHfA7IMfduwO7AsNTGxUPAYPLbLsKeNPd9wfejLxPpYfYMcbXge7ungl8AvypuoMq4yF2jBEz6wAcBXxV3QGV8RBl4jOzgcCJQKa7dwNuTcaFlRDi5O4r3H1O5PU6wg9Zu9RGtT0zaw8cB9yf6liiMbPmQD/gAQB33+Tua1IbVVT1gN3MrB7QGFieymDcfTrw3zKbTwQejrx+GDipWoMqI1qM7v6auxdH3v4HaF/tgW0fT7Q/R4Dbgf8BUtqwWk58FwA3uvtPkX1WJuPaSghVYGYZQE9gZmoj2cEdhH/YW1IdSDn2BYqAiZFqrfvNrEmqgyrN3ZcR7sK+AlYAa939tdRGFdWe7r4Cws0KsEeK46nIucDLqQ6iLDM7AVjm7vNSHUs5fgUcbmYzzezfZnZwMi6ihFBJZtYUmAJc5u7fpzqeEmb2G2Clu89OdSw7UQ/IBv7h7j2BH0l9Vcd2InXxJwKdgH2AJmZ2RmqjqtnMbCyhyjUv1bGUZmaNgbHANamOZSfqAa0I1dR/BJ40M0v0RZQQKsHM6hOSQZ67P5PqeMroC5xgZkuBycAgM5uU2pB2UAgUuntJyeppQoJIJ0cCX7h7kbv/DDwDHJrimKL51sz2Bog8J6UqoarM7CzgN8BIT7++7r8kJP55kf837YE5ZrZXSqPaXiHwjAezCKX/hDd8KyHEKZKVHwAWufttqY6nLHf/k7u3d/cMQiPoW+6eVne27v4N8LWZdY5sOgL4KIUhRfMV0NvMGkf+zo8gzRq+I14Azoq8Pgt4PoWxRGVmg4ErgRPcfX2q4ynL3ee7+x7unhH5f1MIZEf+naaL54BBAGb2K6ABSZiMTwkhfn2BMwl33nMjj2NTHVQNdAmQZ2YFQBbwfymOZzuR0svTwBxgPuH/SkpHs5rZ48AMoLOZFZrZecCNwFFmtoTQQ+bGNIzxbqAZ8Hrk/8u9aRhj2ignvgeBfSNdUScDZyWjpKWRyiIiAqiEICIiEUoIIiICKCGIiEiEEoKIiABKCCIiEqGEICIigBKCiIhEKCGIiAgA/x/EQBstugsR3wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "epochs = range(1, len(loss) + 1)\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'r', label='Validation loss')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_weights('14_ex_8.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can make prediction for test data. Read the test.csv file and fill `NaN` with 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "rain2_test = pd.read_csv('datasets/rain2/test.csv')\n",
    "test_df = rain2_test.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test data has 23 columns not 24 columns. It doesn't have `Expected` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>minutes_past</th>\n",
       "      <th>radardist_km</th>\n",
       "      <th>Ref</th>\n",
       "      <th>Ref_5x5_10th</th>\n",
       "      <th>Ref_5x5_50th</th>\n",
       "      <th>Ref_5x5_90th</th>\n",
       "      <th>RefComposite</th>\n",
       "      <th>RefComposite_5x5_10th</th>\n",
       "      <th>RefComposite_5x5_50th</th>\n",
       "      <th>...</th>\n",
       "      <th>RhoHV_5x5_50th</th>\n",
       "      <th>RhoHV_5x5_90th</th>\n",
       "      <th>Zdr</th>\n",
       "      <th>Zdr_5x5_10th</th>\n",
       "      <th>Zdr_5x5_50th</th>\n",
       "      <th>Zdr_5x5_90th</th>\n",
       "      <th>Kdp</th>\n",
       "      <th>Kdp_5x5_10th</th>\n",
       "      <th>Kdp_5x5_50th</th>\n",
       "      <th>Kdp_5x5_90th</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>11.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>14.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>8.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>15.5</td>\n",
       "      <td>13.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  minutes_past  radardist_km   Ref  Ref_5x5_10th  Ref_5x5_50th  \\\n",
       "0   1             1           8.0   0.0           0.0           0.0   \n",
       "1   1             5           8.0  10.0           0.0          10.0   \n",
       "2   1             8           8.0   0.0           0.0           7.0   \n",
       "3   1            12           8.0  14.0           0.0           9.0   \n",
       "4   1            15           8.0  10.5           0.0           9.0   \n",
       "\n",
       "   Ref_5x5_90th  RefComposite  RefComposite_5x5_10th  RefComposite_5x5_50th  \\\n",
       "0          14.0           0.0                    0.0                    0.0   \n",
       "1          18.0          11.5                    0.0                   11.0   \n",
       "2          14.5           0.0                    0.0                    7.0   \n",
       "3          16.0          14.0                    0.0                    9.0   \n",
       "4          15.5          13.5                    0.0                    9.0   \n",
       "\n",
       "       ...       RhoHV_5x5_50th  RhoHV_5x5_90th  Zdr  Zdr_5x5_10th  \\\n",
       "0      ...                  0.0             0.0  0.0           0.0   \n",
       "1      ...                  0.0             0.0  0.0           0.0   \n",
       "2      ...                  0.0             0.0  0.0           0.0   \n",
       "3      ...                  0.0             0.0  0.0           0.0   \n",
       "4      ...                  0.0             0.0  0.0           0.0   \n",
       "\n",
       "   Zdr_5x5_50th  Zdr_5x5_90th  Kdp  Kdp_5x5_10th  Kdp_5x5_50th  Kdp_5x5_90th  \n",
       "0           0.0           0.0  0.0           0.0           0.0           0.0  \n",
       "1           0.0           0.0  0.0           0.0           0.0           0.0  \n",
       "2           0.0           0.0  0.0           0.0           0.0           0.0  \n",
       "3           0.0           0.0  0.0           0.0           0.0           0.0  \n",
       "4           0.0           0.0  0.0           0.0           0.0           0.0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. Let's check the number of sequences and maximum length just like above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(717625, 19)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_seq = test_df.groupby(['Id'])\n",
    "test_seq_size = test_seq.size()\n",
    "test_seq_size.count(), test_seq_size.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create `X_text` numpy array. Its size is (717625, 19, 22)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = np.zeros((717625, 19, 22))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like train data, copy `test_seq` to `X_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "717625\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "for name, group in test_seq:\n",
    "    # d.shape is (seq_length, 23)\n",
    "    d = group.values\n",
    "    # column 1~22 are features.\n",
    "    # save 1~22 features to 0~21 index of dataset up to d.shape[0].\n",
    "    X_test[i, :d.shape[0], 0:22] = d[:, 1:23]\n",
    "    i += 1;\n",
    "print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can feed `X_test` to our model and get the predictions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making submission file, we stack `pred` array and its index horizontally. Using `numpy.savetxt`, we can make csv file with format(\"int, float\") and title(\"Id,Expected\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_with_index = np.hstack((np.arange(1, pred.shape[0]+1).reshape(-1,1), pred))\n",
    "np.savetxt(\"datasets/rain2/test_prediction.csv\", pred_with_index, \"%d,%f\", \n",
    "           delimiter=\",\", header=\"Id,Expected\", comments=\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Id,Expected\r\n",
      "1,0.473683\r\n",
      "2,0.250590\r\n",
      "3,2.757972\r\n",
      "4,5.341240\r\n",
      "5,0.266192\r\n",
      "6,0.785535\r\n",
      "7,2.634934\r\n",
      "8,0.696415\r\n",
      "9,0.169597\r\n"
     ]
    }
   ],
   "source": [
    "!head \"datasets/rain2/test_prediction.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you submit this file to Kaggle competition(https://www.kaggle.com/c/how-much-did-it-rain-ii/submit), you will get about 23.78 score. It may be ranked before 100. Cool!\n",
    "\n",
    "You can stack another `LSTM` layer and change weight initializer or optimizer. If you want to search more parameters, it's better to use `keras.wrappers.scikit_learn.KerasClassifier` with scikit-learn's `GridSearchCV`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}