Day #22 fastai_tabular_starter_code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/skyprince999/100-Days-Of-ML/blob/master/Day_22_fastai_tabular_starter_code.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ecQkBRa--9er"
   },
   "source": [
    "# Context\n",
    "\n",
    "Traditionally we transfer categorical variables to one-hot encoding. The disadvantage is that this assumes independence among entities & creates a sparse matrix. This can lead to out-of-memory (OOM) while training. \n",
    "\n",
    "Entity embeddings instead use a vector to represent each entity. So each categorical variable is represented as a vector of floating points. Embeddings can does capture richer relationships & complexities. \n",
    "\n",
    "\n",
    "**Related paper**\n",
    "(Entity Embeddings of Categorical Variables)[https://arxiv.org/abs/1604.06737] \n",
    "Rachal Thomas' blog on Fast.ai (Categorical Embeddings)[https://www.fast.ai/2018/04/29/categorical-embeddings/]\n",
    "\n",
    "# Process flow\n",
    "\n",
    ">> I am running this on CPU, but you can add a GPU for faster training (*around 3x speedup*)\n",
    "\n",
    "1. Feature engineering is done for each store (store_id). Features are similar to this (notebook by kkiler)[https://www.kaggle.com/kneroma/m5-first-public-notebook-under-0-50] \n",
    "2. `TabularList` is used to create a DataBunch for store_id = 1\n",
    "3. Model is trained for a single store\n",
    "4. Separate models are trained for each store (*Not shown here. Models are stored in a public dataset*)\n",
    "5. Test data is processed\n",
    "6. (Trained models)[https://www.kaggle.com/skylord/m5-forecasting-models] are used to then predict for each store in the test data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "colab": {},
    "colab_type": "code",
    "id": "DnxFg_2t-9ex"
   },
   "outputs": [],
   "source": [
    "from  datetime import datetime, timedelta\n",
    "import time\n",
    "import gc\n",
    "import numpy as np, pandas as pd\n",
    "import gc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3CbPuSuw-9fB"
   },
   "outputs": [],
   "source": [
    "from fastai import *      # import * is considered as a bad coding practice! will have to change this!\n",
    "from fastai.tabular import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "colab": {},
    "colab_type": "code",
    "id": "jnxV9e7I-9fL"
   },
   "outputs": [],
   "source": [
    "start_nb = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "50tR6UyJ-9fW"
   },
   "outputs": [],
   "source": [
    "CAL_DTYPES={\"event_name_1\": \"category\", \"event_name_2\": \"category\", \"event_type_1\": \"category\", \n",
    "         \"event_type_2\": \"category\", \"weekday\": \"category\", 'wm_yr_wk': 'int16', \"wday\": \"int16\",\n",
    "        \"month\": \"int16\", \"year\": \"int16\", \"snap_CA\": \"float32\", 'snap_TX': 'float32', 'snap_WI': 'float32' }\n",
    "PRICE_DTYPES = {\"store_id\": \"category\", \"item_id\": \"category\", \"wm_yr_wk\": \"int16\",\"sell_price\":\"float32\" }\n",
    "\n",
    "pd.options.display.max_columns = 50\n",
    "h = 28 \n",
    "max_lags = 70\n",
    "tr_last = 1913\n",
    "fday = datetime(2016,4, 25) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sh-19Bul-9ff"
   },
   "source": [
    "# Define helper functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Z2IqG7tJ-9fh"
   },
   "outputs": [],
   "source": [
    "def create_dt(is_train = True, nrows = None, first_day = 1200, store_id = None):\n",
    "    \n",
    "    start = time.time()\n",
    "    if store_id == None and is_train:\n",
    "        print(\"ERROR: No store_id provided.Please provide an id [0-9]\")\n",
    "        return None\n",
    "        \n",
    "    prices = pd.read_csv(\"../input/m5-forecasting-accuracy/sell_prices.csv\", dtype = PRICE_DTYPES)\n",
    "    \n",
    "    \n",
    "    for col, col_dtype in PRICE_DTYPES.items():\n",
    "        if col_dtype == \"category\":\n",
    "            prices[col] = prices[col].cat.codes.astype(\"int16\")\n",
    "            prices[col] -= prices[col].min()\n",
    "            \n",
    "    cal = pd.read_csv(\"../input/m5-forecasting-accuracy/calendar.csv\", dtype = CAL_DTYPES)\n",
    "    cal[\"date\"] = pd.to_datetime(cal[\"date\"])\n",
    "    for col, col_dtype in CAL_DTYPES.items():\n",
    "        if col_dtype == \"category\":\n",
    "            cal[col] = cal[col].cat.codes.astype(\"int16\")\n",
    "            cal[col] -= cal[col].min()\n",
    "    \n",
    "    # Filter out the values for store_id\n",
    "    if is_train:\n",
    "        prices = prices[prices['store_id'] == store_id]\n",
    "        \n",
    "    print(f\"Shape of Store - {store_id} dataframe : \", prices.shape)\n",
    "    \n",
    "    \n",
    "    start_day = max(1 if is_train  else tr_last-max_lags, first_day)\n",
    "    numcols = [f\"d_{day}\" for day in range(start_day,tr_last+1)]\n",
    "    catcols = ['id', 'item_id', 'dept_id','store_id', 'cat_id', 'state_id']\n",
    "    dtype = {numcol:\"float32\" for numcol in numcols} \n",
    "    dtype.update({col: \"category\" for col in catcols if col != \"id\"})\n",
    "    dt = pd.read_csv(\"../input/m5-forecasting-accuracy/sales_train_validation.csv\", \n",
    "                     nrows = nrows, usecols = catcols + numcols, dtype = dtype)\n",
    "    \n",
    "    for col in catcols:\n",
    "        if col != \"id\":\n",
    "            dt[col] = dt[col].cat.codes.astype(\"int16\")\n",
    "            dt[col] -= dt[col].min()\n",
    "    \n",
    "    if not is_train:\n",
    "        for day in range(tr_last+1, tr_last+ 28 +1):\n",
    "            dt[f\"d_{day}\"] = np.nan\n",
    "    \n",
    "    dt = pd.melt(dt,\n",
    "                  id_vars = catcols,\n",
    "                  value_vars = [col for col in dt.columns if col.startswith(\"d_\")],\n",
    "                  var_name = \"d\",\n",
    "                  value_name = \"sales\")\n",
    "    \n",
    "    dt = dt.merge(cal, on= \"d\", copy = False)\n",
    "    dt = dt.merge(prices, on = [\"store_id\", \"item_id\", \"wm_yr_wk\"], copy = False)\n",
    "    end = time.time()\n",
    "    \n",
    "    print(\"Processing time: \", (end-start))\n",
    "    return dt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GeU0DCFX-9fs"
   },
   "outputs": [],
   "source": [
    "def create_fea(dt):\n",
    "    start = time.time()\n",
    "    lags = [7, 28]\n",
    "    lag_cols = [f\"lag_{lag}\" for lag in lags ]\n",
    "    for lag, lag_col in zip(lags, lag_cols):\n",
    "        dt[lag_col] = dt[[\"id\",\"sales\"]].groupby(\"id\")[\"sales\"].shift(lag)\n",
    "\n",
    "    wins = [7, 28]\n",
    "    for win in wins :\n",
    "        for lag,lag_col in zip(lags, lag_cols):\n",
    "            dt[f\"rmean_{lag}_{win}\"] = dt[[\"id\", lag_col]].groupby(\"id\")[lag_col].transform(lambda x : x.rolling(win).mean())\n",
    "   \n",
    "    \n",
    "    date_features = {\n",
    "        \n",
    "        \"wday\": \"weekday\",\n",
    "        \"week\": \"weekofyear\",\n",
    "        \"month\": \"month\",\n",
    "        \"quarter\": \"quarter\",\n",
    "        \"year\": \"year\",\n",
    "        \"mday\": \"day\",\n",
    "#         \"ime\": \"is_month_end\",\n",
    "#         \"ims\": \"is_month_start\",\n",
    "    }\n",
    "    \n",
    "#     dt.drop([\"d\", \"wm_yr_wk\", \"weekday\"], axis=1, inplace = True)\n",
    "    \n",
    "    for date_feat_name, date_feat_func in date_features.items():\n",
    "        if date_feat_name in dt.columns:\n",
    "            dt[date_feat_name] = dt[date_feat_name].astype(\"int16\")\n",
    "        else:\n",
    "            dt[date_feat_name] = getattr(dt[\"date\"].dt, date_feat_func).astype(\"int16\")\n",
    "    \n",
    "    # Drop NA values\n",
    "    dt.dropna(inplace = True)     \n",
    "    \n",
    "    # Sort the dataframe on 'saledate' so we can easily create a validation set that data is in the 'future' of what's in the training set\n",
    "    dt = dt.sort_values(by='date', ascending=False)\n",
    "    dt = dt.reset_index(drop=True)\n",
    "    end = time.time()\n",
    "    print(\"Processing time: \", (end- start))\n",
    "    \n",
    "    return dt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "za_DxRms-9f1"
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "\n",
    "def checkDiskSpace():\n",
    "    total, used, free = shutil.disk_usage(\"/\")\n",
    "    \n",
    "    free = (free // (2**30))\n",
    "    \n",
    "    if free < 10:\n",
    "        return -1\n",
    "    else:\n",
    "        return 0 \n",
    "         \n",
    "\n",
    "# print(\"Total: %d GiB\" % (total // (2**30)))\n",
    "# print(\"Used: %d GiB\" % (used // (2**30)))\n",
    "# print(\"Free: %d GiB\" % (free // (2**30)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sK0MY8Pb-9gD"
   },
   "source": [
    "# Pre-processing of train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UBDuS79Y-9gF"
   },
   "outputs": [],
   "source": [
    "cat_feats = ['item_id', 'dept_id','store_id', 'cat_id', 'state_id'] + [\"event_name_1\", \"event_name_2\", \"event_type_1\", \"event_type_2\"]\n",
    "useless_cols = [\"id\", \"date\", \"sales\",\"d\", \"wm_yr_wk\", \"weekday\"]\n",
    "#train_cols = df.columns[~df.columns.isin(useless_cols)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "y7Hq9FOO-9gP"
   },
   "outputs": [],
   "source": [
    "FIRST_DAY = 1 # If you want to load all the data set it to '1' -->  Great  memory overflow  risk !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PhOAzMyr-9gY",
    "outputId": "1f91e83c-f62d-4514-88cc-4c9f3343b86e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of Store - 1 dataframe :  (637395, 4)\n",
      "Processing time:  62.7872998714447\n",
      "Processing time:  25.003514289855957\n",
      "(4108081, 31)\n",
      "2011-03-25 00:00:00 2016-04-24 00:00:00\n"
     ]
    }
   ],
   "source": [
    "idx = 1\n",
    "train_df = create_dt(is_train=True, first_day= FIRST_DAY, store_id = idx)\n",
    "train_df = create_fea(train_df)\n",
    "print(train_df.shape)\n",
    "print(train_df['date'].min(), train_df['date'].max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VuIlKcG1-9gn"
   },
   "outputs": [],
   "source": [
    "# Sort by date (used for train/validation splits)\n",
    "train_df.sort_values(by='date', inplace=True)\n",
    "\n",
    "# convert sales value to log scale\n",
    "train_df['sales'] = np.log(train_df['sales'] + 1)  # Taking logarithm values for sales\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "I8x-omXA-9gy",
    "outputId": "9c23b74e-a267-4083-f7ab-4ebc34a0425a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "64020\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Calculate where we should cut the validation set. We pick the most recent 'n' records in training set \n",
    "# where n is the number of entries in test set. \n",
    "\n",
    "cut = train_df['date'][(train_df['date'] == train_df['date'][62500])].index.max()\n",
    "print(cut)\n",
    "valid_idx = range(cut)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "M5P6q3j--9g7"
   },
   "outputs": [],
   "source": [
    "# Define categorical, continous & dependent variables\n",
    "\n",
    "cat_vars = ['wm_yr_wk', 'weekday', 'wday', 'month', 'year', 'event_name_1',\n",
    "           'event_type_1', 'event_name_2', 'event_type_2', 'snap_CA', 'snap_TX',\n",
    "           'snap_WI', 'week', 'quarter', 'mday']\n",
    "\n",
    "cont_vars = ['lag_7', 'lag_28', 'rmean_7_7', 'rmean_28_7',\n",
    "           'rmean_7_28', 'rmean_28_28', 'sell_price']\n",
    "\n",
    "dep_var = 'sales'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "y0vFmAly-9hD",
    "outputId": "a3830822-3a09-410d-b840-06984447a444"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0000, 6.5152])\n"
     ]
    }
   ],
   "source": [
    "# We want to limit the price range for our prediction to be within the history sale price range, so we need to calculate the y_range\n",
    "# Note that we multiplied the maximum of 'SalePrice' by 1.2 so when we apply sigmoid, the upper limit will also be covered. \n",
    "max_y = np.max(train_df['sales'])*1.2\n",
    "y_range = torch.tensor([0, max_y], device=defaults.device)\n",
    "print(y_range)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b5MB2m-m-9hM"
   },
   "source": [
    "# Model training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OWi6dVKl-9hO"
   },
   "outputs": [],
   "source": [
    "\n",
    "# Defining pre-processing we want for our fast.ai DataBunch\n",
    "procs=[FillMissing, Categorify, Normalize]\n",
    "\n",
    "# Use fast.ai datablock api to put our training data into the DataBunch, getting ready for training\n",
    "data = (TabularList.from_df(train_df, cat_names=cat_vars, cont_names=cont_vars, procs=procs)\n",
    "                       .split_by_idx(valid_idx)\n",
    "                       .label_from_df(dep_var)\n",
    "                       .databunch())\n",
    "\n",
    "\n",
    "# Create our tabular learner. The dense layer is 1000 and 500 two layer NN. We used dropout, hai \n",
    "learn = tabular_learner(data, layers=[512,256, 128], ps=[0.05,0.01, 0.5], emb_drop=0.04, \n",
    "                            y_range=y_range, metrics=rmse)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Uw7PzG3h-9hX",
    "outputId": "be7b0fc6-53b4-4427-e84d-8a244ad7a7ce"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='0' class='' max='1' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      0.00% [0/1 00:00<00:00]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>root_mean_squared_error</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='99' class='' max='63188' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      0.16% [99/63188 00:04<49:28 3.9863]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "coT6T2Ib-9hh",
    "outputId": "ec750321-0153-4537-a890-31c4902202a9"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3xV9f3H8dcnExJCIIMVdlgim4ACdYPzByoKIu5RF62ztrW2tdo66y5WxYHYKg5EK9Y6akVEQfYS2SthJkDCSELW9/fHvWjEBALcm3Pvzfv5eNwH95577j1vLpd3Tr733O8x5xwiIhJ5orwOICIiwaGCFxGJUCp4EZEIpYIXEYlQKngRkQgV43WAytLS0lzbtm29jiEiEjbmzp2b55xLr+q+kCr4tm3bMmfOHK9jiIiEDTNbX919GqIREYlQKngRkQilghcRiVAqeBGRCKWCFxGJUCp4EZEIpYIXEYlQIXUcvPiUVzjW5u1hycZdbMwvoqLCUe4czkHPVsmc0DGd2Ojg/2wuLi3ng0WbSW0Qx6DMNOJitD8gEk5U8CGitLyCDxZtYuI32SzeWEBRaXm166YkxvF/PZoztGcLerZsFPDidc7xwaLNPPzRMnJ2FgHQsF4MQ7o24+TO6STGR2NmGNAuLZE2qYkB3b6IBIYK3mOFJWW8NTubF75cy8b8IjLTE7moXyu6ZSTTLaMhbVISiYk2os0oq3B8sSKX9xZs5M3Z2bw6Yz1x0VF0bdGQXq0a0a9tCoM6pNIoIe4n2ykrr2B+dj7TVuQybWUeubuKKSl3lFVUUFHhaNGoPq1SEmidksC8DTuZvyGfLs2SmHB1f8rKK/hw8RY+XbqFd+bl/OS5+7VtzIi+rTi7R3MaxOstJRIqLJTO6JSVleUiaaqCfWXlLMwuYFFOPpnpDTiufQoJcb4C3FxQxKsz1jNx1gbyC0vp17YxN5yUySmdmxAVZYd87t3FpXy5Mo/5G3ayIDufxRsLKC6twAy6ZyQzoH0qZRWOzQVFbC4oZtW2PewuLiPKoHfrxrRLSyQ22oiNjsI5X54NOwrZsKOQ5Pqx3DGkMxf0bUl0pSwlZRWs2Lqb8gpHhXOUVzhmrdvBpDk5rMnbS0JcNKP6teaGk9rTpGG9oL2uIvIDM5vrnMuq8j4VfGDtLi7l7Tk5fLp0K/M27GRfWcX398VFR9GvXWOS4mP59LutOOc4vWszrj2hHVltU45qu6XlFSzKyefLlXl8uTKPBdn5xEVH0bxRPZon16NNaiI/65DGoMw0khNiq32e/e8Hs0P/kKn8mHkbdvLazA38a+EmoqOM0f1bc/1J7WmeXP+o/l4icnAq+FqwMb+IV75ayxuzstm9r4xjmjdkQPtUjmufQq9WjVixdbdveGRFHtt2F3Nh35ZcPqAtrVISgpKntLyCmCg7rKIOhHV5e/n71FVMnreRsgpHl2ZJHN8+lQGZqfysQxqJGsIRCSgV/FFyzjF1eS55e/ZxXu+MHx3BUl7hePqzlYz9fBUA53RvzjU/a0fPVo28ihsSsncU8v7CTcxYvZ0563dQXFpBUr0YLu7fmisGtiWjkfbsRQJBBX8UZq7Zzl8/Xs7c9TsB6NIsiQeGd6dP68Zs213MLRMXMGPNds7r1YI7z+yi4qrCvrJy5q7fyevfbOA/S7YAcFa3ZlzUrxUDM9N+NM4vIoenThf85oIiCkvKyUxv8JP7/rt0K2/NyaakvOL7Dw6jzIiPiSI2Oorte0qYtW4HzRrW45bBHWmcEMe9U75ly65ihvduyRcrctmzr5T7zu3GiL4ta304JBzl7CxkwtfreGN2NruLy0hPimdojxb8X8/m9MhIJqYWju8XiSR1tuDLKxxnPDmNNbl7uOz4NtxxRmca1oulsKSMv/z7O17/ZgMtkuuRlhSPmRFtUOF8R4uUlvuOSBnRtxWXDWhDvdhoAPbsK+OxT5Yz4et1tEtL5O+X9KVzs6SAZa4rikvL+XzZNt6dv5Gpy3MpKa8gKT6G4zNTGZSZyjk9WpCeFO91TJGQV2cL/v2Fm7h54nxO6JjG9FV5pDWI54aTMnntm/WszdvLdSe2544hnY/oi0LZOwpJT4r/vvjlyBUUljJtZS5fr85j+qo8sncUER8Txah+rbjupEwyGtX//jj+L1fm0T4tkfN6Z3gdWyQkeFbwZnYL8HPAgBecc08ebP1AFvz+vfcog49uOZFvN+3i7vcWsyingGYN6/H4RT0ZmJkWkG1JYK3atptx09Ywed5GAPq3S2HxxgJ2F5d9v86lx7fmnqHH1sqUDSKhzJOCN7NuwBtAf6AE+Ai40Tm3srrHBLLg9++9PzO6D+f0aA74Sv+LFdvo2zrloMeCS2jYmF/EuC9WM31VHn3bNObkzk0YmJnKc1+s4bkvVjOoQyrPjO5T5Td3ReoKrwp+BHCGc+5a/+0/APucc49U95hAFfyBe+81+WaohJdJc3P43eTFZDSuz2/O7MKpXZpoMjSpkw5W8MH81skS4H4zSwWKgLOBn7S3mV0HXAfQunXrgGz4w8WbWbVtD2NH91a5R6gL+7akbWoCv5w4nxv+OZfUxDjO653B2d2b0yqlPmmJ8fq3lzov2GPw1wBjgD3AUqDIOXdbdesf6R78XZMXkVw/jvbpibRPS+SuyYsB+PhW7b1HurLyCqatzOWt2Tl8tmwrpeW+93NMlNG0YT3O7dWC24d00uGXErG82oPHOfcS8JI/xAPAT6ciPEpl5RXMXb+TtXl7v//PDWjvvY6IiY7i1C5NObVLU3bsLWHOuh1s2VXMloJiVmzdzd+nrmbO+p2MHd2bJkmaAE3qlmDvwTdxzm0zs9bAJ8AA59zO6tY/mjH4svIKNuYXsSZ3L3v2lXFO9+YqeGHyvBx+9+5ikurF8szoPvRvd3STuomEGi8Pk/wSSAVKgdudc58dbP1QnKpAwt+yLbu48Z/z2LCjkMuOb8OtgzvqyBuJGHX2i04i++0qLuXh/yxj4qwNNKwfy22DOzH6uNY6jl7CngpexO+7zbv48wdL+Xr1dlo2rs/w3hmc2zujyrmKRMKBCl6kEucc//1uG6/OWMdXq/KocNCjZTLXntCeoT2aa9I4CSsqeJFqbN1VzJSFm3hzdjYrt+2hZ6tG3H32MfowVsKGCl7kEMorHJPn5fDYJyvYsquYIV2bctvgTnRt0dDraCIHpYIXqaGiknJemr6G579Yw+59ZQzp2pRbTutIt4xkr6OJVEkFL3KYCgpLGf/1Wl6evpZdxWUM75PBg8O7Ex+j6aEltBys4HWMmEgVkhNiuXVwJ6b/9lRuOjmTyfM2cvlLsygoKvU6mkiNqeBFDqJhvVh+fWYXnhrVi3kbdjLiua/ZlF/kdSyRGlHBi9TAub0ymHBVfzbnF3P+37/i42+3UFEROsObIlVRwYvU0MAOabx94wAS4mK4/h9zOfvpL/lw8WYVvYQsFbzIYejSrCGf3nYij4/sSUlZBTe9No+hY6eTvaPQ62giP6GCFzlMMdFRDO/Tkk9vP4knL+pF9o5Czn3mK2av2+F1NJEfUcGLHKHoKOO83hm8O2YQyfVjueSFb3hnbsBPeSByxFTwIkcpM70B7940kKy2jbnj7YU8/slyQun7JVJ3qeBFAqBRQhwTru7PRVmtePp/q/j9e0so14ev4rGgnrJPpC6JjY7ioQu6k9Igjmenria/qJQnRvYiLkb7UeINFbxIAJkZvzmzCykJcdz/4XfsKirlqVG9SUnUGaSk9mnXQiQIfn5iex65sAczVm9n8ONfMHlejsblpdap4EWCZGRWK/598wm0TU3g9rcWculL37Aub6/XsaQOUcGLBFHnZklMumEgfz6vG4uyC7jwua/ZtqvY61hSR6jgRYIsKsq47Pg2vHPTQPbsK+O2txboCBupFSp4kVrSqWkS9w3rxlertvPs1FVex5E6QAUvUotGZLXk3F4tePzTFcxaq6kNJLhU8CK1yMy4//zutE5J4OaJ89mxt8TrSBLBVPAitaxBfAxjR/dhx94Srnh5FvmFKnkJDhW8iAe6ZSTz3GV9WL51N6PGzWT7nn1eR5IIpIIX8cipXZry4uVZrM3by6hxM3X4pAScCl7EQyd2SueVq/qzMb+IUeNmarhGAkoFL+KxAZmpvHJVf7J3FnLrmwt0CkAJGBW8SAjo3y6FPw49lqnLc3n6fyu9jiMRIqgFb2a3mdm3ZrbEzCaaWb1gbk8knF16XGuG98ngqc9W8vnybV7HkQgQtII3swzgZiDLOdcNiAZGBWt7IuHOzLj/vO50bprErW8s0Im85agFe4gmBqhvZjFAArApyNsTCWv146J5/rK+VDjHja/Npbi03OtIEsaCVvDOuY3Ao8AGYDNQ4Jz75MD1zOw6M5tjZnNyc3ODFUckbLRJTeTxkb1YsnEX9//7O6/jSBgL5hBNY+BcoB3QAkg0s0sPXM85N845l+Wcy0pPTw9WHJGwMqRrU647sT3/mLme9xfqF185MsEcohkMrHXO5TrnSoHJwMAgbk8kotx5Rmf6tmnMXe8sYnXuHq/jSBgKZsFvAI43swQzM+A0QL9vitRQbHQUY0f3Ji4mijGvzaOoROPxcniCOQb/DTAJmAcs9m9rXLC2JxKJmifX58lRvVm2ZTcPf7TM6zgSZoJ6FI1z7h7nXBfnXDfn3GXOOc2oJHKYTuqUzlWD2vLK1+v4elWe13EkjOibrCJh4NdndKFdWiJ3TlrE7uJSr+NImFDBi4SB+nHRPDqiJ5sLinTopNSYCl4kTPRt05jrT8rkjdnZfL5MUxnIoangRcLIrYM70rlpEndOWsTG/CKv40iIU8GLhJH4mGj+Nro3+0rLuWr8LAqKNB4v1VPBi4SZTk2TeP6yvqzN28sN/5hLSVmF15EkRKngRcLQwA5pPHxBD2as2c5v3lmEczpJiPxUjNcBROTIDO/Tko07i3js0xV0aNKAMad08DqShBjtwYuEsV+c2oFzejTnyf+uYOXW3V7HkRCjghcJY2bGvcOOJTE+ht9OXqzzucqPqOBFwlxag3j+cE5X5q7fyT+/We91HAkhKniRCDC8TwYndEzj4f8sY5OOjxc/FbxIBDAzHji/OxUO/vDeEh1VI4AKXiRitEpJ4I7TO/HZsm38Y6aGakQFLxJRrh7UjtO6NOG+KUuZuWa713HEYyp4kQgSFWU8MaoXbVITuOm1eeTsLPQ6knhIBS8SYRrWi+WFy7MoLa/gulfn6lR/dZgKXiQCtU9vwNOjevPdll3c/tYCyso1X01dpIIXiVCndGnC3Wcfw3+WbOHOSYso15eg6hzNRSMSwa49oT37yir468fLiTLjkQt7EB1lXseSWqKCF4lwY07pQHmF4/FPVxAdBQ8N70GUSr5OUMGL1AE3n9aRsgrH05+tpHlyfW4b0snrSFILNAYvUkfcNrgjw/tkMPbzVSzMzvc6jtQCFbxIHWFm3DP0WJokxXPH2wspLtXhk5FOBS9ShyTXj+WRC3uwatseHv14uddxJMhU8CJ1zAkd07ns+Da89NVaTWcQ4VTwInXQXWd3oXVKAr96eyHbdhd7HUeCRAUvUgclxMXw5EW92L6nhIvHzWTbLpV8JFLBi9RRvVs35pWr+rG5oJhRL6jkI5EKXqQOO659Kq9c1Z+tBcWMGjeTrSr5iBK0gjezzma2oNJll5ndGqzticiR6d8uhQlX92frrmKuHD9bh09GkKAVvHNuuXOul3OuF9AXKATeDdb2ROTIZbVNYezoPny3eRd//mCp13EkQGpriOY0YLVzTucREwlRp3RpwvUntue1bzbwwaJNXseRAKhRwZtZppnF+6+fbGY3m1mjw9jOKGDikQQUkdrzqzM606d1I377zmLW5e31Oo4cpZruwb8DlJtZB+AloB3wek0eaGZxwDDg7Wruv87M5pjZnNzc3BrGEZFgiI2O4m+j+xAdZYx5fZ7G48NcTQu+wjlXBpwPPOmcuw1oXsPHngXMc85trepO59w451yWcy4rPT29hk8pIsGS0ag+j47oybebdvHkf1d6HUeOQk0LvtTMLgauAD7wL4ut4WMvRsMzImFlSNemjOrXinHTVrNAM0+GrZoW/FXAAOB+59xaM2sH/PNQDzKzBGAIMPnII4qIF353zjE0bViPOzXzZNiqUcE755Y65252zk00s8ZAknPuoRo8rtA5l+qcKzjqpCJSqxrWi+XB4d1ZuW0PT32moZpwVNOjaKaaWUMzSwEWAuPN7PHgRhMRr53cuQkjs1ry/BerdZKQMFTTIZpk59wuYDgw3jnXFxgcvFgiEiruPqcrTZLqceekhZSUVXgdRw5DTQs+xsyaAyP54UNWEakDkuvH8pfzurFi6x5emr7W6zhyGGpa8PcBH+P7NupsM2sPaFBOpI4Y3LUpQ7o25enPVpKzs9DrOFJDNf2Q9W3nXA/n3I3+22uccxcEN5qIhJJ7hnYF4L4pmqsmXNT0Q9aWZvaumW0zs61m9o6ZtQx2OBEJHS0bJ3DzaR35ZOlWPvuuyu8tSoip6RDNeOB9oAWQAUzxLxOROuSan7WjQ5MG3PP+txSV6Nj4UFfTgk93zo13zpX5L68AmldApI6Ji4niz+d2I2dnEWM/18dwoa6mBZ9nZpeaWbT/cimg07GL1EEDMlMZ3ieDcdPWsGLrbq/jyEHUtOCvxneI5BZgM3AhvukLRKQOuvvsY0iMj+HudxdTUeG8jiPVqOlRNBucc8Occ+nOuSbOufPwfelJROqg1Abx/O6sY5i9bidvz832Oo5U42jO6HR7wFKISNgZkdWS/u1SeODDZeTt2ed1HKnC0RS8BSyFiIQdM+OB87tRWFLG/f/+zus4UoWjKXgNvInUcR2aJHHjSZm8O38jU5dv8zqOHOCgBW9mu81sVxWX3fiOiReROu6mUzrQqWkDfvPOIgoKS72OI5UctOCdc0nOuYZVXJKcczG1FVJEQle92GgeG9GL7XtKuOf9JV7HkUqOZohGRASA7i2T+cWpHXhvwSb+s3iz13HETwUvIgEx5pQOdM9I5u73lpC7W0fVhAIVvIgERGx0FI+P7MmefWX87t3FOKfjMLymgheRgOnYNIk7T+/Mp0u3Mmlujtdx6jwVvIgE1NU/a0f/dincO2WpTg7iMRW8iARUdJTx2IieOOf41dsLNVeNh1TwIhJwrVISuGfoscxcs4OXv9J5XL2igheRoBiR1ZLBxzTlkY+Xs1LTCntCBS8iQWFmPDi8O4lx0fz6nUWUa6im1qngRSRo0pPi+dOwY5m/IZ8JX6/zOk6do4IXkaAa1rMFp3Zpwl8/Xk72Dh1VU5tU8CISVGbGX87rRnSUcddkfQGqNqngRSToWjSqz2/O6sL0VXm8rS9A1RoVvIjUikv6t6Z/2xT+8sFSNuUXeR2nTghqwZtZIzObZGbLzOw7MxsQzO2JSOiKijIeubAHZRWO299aoKNqakGw9+CfAj5yznUBegI6r5dIHdY2LZE/DfN9AWrctDVex4l4QSt4M2sInAi8BOCcK3HO5QdreyISHkb0bcnZ3Zvx2CfLWZSjSgimYO7BtwdygfFmNt/MXjSzxCBuT0TCgJnx4Pk9SE+K55Y3FlBYUuZ1pIgVzIKPAfoAzzrnegN7gd8euJKZXWdmc8xsTm5ubhDjiEioSE6I5fGRvVi3fS/3TVnqdZyIFcyCzwFynHPf+G9Pwlf4P+KcG+ecy3LOZaWnpwcxjoiEkgGZqdxwUiZvzM7moyVbvI4TkYJW8M65LUC2mXX2LzoN0I9qEfnebYM70S2jIXdNXsTWXcVex4k4wT6K5pfAa2a2COgFPBDk7YlIGImLieLJi3pTVFquueODIKgF75xb4B9+6eGcO885tzOY2xOR8NOhSQN+f05XvlyZxyuakCyg9E1WEfHcJce1ZvAxTXjoo2UszinwOk7EUMGLiOfMjIcu6EF6g3iunjBbs04GiApeREJCWoN4xl/Vj32l5Vz1ymzyC0u8jhT2VPAiEjI6NU1i3OVZbNheyHWvzqW4tNzrSGFNBS8iIeX49qk8NrIns9bt4I63dGTN0YjxOoCIyIGG9mzB5oIiHvhwGS1T6nPXWcd4HSksqeBFJCT9/IT2bNhRyPNfrKFNSiKjj2vtdaSwo4IXkZBkZvxp6LFs3FnEH/61hBaN6nFy5yZexworGoMXkZAVEx3F30b3oXPTJMa8No+lm3Z5HSmsqOBFJKQ1iI/h5Sv70bB+LJe/PIt1eXu9jhQ2VPAiEvKaJdfjH9f0p8I5LnnxGzYX6JyuNaGCF5Gw0KFJEhOu6k9BUSmXvTSLHXv1RahDUcGLSNjo3jKZF6/IIntHIVe8PIvdxaVeRwppKngRCSvHt0/l2Uv78N3mXVzzyhyKSvRt1+qo4EUk7JzapSlPXNSLOet3cP0/57KvTCVfFRW8iISloT1b8NDwHkxbkcstExdQVl7hdaSQo4IXkbA1sl8r/vh/Xfno2y3cNXkxzmnemsr0TVYRCWtX/6wd+UWlPP3ZSjo3S+LaE9p7HSlkaA9eRMLerad15Mxjm/HAh98xfWWe13FChgpeRMJeVJTx2MiedGySxJjX57F+u77tCip4EYkQifExvHB5FmZw3atz2buvzOtInlPBi0jEaJ2awDOj+7Aqdw83T5xPeR0/WYgKXkQiyqAOadw77Fg+W7aNe6d8W6ePrNFRNCIScS49vg3ZOwp5ftoaWqck1Nkja1TwIhKRfnNmF7J3FnL/h9/RsnF9zuzW3OtItU5DNCISkaKijMdH9qJXq0bc8sYCZq/b4XWkWqeCF5GIVS82mhcvzyKjUX2ueWU2y7bUrTNCqeBFJKKlNojn1Wv6Uz8umstfmkX2jkKvI9UaFbyIRLyWjRN49erjKC4t5/KXZ5G3Z5/XkWqFCl5E6oTOzZJ4+cp+bMov4oqXZ1FQGPknCwlqwZvZOjNbbGYLzGxOMLclInIoWW1TeP6yvqzYupsrxkf+GaFqYw/+FOdcL+dcVi1sS0TkoE7u3ISxo/uweGMBV78ym8KSyJ3SQEM0IlLnnHFsM568qBdz1+/k2gmRe9q/YBe8Az4xs7lmdl2QtyUiUmNDe7bg0RE9mbFmOyOfn8GWgmKvIwVcsAt+kHOuD3AWMMbMTjxwBTO7zszmmNmc3NzcIMcREfnB8D4tGXdZFmty9zBs7HQW5eR7HSmgglrwzrlN/j+3Ae8C/atYZ5xzLss5l5Wenh7MOCIiPzGka1PeuWkgcTFRjHhuBlMWbvI6UsAEreDNLNHMkvZfB04HlgRreyIiR6pLs4b8a8wgerRM5pcT53PvlG8pKQv/k3gHcw++KTDdzBYCs4B/O+c+CuL2RESOWGqDeF679niuGtSW8V+tY+TzM9iYX+R1rKNioTRXclZWlpszR4fLi4i3Ply8mV9PWkRMtPHwBT0449hmXkeqlpnNre4wdB0mKSJygLO7N+eDX/6MjEb1uf4fc7n1jfnkF5Z4HeuwqeBFRKrQNi2Rd28axC2ndeSDRZsZ8sQ0/rt0q9exDosKXkSkGnExUdw2pBPvjRlEamIc1746h7H/Wxk2pwFUwYuIHEK3jGT+9YtBnN87g0c/WcG9U5ZSEQYn9NYp+0REaiA+JprHRvQkJTGOl6avZfveEh4b0ZO4mNDdT1bBi4jUUFSU8ftzjiGtQTwPf7SM3cWlPH9ZX+Jjor2OVqXQ/dEjIhKCzIwbT87kweHdmbo8l1++Pp/S8tD8UpQKXkTkCFzcvzX3DjuWT5Zu5Y63FlIegmPyGqIRETlCVwxsS2FJOQ9/tIz6sdE8OLw7UVHmdazvqeBFRI7CjSdnUlRSxtP/W4UZ3H9+d6JDpORV8CIiR+m2IZ1wwN/+t4rd+8p4YmSvkDi6RgUvInKUzIw7Tu9MUr0YHvhwGXv3lfHsJX2pH+ft0TXe/4gREYkQ153oO7rmixW5XPGy9yf1VsGLiATQxf1b8/So3szbsJNLX5pFQaF3Ja+CFxEJsKE9W/D3S/qwdFMBo1+cyc693sxEqYIXEQmC049txrjLs1i5bQ8XvzCT3N37aj2DCl5EJEhO6dyE8Vf2Y932vYx8fgYbthfW6vZV8CIiQTSoQxqvXXscOwtLGP7sVyzOKai1bavgRUSCrG+bFCbdMJD4mGguGjeDqcu31cp2VfAiIrWgQ5MGvHvTQNqmJnLNhDm8NTs76NtUwYuI1JImDevx5vXHMzAzlV+/s4hHP14e1LNDqeBFRGpRUr1YXr6yH6P6tWLs56u49c0F7CsrD8q2NFWBiEgti42O4sHh3WmVksBfP17O5oJixl/Zj8T4wFayCl5ExANmxphTOtAqJYGvVuaREIR5a1TwIiIeGtazBcN6tgjKc2sMXkQkQqngRUQilApeRCRCqeBFRCKUCl5EJEKp4EVEIpQKXkQkQqngRUQilAVzopvDZWYFwMoq7koGCmp4e//1qpalAXmHGevAbdX0/qqWV5WpuutHk/lguWqaL1wyV7U8HN8fNclc+breHzW/P9LfHx2dc8lVPrtzLmQuwLiaLD/Y7f3Xq1k2J1CZDjdzdZkOlf9IMh9p7nDMHCnvj5pk9vq11vsj9N8fB15CbYhmSg2XH+z2lIMsC2SmQ91f1fLqMh0q/5E4ktzhmLmq5eH4/qhJ5srX9f6o+f116f3xIyE1RBNsZjbHOZfldY7Docy1JxxzK3PtCcfcobYHH2zjvA5wBJS59oRjbmWuPWGXu07twYuI1CV1bQ9eRKTOUMGLiESosC14M3vZzLaZ2ZIjeGxfM1tsZqvM7Gkzs0r3jTSzpWb2rZm9HuqZzexKM8s1swX+y7WhnrnS/ReamTOzgH5wFaTX+Qb/8gVmNt3MugYycxBz3+5/Py8ys8/MrE0YZD7RzOaZWZmZXRgKWat5vivMbKX/ckWl5e3M7Bv/8jfNLC4Q2zsiR3I8aihcgBOBPsCSI3jsLGAAYMB/gLP8yzsC84HG/ttNwiDzlcDYcHqd/fclAdOAmUBWqGcGGlZaZxjwUTi81sApQIL/+o3Am2GQuS3QA3gVuNDrrMBUoO0By1KANf4/G/uv7++Nt4BR/uvPATcG+r1S00vY7sE756YBOyovM4lpn7EAAAY3SURBVLNMM/vIzOaa2Zdm1uXAx5lZc3z/WWc437/Aq8B5/rt/DjzjnNvp38a2MMgcVEHM/GfgEaA4HDI753ZVWjURCPjRCUHK/blzrtC/6kygZRhkXuecWwRUhELWapwBfOqc2+Hvi0+BM/2/hZwKTPKvN4Fa+r9albAt+GqMA37pnOsL/Ar4exXrZAA5lW7n+JcBdAI6mdlXZjbTzM4Malqfo80McIH/V/BJZtYqeFG/d1SZzaw30Mo590Gwg1Zy1K+zmY0xs9X4fjDdHMSslQXi/bHfNfj2lIMtkJmDrSZZq5IBZFe6vT9/KpDvnCs7YLknIuak22bWABgIvF1pqDe+qlWrWLZ/bywG3zDNyfj2dL40s27OufzApvUHCUzmKcBE59w+M7sB3x7DqYHO+n2Qo8xsZlHAE/iGlmpFgF5nnHPPAM+Y2Wjg98AVVawfMIHK7X+uS4Es4KRAZvxJkABmDraDZTWzq4Bb/Ms6AB+aWQmw1jl3PtXn9/zvVVnEFDy+30bynXO9Ki80s2hgrv/m+8Cz/PjX1JbAJv/1HGCmc64UWGtmy/EV/uxQzeyc215p+QvAw0HKut/RZk4CugFT/f+pmgHvm9kw59ycEM18oDf86wZbQHKb2WDgbuAk59y+oCYO/GsdTFVmBXDOjQfGA5jZVOBK59y6Sqvk4NsR3K8lvrH6PKCRmcX49+K9+Hv9wKvB/0Bc8H0Ys6TS7a+BEf7rBvSs5nGzgeP54cOds/3LzwQm+K+n4fsVLDXEMzevtM75+H5AhfTrfMA6Uwnwh6xBep07VlpnKEc4YZYHuXsDqyvnD/XMle5/hQB+yHqkWan+Q9a1+D5gbey/nuK/721+/CHrTcF67Q/59/VqwwH4h5oIbAZK8f00vQZoB3wELASWAn+s5rFZwBL/G38sP3yj14DH/Y9dvP8fKcQzPwh863/850CXUM98wDpTCfxRNMF4nZ/yv84L/K/zsWHynv4vsNWfewHwfhhk7ud/rr3AduBbL7NSRcH7l18NrPJfrqq0vD2+I4RW4Sv7+EC/V2p60VQFIiIRKtKOohERET8VvIhIhFLBi4hEKBW8iEiEUsGLiEQoFbyENDPbU8vbezFQM0WaWbn5Zp9cYmZTzKzRIdZvZGY3BWLbIqAzOkmIM7M9zrkGAXy+/d8wDLrK2c1sArDCOXf/QdZvC3zgnOtWG/kk8mkPXsKOmaWb2TtmNtt/GeRf3t/Mvjaz+f4/O/uXX2lmb5vZFOATMzvZzKb6J2dbZmav+WcBxL88y399j5ndb2YL/ZPPNfUvz/Tfnm1m99Xwt4wZ/DDZWgPzzc0+z3zzoZ/rX+chINO/1/9X/7p3+rezyMzuDeDLKHWACl7C0VPAE865fsAFwIv+5cuAE51zvYE/Ag9UeswA4Arn3P6J2HoDtwJd8X3zcFAV20nEN/VDT3xz1/+80vaf8m//kPOM+OdhOQ3fHCzgmyL5fOdcH3zztT/m/wHzW2C1c66Xc+5OMzsd31xI/YFeQF8zO/FQ2xPZL5ImG5O6YzDQtdIMgA3NLAlIBiaYWUd8M/jFVnrMp865ynOBz3LO5QCY2QJ8c5RMP2A7JcD+KY3nAkP81wfwwxzfrwOPVpOzfqXnnotvznDwTYnxgL+sK/Dt2Tet4vGn+y/z/bcb4Cv8adVsT+RHVPASjqKAAc65osoLzexvwOfOufP949lTK92994DnqDyrYjlV/18odT98SFXdOgdT5JzrZWbJ+H5QjAGeBi4B0oG+zrlSM1sH1Kvi8QY86Jx7/jC3KwJoiEbC0yfAL/bfMLP9070mAxv9168M4vZn4hsaAhh1qJWdcwX4ThDyKzOLxZdzm7/cTwH2nyd1N77plPf7GLjaP285ZpZhZk0C9HeQOkAFL6EuwcxyKl1ux1eWWf4PHpcCN/jXfQR40My+AqKDmOlW4HYzmwU0BwoO9QDn3Hx8MxaOAl7Dl38Ovr35Zf51tgNf+Q+r/Ktz7hN8Q0AzzGwxvtPAJVW5AZEq6DBJkcNkZgn4hl+cmY0CLnbOnXuox4nUNo3Bixy+vsBY/5Ev+fjmBRcJOdqDFxGJUBqDFxGJUCp4EZEIpYIXEYlQKngRkQilghcRiVD/D7r8g42IbUs6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the learning rates\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "obZ4gREh-9hs"
   },
   "outputs": [],
   "source": [
    "# learn.fit_one_cycle(3, 1e-2, wd=0.2)\n",
    "    \n",
    "# print(f\"Saving model...export_{idx}\")\n",
    "# learn.export(file = Path(f\"/kaggle/working/export_{idx}.pkl\")) # Save the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Q5sbW-xW-9h0"
   },
   "outputs": [],
   "source": [
    "learn = load_learner('/kaggle/input/m5-forecasting-models/', file=f'export_{idx}.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "L4kfxka2-9h9",
    "outputId": "2658cbdb-35e2-4a29-94a8-1abecaaba38a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of Store - None dataframe :  (6841121, 4)\n",
      "Processing time:  11.380295515060425\n",
      "Processing time:  104.87205600738525\n",
      "CPU times: user 1min 54s, sys: 2.01 s, total: 1min 56s\n",
      "Wall time: 1min 56s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "te = create_dt(False)\n",
    "te = create_fea(te)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ArhYwuFh-9iF",
    "outputId": "85eeb657-4424-4448-e697-3aa434b5afe7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(487840, 31)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>dept_id</th>\n",
       "      <th>store_id</th>\n",
       "      <th>cat_id</th>\n",
       "      <th>state_id</th>\n",
       "      <th>d</th>\n",
       "      <th>sales</th>\n",
       "      <th>date</th>\n",
       "      <th>wm_yr_wk</th>\n",
       "      <th>weekday</th>\n",
       "      <th>wday</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>event_name_1</th>\n",
       "      <th>event_type_1</th>\n",
       "      <th>event_name_2</th>\n",
       "      <th>event_type_2</th>\n",
       "      <th>snap_CA</th>\n",
       "      <th>snap_TX</th>\n",
       "      <th>snap_WI</th>\n",
       "      <th>sell_price</th>\n",
       "      <th>lag_7</th>\n",
       "      <th>lag_28</th>\n",
       "      <th>rmean_7_7</th>\n",
       "      <th>rmean_28_7</th>\n",
       "      <th>rmean_7_28</th>\n",
       "      <th>rmean_28_28</th>\n",
       "      <th>week</th>\n",
       "      <th>quarter</th>\n",
       "      <th>mday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>FOODS_3_827_WI_3_validation</td>\n",
       "      <td>3048</td>\n",
       "      <td>6</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>d_1913</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2016-04-24</td>\n",
       "      <td>11613</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>2016</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.428571</td>\n",
       "      <td>1.678571</td>\n",
       "      <td>1.964286</td>\n",
       "      <td>16</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HOUSEHOLD_2_257_TX_1_validation</td>\n",
       "      <td>1352</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>d_1913</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016-04-24</td>\n",
       "      <td>11613</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>2016</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.43</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.285714</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.607143</td>\n",
       "      <td>16</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HOUSEHOLD_2_264_TX_1_validation</td>\n",
       "      <td>1359</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>d_1913</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2016-04-24</td>\n",
       "      <td>11613</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>2016</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.98</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>0.321429</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>16</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>HOUSEHOLD_2_263_TX_1_validation</td>\n",
       "      <td>1358</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>d_1913</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2016-04-24</td>\n",
       "      <td>11613</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>2016</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.97</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>0.178571</td>\n",
       "      <td>0.214286</td>\n",
       "      <td>16</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>HOUSEHOLD_2_262_TX_1_validation</td>\n",
       "      <td>1357</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>d_1913</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2016-04-24</td>\n",
       "      <td>11613</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>2016</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.97</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>0.285714</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0.392857</td>\n",
       "      <td>16</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                id  item_id  dept_id  store_id  cat_id  \\\n",
       "0      FOODS_3_827_WI_3_validation     3048        6         9       2   \n",
       "1  HOUSEHOLD_2_257_TX_1_validation     1352        3         4       1   \n",
       "2  HOUSEHOLD_2_264_TX_1_validation     1359        3         4       1   \n",
       "3  HOUSEHOLD_2_263_TX_1_validation     1358        3         4       1   \n",
       "4  HOUSEHOLD_2_262_TX_1_validation     1357        3         4       1   \n",
       "\n",
       "   state_id       d  sales       date  wm_yr_wk  weekday  wday  month  year  \\\n",
       "0         2  d_1913    0.0 2016-04-24     11613        3     2      4  2016   \n",
       "1         1  d_1913    2.0 2016-04-24     11613        3     2      4  2016   \n",
       "2         1  d_1913    0.0 2016-04-24     11613        3     2      4  2016   \n",
       "3         1  d_1913    0.0 2016-04-24     11613        3     2      4  2016   \n",
       "4         1  d_1913    1.0 2016-04-24     11613        3     2      4  2016   \n",
       "\n",
       "   event_name_1  event_type_1  event_name_2  event_type_2  snap_CA  snap_TX  \\\n",
       "0             0             0             0             0      0.0      0.0   \n",
       "1             0             0             0             0      0.0      0.0   \n",
       "2             0             0             0             0      0.0      0.0   \n",
       "3             0             0             0             0      0.0      0.0   \n",
       "4             0             0             0             0      0.0      0.0   \n",
       "\n",
       "   snap_WI  sell_price  lag_7  lag_28  rmean_7_7  rmean_28_7  rmean_7_28  \\\n",
       "0      0.0        1.00    0.0     5.0   0.000000    2.428571    1.678571   \n",
       "1      0.0        7.43    0.0     1.0   0.285714    1.000000    0.500000   \n",
       "2      0.0        7.98    0.0     0.0   0.142857    0.428571    0.321429   \n",
       "3      0.0        3.97    0.0     0.0   0.000000    0.428571    0.178571   \n",
       "4      0.0       10.97    0.0     1.0   0.428571    0.285714    0.357143   \n",
       "\n",
       "   rmean_28_28  week  quarter  mday  \n",
       "0     1.964286    16        2    24  \n",
       "1     0.607143    16        2    24  \n",
       "2     0.428571    16        2    24  \n",
       "3     0.214286    16        2    24  \n",
       "4     0.392857    16        2    24  "
      ]
     },
     "execution_count": 21,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(te.shape)\n",
    "te.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wWMQ6ur4-9iN"
   },
   "source": [
    "# Lets predict with a sample test row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uqDfRK3t-9iO",
    "outputId": "5dc56ac0-2af2-4087-b27e-12a81c95dafe"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.4182])"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(te.loc[1])[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2eXO8UAQ-9iY",
    "outputId": "e2c04575-8763-440c-d4ba-5679a9b3c98e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw prediction:  (FloatItem [0.418167], tensor([0.4182]), tensor([0.4182]))\n",
      "Taking exponentials:  tensor([0.5192])\n"
     ]
    }
   ],
   "source": [
    "print(\"Raw prediction: \",learn.predict(te.loc[1]))\n",
    "print(\"Taking exponentials: \",(np.exp(learn.predict(te.loc[1])[1]) -1) )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2g9zoqhr-9ig",
    "outputId": "ac56a9a1-2ef9-4f32-9700-a8bff4b7cb4f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Notebook processing time:  241.76258730888367\n"
     ]
    }
   ],
   "source": [
    "end_nb = time.time()\n",
    "\n",
    "print(\"Notebook processing time: \", (end_nb- start_nb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mV_IkVFY-9jq"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "name": "Day 22#fastai-tabular-starter-code.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}