Filipino_ULMFiT.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Filipino-ULMFiT.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOJFy3varXFmJihsMaNhaIC",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/danjohnvelasco/Filipino-ULMFiT/blob/master/Filipino_ULMFiT.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q0wRH746s7Ns"
      },
      "source": [
        "# Filipino ULMFiT\n",
        "This notebook shows the ULMFiT approach to Filipino text classification task on [Hate Speech Dataset](https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks#datasets) using a pre-trained Filipino language model. To learn more about the ULMFiT approach check the [Docs](https://docs.fast.ai/tutorial.text) and [Paper](https://arxiv.org/abs/1801.06146).\n",
        "\n",
        "If you wish to apply this on another text classification dataset, you may do so.\n",
        "It's just a matter of replacing the dataset in the code.\n",
        "\n",
        "Originally posted in this [repository](https://github.com/danjohnvelasco/Filipino-ULMFiT)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fvh9xdr2sak0"
      },
      "source": [
        "# Install fastai v2\n",
        "# uncomment if your environment doesn't use fastai >= v2.\n",
        "# run pip freeze to check if fastai is installed\n",
        "\n",
        "# !pip install -U fastai"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l98GbAyrss4r",
        "outputId": "2d29e6f0-bb05-49bc-9aa5-f38b55b42845",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 350
        }
      },
      "source": [
        "# if you're on Colab, make sure you're using a GPU instance.\n",
        "# Make sure that your GPU supports mixed-precisoin traning (e.g. Tesla T4, P4, P100, V100)\n",
        "!nvidia-smi"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Thu Oct 15 03:45:08 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 455.23.05    Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   64C    P8    11W /  70W |      0MiB / 15079MiB |      0%      Default |\n",
            "|                               |                      |                 ERR! |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jE3Z8nKtWB3R"
      },
      "source": [
        "# if you're on Colab, and what to save files in Google Drive:\n",
        "#from google.colab import drive\n",
        "#drive.mount('/gdrive')\n",
        "#%cd \"/gdrive/My Drive/\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "siecmphg86M0"
      },
      "source": [
        "# Download pre-trained language model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hUIZpjkc7wOP"
      },
      "source": [
        "# create 'models' folder\n",
        "!mkdir models\n",
        "\n",
        "# download pre-trained model and language model vocab (size: 432MB)\n",
        "!gdown --id 19jdv8-XEbDNiqlm_lPb1csbVZYkn3gfA\n",
        "\n",
        "# unzip the pre-trained model\n",
        "!unzip pretrained.zip -d models"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jrJt_BXy9bl3"
      },
      "source": [
        "# Filenames of pre-trained LM weights and vocab\n",
        "# To be passed to text_classifier_learner() later on\n",
        "pretrained_fnames = ['finetuned_weights_20', 'vocab']"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fC0NYoErYZu0"
      },
      "source": [
        "## Download Text Classification Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jgW46ZMPYcqh"
      },
      "source": [
        "!wget https://s3.us-east-2.amazonaws.com/blaisecruz.com/datasets/hatenonhate/hatespeech_raw.zip\n",
        "!unzip hatespeech_raw.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YivYGXqKuBCO"
      },
      "source": [
        "# Before you start...\n",
        "\n",
        "1.  Import dependencies\n",
        "2.  Define convenient functions for later use\n",
        "3.  Load data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hJ8NkN3ks09j"
      },
      "source": [
        "from fastai.text.all import *\n",
        "from sklearn.metrics import accuracy_score"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-M-m2zJ0u0Eh"
      },
      "source": [
        "# Run this function before creating a learner if you want  \n",
        "# your work to be reproducible\n",
        "\n",
        "# Convenience function for setting the random seed manually\n",
        "def set_random_seed(seed):\n",
        "    # python RNG\n",
        "    import random\n",
        "    random.seed(seed)\n",
        "\n",
        "    # pytorch RNGs\n",
        "    import torch\n",
        "    torch.manual_seed(seed)\n",
        "    torch.backends.cudnn.deterministic = True\n",
        "    if torch.cuda.is_available(): torch.cuda.manual_seed_all(seed)\n",
        "\n",
        "    # numpy RNG\n",
        "    import numpy as np\n",
        "    np.random.seed(seed)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k6X9FthozhBj"
      },
      "source": [
        "# Convenience function for testing model on test set\n",
        "def get_test_acc(learn):\n",
        "    # Create dataloader for test set\n",
        "    test_dl = learn.dls.test_dl(test_df, with_label=True)\n",
        "    # Make predictions on test set\n",
        "    pred_probas, _ , pred_classes = learn.get_preds(dl=test_dl, with_decoded=True) \n",
        "    # get accuracy of (y_true, y_pred)\n",
        "    return accuracy_score(test_df.label.values, pred_classes)"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eLDggOHht90q"
      },
      "source": [
        "# Modify this to match your data directory\n",
        "train_df = pd.read_csv(\"hatespeech/train.csv\", lineterminator='\\n')\n",
        "valid_df = pd.read_csv(\"hatespeech/valid.csv\", lineterminator='\\n')\n",
        "test_df = pd.read_csv(\"hatespeech/test.csv\", lineterminator='\\n')"
      ],
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6IIPPtSeuiIl"
      },
      "source": [
        "# add 'is_valid' column (for fastai train-val splitting)\n",
        "valid_df['is_valid'] = True\n",
        "test_df['is_valid'] = False\n",
        "train_df['is_valid'] = False"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iekJzpy3YtSc",
        "outputId": "4a4ae2e9-0c3a-461e-9503-a2c85ec7033f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 197
        }
      },
      "source": [
        "# label: hate = 1, non-hate = 0\n",
        "train_df.head()"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "      <th>is_valid</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>GASTOS NI VP BINAY SA POLITICAL ADS HALOS P7-M NA\\r\\rInaasahan na ni Vice President Jejomar Binay na may mga taong... https://t.co/SDytgbWiLh</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Mar Roxas TANG INA TUWID NA DAAN DAW .. EH SYA NGA DI STRAIGHT</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Salamat sa walang sawang suporta ng mga taga makati! Ang Pagbabalik Binay In Makati #OnlyBinayInMakatiSanKaPa https://t.co/iwAOdtZPRE</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>@rapplerdotcom putangina mo binay TAKBO PA</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Binay with selective amnesia, forgetting about the past six years he spent preparing to be president.  #PiliPinasDebates2016</td>\n",
              "      <td>0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                                                                                                            text  ...  is_valid\n",
              "0  GASTOS NI VP BINAY SA POLITICAL ADS HALOS P7-M NA\\r\\rInaasahan na ni Vice President Jejomar Binay na may mga taong... https://t.co/SDytgbWiLh  ...     False\n",
              "1                                                                                 Mar Roxas TANG INA TUWID NA DAAN DAW .. EH SYA NGA DI STRAIGHT  ...     False\n",
              "2          Salamat sa walang sawang suporta ng mga taga makati! Ang Pagbabalik Binay In Makati #OnlyBinayInMakatiSanKaPa https://t.co/iwAOdtZPRE  ...     False\n",
              "3                                                                                                     @rapplerdotcom putangina mo binay TAKBO PA  ...     False\n",
              "4                   Binay with selective amnesia, forgetting about the past six years he spent preparing to be president.  #PiliPinasDebates2016  ...     False\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xeq8PeNSvU_2"
      },
      "source": [
        "# Language Model Fine-tuning\n",
        "\n",
        "Here, we fine-tune the pre-trained language model to better learn the vocab and syntax of the target corpus which is, in our case, the Hate Speech Dataset.\n",
        "\n",
        "**About the pre-trained language model files:**\n",
        "\n",
        "By default, fastai looks for models inside the 'models' folder. Make sure that the pre-trained models and vocab are in 'models' folder."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dqGHYfF4ZSIy",
        "outputId": "64e3b1eb-862e-434d-87e7-ea7b16b8a833",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "# Concatenate train, validation, and test set for LM fine-tuning\n",
        "# Language modelling task doesn't need labels so it's safe to use the test set\n",
        "lm_df_10k =  pd.concat([train_df, valid_df, test_df])\n",
        "lm_df_10k.shape"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(18464, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jVQCKuYNvWIE",
        "outputId": "083a8fde-a82f-4672-f97e-3b4b5247d72a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 16
        }
      },
      "source": [
        "# Create a dataloader for language model fine-tuning\n",
        "# ColSplitter() splits the dataframe by 'is_valid' column\n",
        "dls_lm = DataBlock(blocks=TextBlock.from_df('text', is_lm=True),\n",
        "                    get_x=ColReader('text'),\n",
        "                    splitter=ColSplitter()).dataloaders(lm_df_10k, bs=128, seq_len=72, min_freq=2, num_workers=0)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IVtS9TNez9mg"
      },
      "source": [
        "# Uncomment this if you want early stopping and save your best model.\n",
        "# This is fastai callbacks, see docs for more info\n",
        "cbs = [EarlyStoppingCallback(monitor='valid_loss', patience=2), SaveModelCallback()]\n",
        "\n",
        "# Notice the pretrained_fnames parameter.\n",
        "# Here we pass the list of filenames of pretrained weights and vocab.pkl\n",
        "# This is where the use of pre-trained language model happens.\n",
        "learn = language_model_learner(\n",
        "    dls_lm, AWD_LSTM, drop_mult=0.5, \n",
        "    metrics=[accuracy, Perplexity()],\n",
        "    pretrained_fnames=pretrained_fnames, cbs=cbs).to_fp16()"
      ],
      "execution_count": 63,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L92DgUuv0S3Q",
        "outputId": "88daf27e-e521-4758-d7c7-5959f63749f4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        }
      },
      "source": [
        "# Learning rate finder\n",
        "learn.lr_find()"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SuggestedLRs(lr_min=0.19054607152938843, lr_steep=0.0831763744354248)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAENCAYAAAACHGKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxcdb3/8dcnmSzN3jbp3nTf9xJaWVrAIlB2UBFBUFBKvRdEvXiBy0/B65UrV2UXalkVEdQKyiL71rKbQind6L7TNl2TNG3Wz++PmUKIaZq2OTmTzPv5eMyDzDknM+9MS9492/dr7o6IiCSupLADiIhIuFQEIiIJTkUgIpLgVAQiIglORSAikuBUBCIiCS6wIjCzIWY2r96j1My+32CbC81svpl9ZGZvmdmYoPKIiEjjrDXuIzCzZGADMNHd19RbfjSw2N13mNlU4EZ3nxh4IBER+VSkld5nCrCifgkAuPtb9Z6+A/RqpTwiIhLTWkVwPvDoAbb5NvDsgV4oPz/f+/bt2xKZREQSxty5c7e6e0Fj6wI/NGRmqcBGYIS7b97PNicAdwPHuvu2RtZPA6YBFBYWHrFmzZqGm4iISBPMbK67FzW2rjWuGpoKvN9ECYwG7gPOaqwEANx9prsXuXtRQUGjhSYiIoeoNYrg6+znsJCZFQKPAxe5+9JWyCIiIg0Eeo7AzDKBLwGX11s2HcDdZwA/AToDd5sZQM3+dl1ERCQYgRaBu+8m+ou+/rIZ9b7+DvCdIDOIiEjTdGexiEiCUxGIiCS4hCmCkrJKnvpwI1U1dWFHERGJKwlTBO+u2saVj37A0s1lYUcREYkrCVMEI3vkArBw466Qk4iIxJeEKYLCThlkp0VYsKE07CgiInElYYogKckY3iOHBdojEBH5nIQpAoCRPXNZ/EkpNbU6YSwisk+CFUEOe6vrWLl1d9hRRETiRmIVQeyE8YINOjwkIrJPQhVB/4Is0lOSdMJYRKSehCqC5CRjeHedMBYRqS+higCiJ4wXbSylri74uZpFRNqCxCuCHrmUV9awZntF2FFEROJCwhXBiJ45gE4Yi4jsk3BFMKhLNqnJSTpPICISk3BFkBpJYki3bBbqyiERESABiwCiN5Yt2LgLd50wFhFJyCIY0SOXnRXVbNi5J+woIiKhS8giGNkzeodx8eodIScREQlfQhbB8O459M/P5Md/X6Crh0Qk4SVkEaRGknj4OxPJTovwzQfeY/mW8rAjiYiExoI6YWpmQ4A/1VvUH/iJu99WbxsDbgdOBSqAb7n7+029blFRkRcXF7dIxpUl5Zz327dJSU7ixjNHkGxGTeyO45RkIyU5iU6ZqYzokUM0qohI22Rmc929qLF1kaDe1N0/BsbGAiQDG4AnGmw2FRgUe0wE7on9t1X0L8ji95dO5PyZb3P5w3P3u92Efp245pQhHNGnU2tFExFpNYEVQQNTgBXuvqbB8rOA33t0t+QdM8szs+7u/kkr5WJ4jxxevfp41u3YQ7IZyUmG49TUOtW1dSzYsIu7Xl3Bl+95m+OHFNAtJ50dFVXsrKimIDuN0b1yGdUzj9G9cslMa62PU0Sk5bTWb67zgUcbWd4TWFfv+frYss8VgZlNA6YBFBYWtni4zllpdM5Ka3RdUd9OnHdkbx58czUPvbWaRRtLyctIIbdDCh+s3cnT86NR01OSOGl4N84Z35NJA/OJJCfk6RcRaYMCO0fw6RuYpQIbgRHuvrnBuqeBX7j7G7HnLwPXuPt+TwK05DmClrC1vJKP1u/ipcWbeXr+J+zaU0333HR+/dUxHD0wP+x4IiJA0+cIWuOfrVOB9xuWQMwGoHe9571iy9qM/Kw0ThjahZ+fM4r3rp/CjG8cQUZqMhfe/y63vriU2nrDXdfU1mn4axGJO61xaOjrNH5YCOBJ4Aoze4zoSeJdrXl+oKWlRZI5ZWQ3Jg3K5yd/X8jtLy9jzrISOmelsaKknLXbKqhzJ6dD9NBSr44dGNe7I+MK8xjWPYes9AgZKckkmbGpdC+rt+1m7bYKyvbWUFlTS2VNHZ0yUxlX2JHh3XNIjejwk4gcvkAPDZlZJrAW6O/uu2LLpgO4+4zY5aN3AacQvXz0kqYOC0H8HRpqyqy56/m/55aQ2yGFAQVZ9C/IJDnJ2LWnmp0V1azcWs7iT8o+t9cAkGTQ2I6DGez740qLJHFEn458fUIhp4zsRorOSYhIE5o6NBT4OYKW1paKoDn2VNXy0YZdLN9STkVVDbsra6muraN7Xjr9OmdS2DmDvIxU0iJJRJKMzaWVvL92B++v2cELizazdnsFXbLT+PIRveiSnUZKchKpkSSGd89hePcckpL2f//D3upalm0uZ/GmUpZvKSc9kkRBdhr5WWnkdEghIzWZjNQIaZEkzMAwUiJGflZao8Xj7mzctZeFG3axdnsFu/ZUs6Oiito6Z3DXbEb0yGVglyxKyipZtXU3q7ftZktpJTsqqti+uwoHOmWk0DEzlW456Yzv05HRvXJJiyQH+CcgkhhUBO1UXZ3z+tISfvf2al5fWkLDP8r8rDQmDcpnQr9OjOiRw+Cu2VTV1vH8gk08+eFG3lqx7dO9kdRIEtW1df/yGo0xg4KsNLrmpBNJNtyjJbBmewU7K6o/3S7JILdDCgA76i2vLzM1mY6ZqXTOTP10ux27qyirrPk017Bu2dTUOaV7qynbW0NWWoRuOel0zU0nOy1C9F4/IyXZyEqLkNMhhez0CFlpEbLTI2SkRthZUcXGnXvZVLqXzaV7KSmrZGt5JeV7az69WdAM9t03aBiRZCM1VqzpKclkpiaTmRZ93ehrp5CXkcLALlkM6ZZN/n6uPBOJByqCBLCnqpbKmlqqauvYU1VL8eodzF5WwpxlW9m+uwqA5CQj2Yyq2jp6dezAaaO6M7Z3HkO759CnUwZ17myvqGJrWRVle6upqKqlIva67uBAZU0tm0sr2bRrD5tLK6mL/f0xM3rkpjOiZy4jekTHcspJTyEpyXB3tpRVsnDjLlaW7KYgO41++Zn0jW3TmG3llcxds4PiNTtYuHEX6ZHkT3/Bl+6pjv1Cr2R3ZQ37/gZX19ZRtrfmXw611ZcWSaJLThoFWWkUZKeRHXv/un0/YEydOzV10XtJqmrq2FNdy+7KWnZX1lBWWUP53hr2VNd+7rXzs1IZ27sjR/btyJH9OjGyR67O40jcUBEksLo6Z92OChZuLGXhxl1U1zqnjOzGuN557XLYDHenoqqWsr01lFdGHxWVNeRmpNA9twMdM1Ja7Oeuqa1j++4qlm4u5+PNZSz+pJT31+xg5dbdQHRvZlTPXMYX5jGoazY56RGy0lLITEsmOz1CZlqEnPQU3YgorUJFINKKSsoqKV69PXouZ+1OPtqwi6qauv1u378gk4n9OnFk305M7N+ZnnkdWjGtJAoVgUiIKmtq2VJaSXllTWxPpZry2GGmbeWVfLB2J++t3k7Z3uh5kd6dOjCxX2emDO3CicO76oowaRGhDDonIlFpkWR6d8pocpvaOmfJplLeW7Wdd1Zu4+XFm5k1dz3dctK5cGIh508opCBbJ6MlGNojEIlDtXXOq0u28Lu3VzNn2VaSk4xjB+Zz+ujunDSi26dXY4k0lw4NibRhy7eUM2vuep6ev5H1O/aQGkninLE9+c6kfgzqmh12PGkjVAQi7YC7M2/dTv4ydz1/nbueypo6ThhSwIUT+3DckAKdS5AmqQhE2plt5ZX84Z21PPzOaraWV9ExI4UzxvTggomFDO2WE3Y8iUMqApF2qqqmjtlLS3hi3gZeWrSZ6to6zp9QyH98afB+59iQxKSrhkTaqdRIEicO78qJw7uys6KK219exu/fXsPTH27kB18azMVH9SW5ifGmRKB15iMQkVaQl5HKDWeM4LmrJjGmdx4/fWoRX/vt26wsKQ87msQ5FYFIOzOoaza/v3QCt5w3hqWby5h6+xzum7NSkyLJfqkIRNohM+Pc8b146YfHMWlQPv/zzGIufuA9tpTuDTuaxCEVgUg71iUnnXsvLuIX546ieM12Trl9Dq8saWzWWElkKgKRds7MOH9CIU9feSxdc9K59KFibvj7AvY2GEZbEpeKQCRBDOySzRP/djSXHNOX3729htPvfIMFG3aFHUvigIpAJIGkpyRzwxkjePjbEyjbW805d7/J795aHXYsCZmKQCQBTRpUwPPfn8xxg7tww5MLuee1FWFHkhCpCEQSVF5GKjO+MZ4zx/Tg5ueWcNcry8KOJCEJ9M5iM8sD7gNGEp0R9lJ3f7ve+lzgD0BhLMuv3P3BIDOJyGciyUnc+rWxRJKMX72wlNo6uOrEQWHHklYW9BATtwPPuftXzCwVaDg7x78Di9z9DDMrAD42s0fcvSrgXCISk5xk/PKrYzAzbn1pKd1y0/jakYVhx5JWFFgRxP61Pxn4FkDsl3vDX/AOZFt0NvEsYDtQE1QmEWlccpJx85dHsaVsL9c/sYBeHTM4ZmB+2LGklQR5jqAfUAI8aGYfmNl9ZpbZYJu7gGHARuAj4Cp33/8s3yISmEhyEr+5cDz9CzKZ/oe5LN9SFnYkaSVBFkEEGA/c4+7jgN3AtQ22ORmYB/QAxgJ3mdm/DKZuZtPMrNjMiktKSgKMLJLYctJTuP+bR5IWSeLSh4rZvltHaRNBkEWwHljv7u/Gns8iWgz1XQI87lHLgVXA0IYv5O4z3b3I3YsKCgoCjCwivTtlcO/FRWwq3ct3/zCXqhrtpLd3gRWBu28C1pnZkNiiKcCiBputjS3HzLoCQ4CVQWUSkeYZV9iR//vyaN5dtZ0bn1pIW5vASg5O0FcNXQk8ErtiaCVwiZlNB3D3GcDPgIfM7CPAgGvcfWvAmUSkGc4e15Mlm8qY8foKhnbL5uKj+oYdSQISaBG4+zyg4dRoM+qt3wicFGQGETl0Pzp5CMs2l/HTpxYxqEs2Rw3oHHYkCYDuLBaR/UpOMm47fyx9Omfwvcc+oKSsMuxIEgAVgYg0KTs9hd9cMJ7SPdX84E/zqNVMZ+2OikBEDmhY9xx+euYI3li+lbtfXR52HGlhKgIRaZavHdmbs8b24NaXlvLOym1hx5EWpCIQkWYxM35+zij6dM7k6r98SEWVRoNpL1QEItJsWWkRbv7yaNbv2MMtLywNO460EBWBiByUCf06ceHEQh54cxUfrtsZdhxpASoCETlo10wdSkF2Gtf8dT7VtRqCoq1TEYjIQctJT+FnZ41kyaYyZs7WqDBtnYpARA7JSSO6ceqobtz+8jJWbd0ddhw5DCoCETlkN54xgrRIEv/1+EcamK4NUxGIyCHrkpPOdVOH8fbKbfyleH3YceQQqQhE5LCcf2RvJvTtxM//sVhjEbVRKgIROSxJScZN545iT1Ut//10wylHpC1QEYjIYRvYJYsrvjiQpz7cyFvLNaVIW6MiEJEWMW1yf3rmdeBnzyzWCKVtjIpARFpEekoy104dyuJPSpk1d13YceQgqAhEpMWcPro74wvz+NULSymv1KB0bYWKQERajJnx49OHU1JWyYzXVoQdR5pJRSAiLWpcYUfOGtuDe+esZMPOPWHHkWZQEYhIi/vPU4biwB0vLQs7ijSDikBEWlzPvA5cMKGQWe+vZ7XGIYp7gRaBmeWZ2SwzW2Jmi83sqEa2Od7M5pnZQjN7Pcg8ItJ6/u2EAaQkG7e/rL2CeBf0HsHtwHPuPhQYAyyuv9LM8oC7gTPdfQTw1YDziEgr6ZKdzsVH9eVv8zawfEtZ2HGkCYEVgZnlApOB+wHcvcrdG05ndAHwuLuvjW2zJag8ItL6Lp/cn4yUZG7VuYK4FuQeQT+gBHjQzD4ws/vMLLPBNoOBjmb2mpnNNbOLA8wjIq2sc1YalxzTj2fmf8LiT0rDjiP7EWQRRIDxwD3uPg7YDVzbyDZHAKcBJwM/NrPBDV/IzKaZWbGZFZeUlAQYWURa2mWT+pOdHuHXmuw+bgVZBOuB9e7+buz5LKLF0HCb5919t7tvBWYTPZfwOe4+092L3L2ooKAgwMgi0tJyM1K4fHJ/Xlq8mblrdoQdRxoRWBG4+yZgnZkNiS2aAjQco/bvwLFmFjGzDGAiDU4oi0jbd8kx/cjPSuWXzy/RTGZxKOirhq4EHjGz+cBY4CYzm25m0wHcfTHwHDAfeA+4z90XBJxJRFpZZlqEK04YyDsrtzNnmYapjjfW1tq5qKjIi4uLw44hIgepsqaWL/7qdTplpvLkFcdgZmFHSihmNtfdixpbpzuLRaRVpEWS+cGXBvPRhl08u2BT2HGkHhWBiLSac8b1ZFCXLH71wsfU1NaFHUdimlUEZpZpZkmxrweb2ZlmlhJsNBFpb5KTjP84aTArS3bzxAcbwo4jMc3dI5gNpJtZT+AF4CLgoaBCiUj7dfKIbozqmcttLy2jsqY27DhC84vA3L0COBe4292/CowILpaItFdmxtUnD2HDzj386Z+a0jIeNLsIYiOHXgg8E1uWHEwkEWnvJg/KZ0LfTtz5ynL2VGmvIGzNLYLvA9cBT7j7QjPrD7waXCwRac/27RWUlFXy+7dXhx0n4TWrCNz9dXc/091vjp003uru3ws4m4i0YxP6deK4wQXc8/oKSvdWhx0noTX3qqE/mllObPTQBcAiM/tRsNFEpL370clD2FlRzczXV4YdJaE199DQcHcvBc4GniU6xPRFgaUSkYQwsmcup4/uzv1vrGJL2d6w4ySs5hZBSuy+gbOBJ929GmhbY1OISFy6+qQhVNfWcefLy8OOkrCaWwS/BVYDmcBsM+sDaJYJETlsffMz+dqRvXn0vbWa6D4kzT1ZfIe793T3Uz1qDXBCwNlEJEFcNWUQKclJ3PKiJq8JQ3NPFuea2S37Zgkzs18T3TsQETlsXXLSufTYvjz54UYWbtwVdpyE09xDQw8AZcB5sUcp8GBQoUQk8UybPICc9Ai3vqiJ7ltbc4tggLvf4O4rY4+fAv2DDCYiiSW3QwqXTYpOaTl//c6w4ySU5hbBHjM7dt8TMzsG2BNMJBFJVN86pi95GSncqnMFraq5RTAd+I2ZrTaz1cBdwOWBpRKRhJSdnsK0yf159eMS3l+rie5bS3OvGvrQ3ccAo4HR7j4O+GKgyUQkIX3zqL50ykzVXkErOqgZyty9NHaHMcAPA8gjIgkuMy3C5ZP7M2fZVv65envYcRLC4UxVqZmnRSQQFx3Vh/ysVO54WVcQtYbDKQINMSEigchIjXDZpOhegc4VBK/JIjCzMjMrbeRRBvQ40IubWZ6ZzTKzJWa2ODa5TWPbHWlmNWb2lUP8OUSknfnGF/rQMSOFO7VXELgmi8Dds909p5FHtrtHmvH6twPPuftQYAywuOEGZpYM3Ex0LmQRESB6ruA7k6JXEOm+gmAdzqGhJplZLjAZuB/A3avcvbE/zSuBvwJbgsoiIm3TxUf1ISc9wp2vaGTSIAVWBETnLCgBHjSzD8zsvtjENp8ys57AOcA9Tb2QmU3bN85RSUlJcIlFJK5kp6dw6bH9eHHRZhZt1IDHQQmyCCLAeOCe2H0Hu4FrG2xzG3CNu9c19ULuPtPdi9y9qKCgIJi0IhKXLjm6H9lpEe56VecKghJkEawH1rv7u7Hns4gWQ31FwGOxu5W/AtxtZmcHmElE2pjcjBQuProPzy7YxPIt5WHHaZcCKwJ33wSsM7MhsUVTgEUNtunn7n3dvS/Rovg3d/9bUJlEpG265Jh+pEWSmPH6irCjtEtB7hFA9ETwI2Y2HxgL3GRm081sesDvKyLtSH5WGucfWcjfPtjAhp0a77KlBVoE7j4vdmx/tLuf7e473H2Gu89oZNtvufusIPOISNt12eToyPf3zl4ZcpL2J+g9AhGRFtEzrwPnjOvJo++tZWt5Zdhx2hUVgYi0GdOPH0BVbR0PvLEq7CjtiopARNqMAQVZTB3ZjYffXsPOiqqw47QbKgIRaVO+N2UQ5VU13KMriFqMikBE2pSh3XI4e2xPHnpzNZt27Q07TrugIhCRNucHJw6mts654xXdbdwSVAQi0uYUds7ggomF/Omf61i1dXfYcdo8FYGItElXfHEgqclJ3KK5jQ+bikBE2qQu2elcemxfnvpwIws27Ao7TpumIhCRNmva5AF0zEjhpn8sxl2z5x4qFYGItFm5HVK4asog3lqxjdc+1lwlh0pFICJt2gUT+9C3cwY3/WMxNbVNTm0i+6EiEJE2LTWSxLVTh7JsSzl/mbs+7DhtkopARNq8k0d0o6hPR379wlJ2V9aEHafNURGISJtnZlx/2jC2lldq8ppDoCIQkXZhXGFHzhzTg5mzV7Jue0XYcdoUFYGItBvXTh2KGfzi2SVhR2lTVAQi0m70yOvAd48byDMffcI7K7eFHafNUBGISLsybXJ/euZ14KdPLaK2TjeZNYeKQETalQ6pyVx36lAWf1LKn/65Luw4bYKKQETandNGdWdC307c8uLHupy0GQItAjPLM7NZZrbEzBab2VEN1l9oZvPN7CMze8vMxgSZR0QSg5lx7alD2VpexX1zNL/xgQS9R3A78Jy7DwXGAIsbrF8FHOfuo4CfATMDziMiCWJ8YUemjuzGzNkr2FpeGXacuBZYEZhZLjAZuB/A3avcfWf9bdz9LXffEXv6DtArqDwikniuPnkIe2vquPNlzWTWlCD3CPoBJcCDZvaBmd1nZplNbP9t4NkA84hIghlQkMX5R/bmkXfXslozme1XkEUQAcYD97j7OGA3cG1jG5rZCUSL4Jr9rJ9mZsVmVlxSoqFmRaT5rpoyiJTkJH75wsdhR4lbQRbBemC9u78bez6LaDF8jpmNBu4DznL3Ru8AcfeZ7l7k7kUFBQWBBRaR9qdLTjqXTerHM/M/Yd66nQf+hgQUWBG4+yZgnZkNiS2aAiyqv42ZFQKPAxe5uyYeFZFATDtuAPlZqdz0jGYya0zQVw1dCTxiZvOBscBNZjbdzKbH1v8E6AzcbWbzzKw44DwikoCy0iJ8/8TBvLd6Oy8u2hx2nLhjba0di4qKvLhYfSEiB6emto6Tb5uNA89/fzIpyYl1P62ZzXX3osbWJdYnISIJK5KcxHVTh7GyZDePaeiJz1ERiEjCmDKsCxP7deK2F5dStrc67DhxQ0UgIgnDzPh/pw1ne0UVd76yPOw4cUNFICIJZVSvXM47ojcPvLGK5VvKw44TF1QEIpJwfnTKEDqkJvPTpxbqclJUBCKSgPKz0vjhlwYzZ9lWXtDlpCoCEUlM3/hCHwZ3zeJnTy9ib3Vt2HFCpSIQkYSUkpzEjWeOYP2OPcycvTLsOKFSEYhIwjp6QD6njurG3a8tZ+POPWHHCY2KQEQS2nVTh+EOv3h2SdhRQqMiEJGE1rtTBpdP7s+TH27kn6u3hx0nFCoCEUl4048fQPfcdG58ciG1dYl3OamKQEQSXkZqhOtOHcbCjaX8pTjxxiFSEYiIAGeM7s6RfTvyy+c/pjTBxiFSEYiIEB2H6IYzRkTHIUqwye5VBCIiMSN7RschevDN1awoSZxxiFQEIiL1XH3yENJTkvn5M4vDjtJqVAQiIvUUZKfxvSkDeWXJFl79eEvYcVqFikBEpIFvHd2PfvmZ/OzpRVTX1oUdJ3AqAhGRBlIjSfzXqdFpLR99b23YcQKnIhARacSJw7rwhf6duO2lZe3+clIVgYhII/ZNa7mjoorfvNq+p7UMtAjMLM/MZpnZEjNbbGZHNVhvZnaHmS03s/lmNj7IPCIiB2Nkz1zOGdeTB99YzbrtFWHHCUzQewS3A8+5+1BgDNDweqypwKDYYxpwT8B5REQOytUnDcEMfvn8x2FHCUxgRWBmucBk4H4Ad69y950NNjsL+L1HvQPkmVn3oDKJiBysHnkduGxSdHTSlxe3z2ktg9wj6AeUAA+a2Qdmdp+ZZTbYpidQf4Sn9bFln2Nm08ys2MyKS0pKgkssItKIK744kJE9c/j+Y/NYvqX93XEcZBFEgPHAPe4+DtgNXHsoL+TuM929yN2LCgoKWjKjiMgBpack89uLikiNJDHt4eJ2dxVRkEWwHljv7u/Gns8iWgz1bQB613veK7ZMRCSu9MzrwN0Xjmfttgp+8Ng86trRvAWBFYG7bwLWmdmQ2KIpwKIGmz0JXBy7eugLwC53/ySoTCIih2Ni/8785IzhvLxkC/e8viLsOC0mEvDrXwk8YmapwErgEjObDuDuM4B/AKcCy4EK4JKA84iIHJaLvtCHd1du57aXlnL8kAJG9MgNO9JhM/e2tXtTVFTkxcXFYccQkQS2Y3cVJ982m44Zqfz9imNIT0kOO9IBmdlcdy9qbJ3uLBYROUgdM1O5+Suj+XhzGbe8uDTsOIdNRSAicghOGNKFCyYWcu+clbyzclvYcQ6LikBE5BBdf+ow+nTK4KrHPqCkrDLsOIdMRSAicogy0yLcfeER7NpTzRV/fJ+aNjp3gYpAROQwDO+Rw/+eO4p3V23n5ueWhB3nkKgIREQO0znjevHNo/pw75xVPD1/Y9hxDpqKQESkBVx/2nCO6NOR/5w1nyWbSlv89fdU1bb4a+6jIhARaQGpkSTuuXA8WWkRLvt9MTt2V7XYa1fW1DL19tnc+fKyFnvN+lQEIiItpEtOOr+96Ag276rkikdb7uTxw2+vYfW2Ckb3zmuR12tIRSAi0oLGFXbkf84ZyZvLt/G/zx7+yeNdFdXc+cpyJg3K57jBwYy+HPRYQyIiCee8ot4s3LCL+99YRWZahB+cOAgzO6TXuvu15ZTurea6qcNaOOVnVAQiIgH4yRkjqKiq5Y6Xl1G+t4Yfnz7soMtg/Y4KHnxrNeeO68XwHjkBJVURiIgEIjnJuPnLo8lKj/DAm6sor6zmpnNGEUlu/hH5X7+wFAP+46TBwQVFRSAiEpikJOMnpw8nOz2FO15expJNZdz85dEM637gf93/fd4GnvhgA989fgA98joEmzPQVxcRSXBmxg+/NJi7LhjHhh17OOPON/j1Cx+zt3r/9wXcO3slVz02jy/078QVJwwMPKP2CEREWsHpo3twzIB8fvbMIu58ZTl/Ll7Hd47tz9cnFpKVFv1VXFVTx83PLeH+N1Zx2uju3HLeGNIiwc91oIlpRE7OLz0AAAhySURBVERa2VsrtnLXK8t5a8U2ctIjjOmdx9rtFazbXkGdwyXH9OXHpw0nKenQrjRqTFMT02iPQESklR09IJ+jB+Qzb91OZs5eweqtFYzsmctZY3owpnceXxza5ZAvNz0UKgIRkZCM7Z3H3RceEXYMnSwWEUl0KgIRkQQX6KEhM1sNlAG1QE3DExVmlgv8ASiMZfmVuz8YZCYREfm81jhHcIK7b93Pun8HFrn7GWZWAHxsZo+4e8uN3yoiIk0K+9CQA9kWPT2eBWwHasKNJCKSWIIuAgdeMLO5ZjatkfV3AcOAjcBHwFXu3jZnfxYRaaOCLoJj3X08MBX4dzOb3GD9ycA8oAcwFrjLzP5lEA4zm2ZmxWZWXFJSEnBkEZHEEmgRuPuG2H+3AE8AExpscgnwuEctB1YBQxt5nZnuXuTuRQUFwUzMICKSqAI7WWxmmUCSu5fFvj4J+O8Gm60FpgBzzKwrMARY2dTrzp07d6uZrYk9zQV2NfF1w2UpwP5OXO9P/ddozrqGy5qbcd9/8w8yY2vl27dMn2F85WsLGeM93+FkbGpZvH2Gffb7zu4eyAPoD3wYeywEro8tnw5Mj33dA3iB6PmBBcA3DvI9Zjb1dcNlQPEh/BwzD2Zdw2XNzVjvvweVsbXy6TOMz3xtIWO85zucjAfIGlefYVOPwPYI3H0lMKaR5TPqfb2R6J7CoXrqAF83XDb+MN+jOesaLmtuxqbepymtlW/f1/oM4yvf/tbHU8Z4z7e/9c3JeKBlByPoz3C/2tzoo4fDzIp9P6PvxYt4zxjv+SD+M8Z7Poj/jPGeD9pGxn3Cvo+gtc0MO0AzxHvGeM8H8Z8x3vNB/GeM93zQNjICCbZHICIi/yrR9ghERKQBFYGISIJTEYiIJDgVQYyZTTKzGWZ2n5m9FXaexphZkpn93MzuNLNvhp2nITM73szmxD7H48PO0xgzy4wNV3J62FkaY2bDYp/fLDP7bth5GmNmZ5vZvWb2JzM7nMu/A2Fm/c3sfjObFXaWfWJ/734X+9wuDDtPQ+2iCMzsATPbYmYLGiw/xcw+NrPlZnZtU6/h7nPcfTrwNPC7eMwInAX0AqqB9XGYz4FyID1O8wFcA/y5JbO1ZEZ3Xxz7e3gecEycZvybu19G9ObQr8VhvpXu/u2WzNWYg8x6LjAr9rmdGXS2g3Ywd77F6wOYTPRGpwX1liUDK4je4ZxK9A7n4cAoor/s6z+61Pu+PwPZ8ZgRuBa4PPa9s+IwX1Ls+7oCj8Rhvi8B5wPfAk6Pxz/j2PecCTwLXBCvGWPf92tgfBzna9H/Rw4z63XA2Ng2fwwy16E82sXk9e4+28z6Nlg8AVju0TucMbPHgLPc/X+BRg8LmFkhsMvdy+Ixo5mtB/ZN2lMbb/nq2QGkxVu+2OGqTKL/Y+4xs394Cw573lKfobs/CTxpZs8Af2ypfC2VMTZ/yC+AZ939/XjL11oOJivRPeReREdbjrsjMe2iCPajJ7Cu3vP1wMQDfM+3gdacKvNgMz4O3Glmk4DZQQaLOah8ZnYu0aHF84jONRG0g8rn7tcDmNm3gK0tWQJNONjP8HiihxHSgH8EmuwzB/v38ErgRCDXzAZ6vWFjAnKwn2Fn4OfAODO7LlYYrWV/We8gOsz+aRz6EBSBac9FcNDc/YawMzTF3SuIllVccvfHiZZVXHP3h8LOsD/u/hrwWsgxmuTudxD9xRaX3H0b0fMXccPddxMddj8uxd0uSgvaAPSu97xXbFk8ifeMynf4lPHwxXu++tpS1k+15yL4JzDIzPqZWSrRk4RPhpypoXjPqHyHTxkPX7znq68tZf1M2GerW+js/aPAJ3x2WeW3Y8tPBZYSPYt/vTIqnzLGd8Z4z9dWsx7ooUHnREQSXHs+NCQiIs2gIhARSXAqAhGRBKciEBFJcCoCEZEEpyIQEUlwKgJpF8ysvJXfr0XmrLDoHA67zGyemS0xs18143vONrPhLfH+IqAiEGmUmTU5Dpe7H92CbzfH3ccC44DTzexA8xCcTXQEVZEWoSKQdsvMBpjZc2Y216Izpw2NLT/DzN41sw/M7CUz6xpbfqOZPWxmbwIPx54/YGavmdlKM/tevdcuj/33+Nj6WbF/0T8SG6YZMzs1tmyumd1hZk83ldfd9xAdprhn7PsvM7N/mtmHZvZXM8sws6OJzlfwy9hexID9/ZwizaUikPZsJnClux8BXA3cHVv+BvAFdx8HPAb8Z73vGQ6c6O5fjz0fSnRo7QnADWaW0sj7jAO+H/ve/sAxZpYO/BaYGnv/ggOFNbOOwCA+G2L8cXc/0t3HAIuJDmHwFtGxa37k7mPdfUUTP6dIs2gYammXzCwLOBr4S+wf6PDZZDm9gD+ZWXeis0itqvetT8b+Zb7PM+5eCVSa2Rais681nIbzPXdfH3vfeUBfolN2rnT3fa/9KDBtP3EnmdmHREvgNnffFFs+0sz+h+j8DlnA8wf5c4o0i4pA2qskYGfs2HtDdwK3uPuTsYlgbqy3bneDbSvrfV1L4//PNGebpsxx99PNrB/wjpn92d3nAQ8BZ7v7h7HJdI5v5Hub+jlFmkWHhqRdcvdSYJWZfRWi0yua2ZjY6lw+GyP+mwFF+BjoX28qwwNO8h7be/gFcE1sUTbwSexw1IX1Ni2LrTvQzynSLCoCaS8yzGx9vccPif7y/HbssMtConPHQnQP4C9mNhfYGkSY2OGlfwOei71PGbCrGd86A5gcK5AfA+8CbwJL6m3zGPCj2MnuAez/5xRpFg1DLRIQM8ty9/LYVUS/AZa5+61h5xJpSHsEIsG5LHbyeCHRw1G/DTmPSKO0RyAikuC0RyAikuBUBCIiCU5FICKS4FQEIiIJTkUgIpLgVAQiIgnu/wOyThDvXv7E4wAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Rihr5Xw0TQB",
        "outputId": "e0f1e58f-7422-469f-b8e8-a121e250acd4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 94
        }
      },
      "source": [
        "# train last layers first\n",
        "learn.fit_one_cycle(1, 4e-2)"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>perplexity</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>4.713461</td>\n",
              "      <td>4.021697</td>\n",
              "      <td>0.327076</td>\n",
              "      <td>55.795715</td>\n",
              "      <td>00:20</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Better model found at epoch 0 with valid_loss value: 4.021697044372559.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J7hXL_FL0Xht",
        "outputId": "84015cde-1459-4ee1-9133-31f75c7cc1e7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 404
        }
      },
      "source": [
        "# train the whole network with a smaller learning rate\n",
        "learn.unfreeze()\n",
        "learn.fit_one_cycle(8, 4e-3)"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>perplexity</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>3.942059</td>\n",
              "      <td>3.778240</td>\n",
              "      <td>0.353193</td>\n",
              "      <td>43.738991</td>\n",
              "      <td>00:21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>3.783178</td>\n",
              "      <td>3.586052</td>\n",
              "      <td>0.377635</td>\n",
              "      <td>36.091293</td>\n",
              "      <td>00:20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>3.586731</td>\n",
              "      <td>3.460168</td>\n",
              "      <td>0.392891</td>\n",
              "      <td>31.822311</td>\n",
              "      <td>00:20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>3.407196</td>\n",
              "      <td>3.393142</td>\n",
              "      <td>0.403265</td>\n",
              "      <td>29.759310</td>\n",
              "      <td>00:20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>3.252400</td>\n",
              "      <td>3.363641</td>\n",
              "      <td>0.408120</td>\n",
              "      <td>28.894197</td>\n",
              "      <td>00:20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>3.113715</td>\n",
              "      <td>3.356634</td>\n",
              "      <td>0.411461</td>\n",
              "      <td>28.692461</td>\n",
              "      <td>00:21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>6</td>\n",
              "      <td>3.002027</td>\n",
              "      <td>3.357580</td>\n",
              "      <td>0.412802</td>\n",
              "      <td>28.719618</td>\n",
              "      <td>00:20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>7</td>\n",
              "      <td>2.932654</td>\n",
              "      <td>3.361399</td>\n",
              "      <td>0.412839</td>\n",
              "      <td>28.829493</td>\n",
              "      <td>00:20</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Better model found at epoch 0 with valid_loss value: 3.7782399654388428.\n",
            "Better model found at epoch 1 with valid_loss value: 3.5860517024993896.\n",
            "Better model found at epoch 2 with valid_loss value: 3.460167646408081.\n",
            "Better model found at epoch 3 with valid_loss value: 3.393141984939575.\n",
            "Better model found at epoch 4 with valid_loss value: 3.363640785217285.\n",
            "Better model found at epoch 5 with valid_loss value: 3.3566343784332275.\n",
            "No improvement since epoch 5: early stopping\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oTSmWpKrz3Xq"
      },
      "source": [
        "# Save encoder. To be used by text classifier learner.\n",
        "learn.save_encoder('lm_finetune_final_enc')"
      ],
      "execution_count": 67,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "txS6EOJW0m3V",
        "outputId": "3290c1b3-4e2e-42a6-84c9-6dbe81a56e6b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "# This is totally unrelated to training process but just for fun...\n",
        "# you can try generating text with the language model here\n",
        "TEXT = \"Ako ay\"\n",
        "N_WORDS = 40\n",
        "N_SENTENCES = 2\n",
        "preds = [learn.predict(TEXT, N_WORDS, temperature=0.75) \n",
        "         for _ in range(N_SENTENCES)]\n",
        "\n",
        "print(\"\\n\".join(preds))"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Ako ay si Binay ? Maganda pa yung mga ginagawa ni Binay . \" Binay Ad Nog Nog ? ? Mahirap ka Binay ! # wantedpresident Duterte ,\n",
            "Ako ay isang \" pandak \" sa Makati . Ang nognog yun ? Wtf ? ? Knya suportado ng mga mar roxas sa Marawi Lanao del Sur … Hahahaha Mar Roxas\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "swgJLIMAwOkJ"
      },
      "source": [
        "# Text Classifier Fine-tuning\n",
        "\n",
        "Here, we use the encoder of the fine-tuned language model to transfer the learnings to the classifier model. The model will learn to classify the text to binary labels, hate (1) or non-hate (0).\n",
        "\n",
        "Here, we'll apply gradual unfreezing and discriminative learning rates as discussed by [(Howard and Ruder, 2018)](https://arxiv.org/abs/1801.06146)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KzAQlUWX9YRs"
      },
      "source": [
        "# HYPERPARAMETERS\n",
        "lr = 5e-2\n",
        "wd = 0.1\n",
        "moms = (0.8,0.7,0.6)"
      ],
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mrhb-WiQauZL",
        "outputId": "a73057d2-f1b8-4e9b-c965-84d17a08afac",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "# Exclude the test set\n",
        "clas_df_10k = pd.concat([train_df, valid_df])\n",
        "clas_df_10k.shape # 10k + valid set"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(14232, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f9ufQ2u6wQmP",
        "outputId": "36cfd16e-f74f-4c97-84dc-fd405549b212",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 16
        }
      },
      "source": [
        "# Create a dataloader for text classifier fine-tuning\n",
        "dls_clas = DataBlock(blocks=(TextBlock.from_df('text', seq_len=72, min_freq=2, vocab=dls_lm.vocab), CategoryBlock),\n",
        "                      get_x=ColReader('text'),\n",
        "                      get_y=ColReader('label'),\n",
        "                      splitter=ColSplitter()\n",
        "                      ).dataloaders(clas_df_10k, bs=128, num_workers=0)"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CrGXu5ova6dw"
      },
      "source": [
        "# Set random seed manually for reproducibility\n",
        "set_random_seed(2014614837)"
      ],
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mUmhI3xZx0jH",
        "outputId": "dfa5952a-6e3e-45a5-a9c6-6d9a2be12e3c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "# Create learner\n",
        "learn = text_classifier_learner(dls_clas, AWD_LSTM, moms=moms, wd=wd, metrics=accuracy).to_fp16()\n",
        "\n",
        "# Load encoder of fine-tuned language model\n",
        "learn.load_encoder('lm_finetune_final_enc')"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<fastai.text.learner.TextLearner at 0x7f0a5544bd30>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mP1vrjb3yrDf",
        "outputId": "be90f299-dff9-4882-c9ec-083512732507",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 167
        }
      },
      "source": [
        "# Train the last layers\n",
        "learn.fit_one_cycle(4, lr) "
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.696004</td>\n",
              "      <td>0.557743</td>\n",
              "      <td>0.708412</td>\n",
              "      <td>00:19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.672074</td>\n",
              "      <td>0.521690</td>\n",
              "      <td>0.733459</td>\n",
              "      <td>00:19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.654777</td>\n",
              "      <td>0.543808</td>\n",
              "      <td>0.701796</td>\n",
              "      <td>00:19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.636296</td>\n",
              "      <td>0.513312</td>\n",
              "      <td>0.737004</td>\n",
              "      <td>00:19</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iD2eS4EGzM8D",
        "outputId": "aca05486-5f7c-4b39-9dd5-e4ab91f4f7ef",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        }
      },
      "source": [
        "# Unfreeze a little bit\n",
        "learn.freeze_to(-2)\n",
        "\n",
        "# Decrease the learning rate\n",
        "learn.fit_one_cycle(2, slice(lr/(2.6**4),lr)) "
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.603289</td>\n",
              "      <td>0.529903</td>\n",
              "      <td>0.721881</td>\n",
              "      <td>00:20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.541825</td>\n",
              "      <td>0.498178</td>\n",
              "      <td>0.745983</td>\n",
              "      <td>00:20</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SX46acsmzPNA",
        "outputId": "dbee2e9b-2c42-41e5-db21-9e65a15a3917",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        }
      },
      "source": [
        "# Unfreeze a little bit more\n",
        "learn.freeze_to(-3)\n",
        "\n",
        "# Decrease the learning rate more\n",
        "learn.fit_one_cycle(2, slice(lr/2/(2.6**4),lr/2)) "
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.520084</td>\n",
              "      <td>0.484902</td>\n",
              "      <td>0.757089</td>\n",
              "      <td>00:21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.450389</td>\n",
              "      <td>0.478982</td>\n",
              "      <td>0.770085</td>\n",
              "      <td>00:21</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JlZeBs_TzSof",
        "outputId": "80b6aa82-52c0-460d-dcf2-fd02a0cc25a2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 77
        }
      },
      "source": [
        "# Unfreeze the whole network\n",
        "learn.unfreeze()\n",
        "\n",
        "# Train the whole network with really small learning rate\n",
        "learn.fit_one_cycle(1, slice(lr/10/(2.6**4),lr/10)) "
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.381500</td>\n",
              "      <td>0.492208</td>\n",
              "      <td>0.769376</td>\n",
              "      <td>00:22</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y_pxmGaCzcFh",
        "outputId": "2623d682-20e7-4eb0-d1f5-b449f5f5e184",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "# Get accuracy with test set\n",
        "acc = get_test_acc(learn)\n",
        "acc"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7573251417769377"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OBZ6e8hxmMlk"
      },
      "source": [
        "Even though the purpose of this notebook is to show you how to use the pre-trained model in text classification tasks, it's worth noting that there's a slight discrepancy in reported accuracy and the accuracy printed in this notebook. In this notebook, we didn't use the same exact encoder which explains the slight difference in results. In this case, seeds used to initialize the model plays a small role in the accuracy. The accuracy ranges from 75 to 76.9% with different seeds."
      ]
    }
  ]
}