Day_44_LSTM_with_Keras.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNXaCYmQlsudht0xgr69Sdm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "An LSTM layer consists of a set of recurrently connected blocks, known as memory blocks. These blocks can be thought of as a differentiable version of the memory chips in a digital computer. Each one contains one or more recurrently connected memory cells and three multiplicative units — the input, output and forget gates — that provide continuous analogues of write, read and reset operations for the cells. … The net can only interact with the cells via the gates.”"
      ],
      "metadata": {
        "id": "7HOXH2IPSiCZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install livelossplot"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fi3m-BkGXzAv",
        "outputId": "beb290a4-2def-4801-f9d5-8864db8fa9b1"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting livelossplot\n",
            "  Downloading livelossplot-0.5.5-py3-none-any.whl (22 kB)\n",
            "Requirement already satisfied: bokeh in /usr/local/lib/python3.7/dist-packages (from livelossplot) (2.3.3)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from livelossplot) (3.2.2)\n",
            "Requirement already satisfied: ipython==7.* in /usr/local/lib/python3.7/dist-packages (from livelossplot) (7.9.0)\n",
            "Requirement already satisfied: numpy<1.22 in /usr/local/lib/python3.7/dist-packages (from livelossplot) (1.21.6)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython==7.*->livelossplot) (2.6.1)\n",
            "Requirement already satisfied: pexpect in /usr/local/lib/python3.7/dist-packages (from ipython==7.*->livelossplot) (4.8.0)\n",
            "Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.7/dist-packages (from ipython==7.*->livelossplot) (5.1.1)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython==7.*->livelossplot) (4.4.2)\n",
            "Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from ipython==7.*->livelossplot) (2.0.10)\n",
            "Collecting jedi>=0.10\n",
            "  Downloading jedi-0.18.1-py2.py3-none-any.whl (1.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.6 MB 20.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: backcall in /usr/local/lib/python3.7/dist-packages (from ipython==7.*->livelossplot) (0.2.0)\n",
            "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/dist-packages (from ipython==7.*->livelossplot) (57.4.0)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython==7.*->livelossplot) (0.7.5)\n",
            "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from jedi>=0.10->ipython==7.*->livelossplot) (0.8.3)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython==7.*->livelossplot) (1.15.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython==7.*->livelossplot) (0.2.5)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from bokeh->livelossplot) (4.1.1)\n",
            "Requirement already satisfied: pillow>=7.1.0 in /usr/local/lib/python3.7/dist-packages (from bokeh->livelossplot) (7.1.2)\n",
            "Requirement already satisfied: packaging>=16.8 in /usr/local/lib/python3.7/dist-packages (from bokeh->livelossplot) (21.3)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from bokeh->livelossplot) (2.8.2)\n",
            "Requirement already satisfied: PyYAML>=3.10 in /usr/local/lib/python3.7/dist-packages (from bokeh->livelossplot) (6.0)\n",
            "Requirement already satisfied: Jinja2>=2.9 in /usr/local/lib/python3.7/dist-packages (from bokeh->livelossplot) (2.11.3)\n",
            "Requirement already satisfied: tornado>=5.1 in /usr/local/lib/python3.7/dist-packages (from bokeh->livelossplot) (5.1.1)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from Jinja2>=2.9->bokeh->livelossplot) (2.0.1)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=16.8->bokeh->livelossplot) (3.0.9)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->livelossplot) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->livelossplot) (1.4.4)\n",
            "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.7/dist-packages (from pexpect->ipython==7.*->livelossplot) (0.7.0)\n",
            "Installing collected packages: jedi, livelossplot\n",
            "Successfully installed jedi-0.18.1 livelossplot-0.5.5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "fKeDliuISVAF"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "np.random.seed(0)\n",
        "plt.style.use(\"ggplot\")\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "from sklearn.model_selection import train_test_split\n",
        "from tensorflow.keras import Model, Input\n",
        "from tensorflow.keras.layers import LSTM, Embedding, Dense\n",
        "from tensorflow.keras.layers import TimeDistributed, SpatialDropout1D, Bidirectional\n",
        "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
        "from livelossplot.tf_keras import PlotLossesCallback"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Load and Explore the Data"
      ],
      "metadata": {
        "id": "DZVMPtN2SmDa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data = pd.read_csv('https://github.com/raw/thunderstroke325/60-Days-of-Data-Science-and-ML/main/datasets/data44.csv',encoding ='latin1')\n",
        "data = data.fillna(method = 'ffill')\n",
        "# Unique words\n",
        "print(data['Word'].nunique())\n",
        "print(data['Tag'].nunique())\n",
        "words = list(set(data['Word'].values))\n",
        "words.append('ENDPAD')\n",
        "num_words = len(words)\n",
        "tags= list(set(data['Tag'].values))\n",
        "num_tags = len(tags)\n",
        "num_words, num_tags"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tH0XIFglSpXA",
        "outputId": "2361aee6-884e-4437-f8ba-5cd21064642e"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "35178\n",
            "17\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(35179, 17)"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Retrieve Sentences and Tags"
      ],
      "metadata": {
        "id": "V83c45KDSnyw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class sg(object):\n",
        "    def __init__(self,data):\n",
        "        self.n_sent = 1\n",
        "        self.data = data\n",
        "        af = lambda s: [(w,p,t) for w,p,t in zip(s['Word'].values.tolist(),\n",
        "                                                s['POS'].values.tolist(),\n",
        "                                                s['Tag'].values.tolist())]\n",
        "        self.grouped = self.data.groupby('Sentence #').apply(af)\n",
        "        self.sentences = [s for s in self.grouped]\n",
        "g = sg(data)\n",
        "s = g.sentences\n",
        "s[2]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1L5JIzqNSv0r",
        "outputId": "bb3f98a8-bc09-4686-8aba-a86f30d6767e"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Helicopter', 'NN', 'O'),\n",
              " ('gunships', 'NNS', 'O'),\n",
              " ('Saturday', 'NNP', 'B-tim'),\n",
              " ('pounded', 'VBD', 'O'),\n",
              " ('militant', 'JJ', 'O'),\n",
              " ('hideouts', 'NNS', 'O'),\n",
              " ('in', 'IN', 'O'),\n",
              " ('the', 'DT', 'O'),\n",
              " ('Orakzai', 'NNP', 'B-geo'),\n",
              " ('tribal', 'JJ', 'O'),\n",
              " ('region', 'NN', 'O'),\n",
              " (',', ',', 'O'),\n",
              " ('where', 'WRB', 'O'),\n",
              " ('many', 'JJ', 'O'),\n",
              " ('Taliban', 'NNP', 'B-org'),\n",
              " ('militants', 'NNS', 'O'),\n",
              " ('are', 'VBP', 'O'),\n",
              " ('believed', 'VBN', 'O'),\n",
              " ('to', 'TO', 'O'),\n",
              " ('have', 'VB', 'O'),\n",
              " ('fled', 'VBN', 'O'),\n",
              " ('to', 'TO', 'O'),\n",
              " ('avoid', 'VB', 'O'),\n",
              " ('an', 'DT', 'O'),\n",
              " ('earlier', 'JJR', 'O'),\n",
              " ('military', 'JJ', 'O'),\n",
              " ('offensive', 'NN', 'O'),\n",
              " ('in', 'IN', 'O'),\n",
              " ('nearby', 'JJ', 'O'),\n",
              " ('South', 'NNP', 'B-geo'),\n",
              " ('Waziristan', 'NNP', 'I-geo'),\n",
              " ('.', '.', 'O')]"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Mappings"
      ],
      "metadata": {
        "id": "MoxU06cgSoqi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "wi = {w: i+1 for i,w in enumerate(words)}\n",
        "ti = {t: i for i,t, in enumerate(tags)}\n",
        "wi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qWUkDvjZSzJp",
        "outputId": "ede8f4d7-a4f4-463d-eb0f-c5a27e88a438"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'transitional': 1,\n",
              " 'sculptures': 2,\n",
              " 'Alasay': 3,\n",
              " 'Ireland': 4,\n",
              " 'half-a-degree': 5,\n",
              " 'staving': 6,\n",
              " 'Aridi': 7,\n",
              " 'appeals': 8,\n",
              " 'coltan': 9,\n",
              " 'Herceptin': 10,\n",
              " 'Katarina': 11,\n",
              " 'Akash': 12,\n",
              " 'forensics': 13,\n",
              " 'recipients': 14,\n",
              " 'flour': 15,\n",
              " 'Clarkson': 16,\n",
              " 'Secretariat': 17,\n",
              " 'vanilla': 18,\n",
              " 'Maulana': 19,\n",
              " 'Ravi': 20,\n",
              " 'tarnishing': 21,\n",
              " 'tire': 22,\n",
              " 'circumventing': 23,\n",
              " 'Dagger': 24,\n",
              " 'SHEVARDNADZE': 25,\n",
              " 'YouTube': 26,\n",
              " 'Abd': 27,\n",
              " 'Brigades': 28,\n",
              " 'Sum-41': 29,\n",
              " 'Vejjajiva': 30,\n",
              " 'Community': 31,\n",
              " 'Flavia': 32,\n",
              " 'technical': 33,\n",
              " 'nullified': 34,\n",
              " 'barricade': 35,\n",
              " 'charming': 36,\n",
              " 'passwords': 37,\n",
              " 'Madagonians': 38,\n",
              " 'Hen': 39,\n",
              " 'geography': 40,\n",
              " 'Gilgit': 41,\n",
              " '14-year-old': 42,\n",
              " 'licenses': 43,\n",
              " 'six-thousand': 44,\n",
              " 'Harvard': 45,\n",
              " '111th': 46,\n",
              " 'draining': 47,\n",
              " 'Pulwama': 48,\n",
              " 'Naxalites': 49,\n",
              " 'cross-strait': 50,\n",
              " '180,000-employee': 51,\n",
              " 'Knee': 52,\n",
              " 'Protest': 53,\n",
              " 'absorbed': 54,\n",
              " 'addressing': 55,\n",
              " 'Guillaume': 56,\n",
              " 'Olympio': 57,\n",
              " 'pro-Democracy': 58,\n",
              " 'revered': 59,\n",
              " 'implacable': 60,\n",
              " 'predominantly': 61,\n",
              " 'Zap': 62,\n",
              " 'Asians': 63,\n",
              " 'viewers': 64,\n",
              " '2,500': 65,\n",
              " 'partnered': 66,\n",
              " 'European-built': 67,\n",
              " 'Dunem': 68,\n",
              " 'awesome': 69,\n",
              " 'Hodan': 70,\n",
              " 'such': 71,\n",
              " 'POLITICAL': 72,\n",
              " '722': 73,\n",
              " 'circulating': 74,\n",
              " 'disrupting': 75,\n",
              " 'comment': 76,\n",
              " 'Walheim': 77,\n",
              " 'free-trade': 78,\n",
              " 'Nestrenko': 79,\n",
              " 'through': 80,\n",
              " 'militancy': 81,\n",
              " 'outage': 82,\n",
              " '64.5': 83,\n",
              " 'steward': 84,\n",
              " 'Obrador': 85,\n",
              " 'smoldering': 86,\n",
              " 'Peace': 87,\n",
              " 'well-developed': 88,\n",
              " 'Dunleavy': 89,\n",
              " 'Jalawla': 90,\n",
              " 'wisest': 91,\n",
              " 'Schalkwyk': 92,\n",
              " 'earth': 93,\n",
              " '13,000': 94,\n",
              " 'authorizes': 95,\n",
              " 'Syrian-born': 96,\n",
              " 'Ou': 97,\n",
              " 'reign': 98,\n",
              " 'al-Fagih': 99,\n",
              " 'Shanwei': 100,\n",
              " 'booked': 101,\n",
              " 'rose-tinted': 102,\n",
              " 'prosper': 103,\n",
              " 'joins': 104,\n",
              " 'streetcar': 105,\n",
              " 'newly-formed': 106,\n",
              " 'Cowell': 107,\n",
              " '71-26': 108,\n",
              " 'chase': 109,\n",
              " 'Anna': 110,\n",
              " 'Channel': 111,\n",
              " 'squad': 112,\n",
              " 'one-meter': 113,\n",
              " 'Radhika': 114,\n",
              " '52nd': 115,\n",
              " 'Adde': 116,\n",
              " 'Topics': 117,\n",
              " 'acquired': 118,\n",
              " 'truck': 119,\n",
              " 'utilities': 120,\n",
              " 'traveled': 121,\n",
              " 'collapsed': 122,\n",
              " 'legally': 123,\n",
              " 'origami': 124,\n",
              " 'aegis': 125,\n",
              " 'Ganji': 126,\n",
              " 'intermediary': 127,\n",
              " 'Frogs': 128,\n",
              " 'Oleg': 129,\n",
              " '2,700': 130,\n",
              " 'turboprop': 131,\n",
              " 'Dong-Young': 132,\n",
              " 'philosophical': 133,\n",
              " 'Win': 134,\n",
              " 'Indonesians': 135,\n",
              " 'non-American': 136,\n",
              " 'Natanz': 137,\n",
              " 'Buttiglione': 138,\n",
              " '95,000': 139,\n",
              " 'Udi': 140,\n",
              " 'Cosmonauts': 141,\n",
              " 'Asagoe': 142,\n",
              " 'Chilean': 143,\n",
              " 'Myuran': 144,\n",
              " 'denounces': 145,\n",
              " 'Maathai': 146,\n",
              " 'mobilize': 147,\n",
              " 'pals': 148,\n",
              " 'Faeldon': 149,\n",
              " 'Buffett': 150,\n",
              " 'Planning': 151,\n",
              " 'Sherpa': 152,\n",
              " 'Crnojevic': 153,\n",
              " 'Seafarers': 154,\n",
              " 'Ningxia': 155,\n",
              " 'sunspots': 156,\n",
              " 'Olaf': 157,\n",
              " 'Gnassingbe': 158,\n",
              " 'Constituent': 159,\n",
              " 'Fadela': 160,\n",
              " 'Beit': 161,\n",
              " 'place': 162,\n",
              " 'cope': 163,\n",
              " 'Seventeen': 164,\n",
              " 'Nations-sponsored': 165,\n",
              " 'Banaue': 166,\n",
              " 'squalid': 167,\n",
              " 'Regiment': 168,\n",
              " 'Khursheed': 169,\n",
              " 'militarized': 170,\n",
              " '2,400': 171,\n",
              " 'Mar-93': 172,\n",
              " 'particularly': 173,\n",
              " 'video-sharing': 174,\n",
              " 'knew': 175,\n",
              " 'massively': 176,\n",
              " 'fire': 177,\n",
              " 'three': 178,\n",
              " 'Vera': 179,\n",
              " 'Pensacola': 180,\n",
              " 'Taliban-led': 181,\n",
              " 'Alwan': 182,\n",
              " 'Drik': 183,\n",
              " 'CPPCC': 184,\n",
              " 'pullback': 185,\n",
              " 'basement': 186,\n",
              " 'EEBC': 187,\n",
              " 'Montreal': 188,\n",
              " 'impetus': 189,\n",
              " 'Reserve': 190,\n",
              " 'jagged': 191,\n",
              " 'lingered': 192,\n",
              " 'freelance': 193,\n",
              " 'Maastricht': 194,\n",
              " 'popular': 195,\n",
              " 'Kojo': 196,\n",
              " 'Airlift': 197,\n",
              " 'heater': 198,\n",
              " 'model': 199,\n",
              " 'Thimphu': 200,\n",
              " '%': 201,\n",
              " 'peace-keeping': 202,\n",
              " 'al-Mashhadani': 203,\n",
              " 'offshoot': 204,\n",
              " 'teaching': 205,\n",
              " '10.8': 206,\n",
              " 'Somalian': 207,\n",
              " 'Ono': 208,\n",
              " 'Sofyan': 209,\n",
              " 'pro-Tamil': 210,\n",
              " 'anti-Castro': 211,\n",
              " 'fostering': 212,\n",
              " 'useless': 213,\n",
              " 'Zadar': 214,\n",
              " 'slandered': 215,\n",
              " 'Cachet': 216,\n",
              " 'distract': 217,\n",
              " 'Erin': 218,\n",
              " '5,000': 219,\n",
              " 'thanking': 220,\n",
              " 'trumpets': 221,\n",
              " 'Musicians': 222,\n",
              " 'army-led': 223,\n",
              " 'comb': 224,\n",
              " 'Halvorsen': 225,\n",
              " 'achievements': 226,\n",
              " 'world-number-one': 227,\n",
              " 'Clumsy': 228,\n",
              " 'Imports': 229,\n",
              " 'HIV-positive': 230,\n",
              " '17th-century': 231,\n",
              " '71-year-old': 232,\n",
              " 'bars': 233,\n",
              " 'Aberrahman': 234,\n",
              " 'northeastern': 235,\n",
              " 'sci-fi': 236,\n",
              " 'Contreras': 237,\n",
              " 'basically': 238,\n",
              " 'Sassi': 239,\n",
              " 'Ricans': 240,\n",
              " 'rank-and-file': 241,\n",
              " 'Strong': 242,\n",
              " 'curtailed': 243,\n",
              " 'Panama': 244,\n",
              " 'madrassa': 245,\n",
              " 'with': 246,\n",
              " 'historically': 247,\n",
              " 'pro-western': 248,\n",
              " 'British-ruled': 249,\n",
              " 'MPP': 250,\n",
              " 'Esfandiari': 251,\n",
              " 'Liaoning': 252,\n",
              " 'ostrich': 253,\n",
              " '2022': 254,\n",
              " 'Traveller': 255,\n",
              " 'indict': 256,\n",
              " 'Newspapers': 257,\n",
              " 'precinct': 258,\n",
              " 'twigs': 259,\n",
              " 'Europe-wide': 260,\n",
              " 'stylized': 261,\n",
              " 'Pop': 262,\n",
              " 'Kano': 263,\n",
              " 'Legaspi': 264,\n",
              " 'Muga': 265,\n",
              " 'Zanu-PF': 266,\n",
              " 'Thrust': 267,\n",
              " 'Azocar': 268,\n",
              " 'trillions': 269,\n",
              " 'moans': 270,\n",
              " 'replaced': 271,\n",
              " 'minors': 272,\n",
              " 'communism': 273,\n",
              " 'allied': 274,\n",
              " 'chemical-plant': 275,\n",
              " 'retained': 276,\n",
              " 'forbids': 277,\n",
              " 'Baghaichhari': 278,\n",
              " '79': 279,\n",
              " 'this': 280,\n",
              " 'showings': 281,\n",
              " 'Zarqawi': 282,\n",
              " 'Exchange': 283,\n",
              " 'Rawhi': 284,\n",
              " 'Jabbar': 285,\n",
              " 'unconstitutional': 286,\n",
              " 'Botswana': 287,\n",
              " 'sickened': 288,\n",
              " 'Al-Assad': 289,\n",
              " 'blackmail': 290,\n",
              " 'Sirte': 291,\n",
              " 'Ghost': 292,\n",
              " 'Principality': 293,\n",
              " 'Lent': 294,\n",
              " 'underage': 295,\n",
              " 'Jibla': 296,\n",
              " 'prohibitions': 297,\n",
              " 'remnant': 298,\n",
              " '9.8': 299,\n",
              " 'territorial': 300,\n",
              " '350': 301,\n",
              " 'Maoist-inspired': 302,\n",
              " 'roams': 303,\n",
              " 'STATE': 304,\n",
              " 'Promotion': 305,\n",
              " 'northern': 306,\n",
              " 'Yediot': 307,\n",
              " 'non-indigenous': 308,\n",
              " 'Fayaz': 309,\n",
              " 'Bow': 310,\n",
              " 'futures': 311,\n",
              " 'inspected': 312,\n",
              " 'ram': 313,\n",
              " 'Television': 314,\n",
              " 'Money': 315,\n",
              " 'provides': 316,\n",
              " 'wrap': 317,\n",
              " 'B.C.': 318,\n",
              " 'meantime': 319,\n",
              " 'militia': 320,\n",
              " 'Cincinnati': 321,\n",
              " 'Kisangani': 322,\n",
              " 'Lindsey': 323,\n",
              " 'exaggerate': 324,\n",
              " 'near-capacity': 325,\n",
              " 'culminating': 326,\n",
              " 'holder': 327,\n",
              " 'eagles': 328,\n",
              " 'clearly': 329,\n",
              " 'invaded': 330,\n",
              " 'Citgo': 331,\n",
              " 'mayhem': 332,\n",
              " 'tuberculosis': 333,\n",
              " 'Umaru': 334,\n",
              " 'Yerevan': 335,\n",
              " 'fiscally': 336,\n",
              " 'Occupied': 337,\n",
              " 'Siphiwe': 338,\n",
              " 'Metin': 339,\n",
              " 'Beckham': 340,\n",
              " 'sprees': 341,\n",
              " 'Inspections': 342,\n",
              " 'Project': 343,\n",
              " 'goods': 344,\n",
              " 'Mekong': 345,\n",
              " 'Collemaggio': 346,\n",
              " 'defendant': 347,\n",
              " 'organizers': 348,\n",
              " 'Boskovski': 349,\n",
              " 'reasoned': 350,\n",
              " 'Susanne': 351,\n",
              " 'anti-monarchy': 352,\n",
              " 'Qubad': 353,\n",
              " 'narrowed': 354,\n",
              " 'capitalization': 355,\n",
              " 'Dabous': 356,\n",
              " 'Floribert': 357,\n",
              " 'apply': 358,\n",
              " 'Mae': 359,\n",
              " 'Suha': 360,\n",
              " 'Assessment': 361,\n",
              " 'sadness': 362,\n",
              " 'Citizens': 363,\n",
              " 'refill': 364,\n",
              " 'livelihoods': 365,\n",
              " 'Sugiarto': 366,\n",
              " 'Masses': 367,\n",
              " 'applies': 368,\n",
              " 'Nawab': 369,\n",
              " 'turban': 370,\n",
              " 'curtain': 371,\n",
              " 'Saudi-Syrian': 372,\n",
              " 'Oil': 373,\n",
              " 'Yvelines': 374,\n",
              " 'constitutes': 375,\n",
              " 'Jaffna': 376,\n",
              " 'Khurasan': 377,\n",
              " 'Nipayia': 378,\n",
              " 'non-farm': 379,\n",
              " 'educating': 380,\n",
              " 'Shui-bian': 381,\n",
              " 'Dialogue': 382,\n",
              " 'Baden-Baden': 383,\n",
              " 'philanthropy': 384,\n",
              " 'lift': 385,\n",
              " 'Dodd': 386,\n",
              " 'Althing': 387,\n",
              " 'withdrew': 388,\n",
              " 'Lumpur': 389,\n",
              " 'Sultanate': 390,\n",
              " '202': 391,\n",
              " 'Fequiere': 392,\n",
              " 'G7': 393,\n",
              " 'Johnny': 394,\n",
              " 'L.': 395,\n",
              " 'malicious': 396,\n",
              " 'eventual': 397,\n",
              " 'presidents': 398,\n",
              " 'cancers': 399,\n",
              " 'counteracting': 400,\n",
              " '176': 401,\n",
              " 'Massum': 402,\n",
              " 'quake-ravaged': 403,\n",
              " 'six-point': 404,\n",
              " 'conservatives': 405,\n",
              " 'aggravating': 406,\n",
              " 'Arbour': 407,\n",
              " 'hardliners': 408,\n",
              " 'outline': 409,\n",
              " 'recommends': 410,\n",
              " 'Bingu': 411,\n",
              " 'diversity': 412,\n",
              " 'Magazine': 413,\n",
              " 'Al-Shabab': 414,\n",
              " 'Golkar': 415,\n",
              " 'destinations': 416,\n",
              " 'clearing': 417,\n",
              " 'Galloway': 418,\n",
              " 'wise': 419,\n",
              " 'stocks': 420,\n",
              " 'launch': 421,\n",
              " 'IMB': 422,\n",
              " 'neglect': 423,\n",
              " '14.75': 424,\n",
              " 'concerning': 425,\n",
              " 'wars': 426,\n",
              " 'Atlantis': 427,\n",
              " 'suitcase': 428,\n",
              " 'Woody': 429,\n",
              " 'health-care': 430,\n",
              " 'handmade': 431,\n",
              " 'current': 432,\n",
              " 'convince': 433,\n",
              " '32nd': 434,\n",
              " 'Coordination': 435,\n",
              " 'Jihad': 436,\n",
              " 'rumor': 437,\n",
              " 'lower-house': 438,\n",
              " 'Restrictions': 439,\n",
              " 'automated': 440,\n",
              " '45-thousand': 441,\n",
              " '85-year-old': 442,\n",
              " 'Saqlawiyah': 443,\n",
              " 'permission': 444,\n",
              " 'Everything': 445,\n",
              " 'Newell': 446,\n",
              " 'Nova': 447,\n",
              " 'Rally': 448,\n",
              " 'rapidly-growing': 449,\n",
              " 'demobilize': 450,\n",
              " 'Kosumi': 451,\n",
              " 'flu': 452,\n",
              " 'Treaty': 453,\n",
              " 'Dahoud': 454,\n",
              " 'Kajo': 455,\n",
              " 'typhoons': 456,\n",
              " 'Domestic': 457,\n",
              " 'stage': 458,\n",
              " 'Yao': 459,\n",
              " 'humiliation': 460,\n",
              " 'sisters': 461,\n",
              " '290': 462,\n",
              " 'Militant': 463,\n",
              " '0.8': 464,\n",
              " 'Small': 465,\n",
              " 'betterment': 466,\n",
              " 'Massouda': 467,\n",
              " 'Busta': 468,\n",
              " 'translates': 469,\n",
              " 'Gunmen': 470,\n",
              " 'exploded': 471,\n",
              " 'Hurriyet': 472,\n",
              " 'Marty': 473,\n",
              " 'skiers': 474,\n",
              " 'Guangrong': 475,\n",
              " 'Osprey': 476,\n",
              " '03-Jan': 477,\n",
              " 'jets': 478,\n",
              " 'Unity': 479,\n",
              " 'Bartlett': 480,\n",
              " 'Ruhpolding': 481,\n",
              " 'anti-Japanese': 482,\n",
              " 'witnessed': 483,\n",
              " '48th': 484,\n",
              " 'painful': 485,\n",
              " 'learns': 486,\n",
              " 'Suleyman': 487,\n",
              " 'Razek': 488,\n",
              " 'sacking': 489,\n",
              " 'depiction': 490,\n",
              " 'by-products': 491,\n",
              " 'Farewell': 492,\n",
              " 'raising': 493,\n",
              " '13.73': 494,\n",
              " 'Branch': 495,\n",
              " 'Mengistu': 496,\n",
              " 'Fahd': 497,\n",
              " 'recently-completed': 498,\n",
              " 'wild': 499,\n",
              " 'UBT': 500,\n",
              " 'incaudate': 501,\n",
              " 'corners': 502,\n",
              " 'Early': 503,\n",
              " 'calls': 504,\n",
              " 'Quand': 505,\n",
              " 'Massaua': 506,\n",
              " 'rekindle': 507,\n",
              " 'massive': 508,\n",
              " 'stern': 509,\n",
              " 'Lantos': 510,\n",
              " 'Artemisian-based': 511,\n",
              " 'confronting': 512,\n",
              " 'informed': 513,\n",
              " 'examiner': 514,\n",
              " 'TSZ': 515,\n",
              " 'Satellite': 516,\n",
              " 'Portland': 517,\n",
              " '24,000': 518,\n",
              " 'Zhuang': 519,\n",
              " 'Number': 520,\n",
              " 'Taupin': 521,\n",
              " '144': 522,\n",
              " 'accelerate': 523,\n",
              " 'lambs': 524,\n",
              " 'Grant': 525,\n",
              " 'misdeeds': 526,\n",
              " 'rippled': 527,\n",
              " 'Yousuf': 528,\n",
              " '1878': 529,\n",
              " 'stints': 530,\n",
              " 'climbed': 531,\n",
              " 'embarrass': 532,\n",
              " 'cooperated': 533,\n",
              " 'S.P.': 534,\n",
              " 'Man': 535,\n",
              " 'acrobatic': 536,\n",
              " 'estimating': 537,\n",
              " 'Health': 538,\n",
              " 'jumping': 539,\n",
              " 'Freed': 540,\n",
              " 'Zabihullah': 541,\n",
              " 'Retailers': 542,\n",
              " 'Eleven': 543,\n",
              " 'firings': 544,\n",
              " 'Kian': 545,\n",
              " 'counselor': 546,\n",
              " 'Martian': 547,\n",
              " 'match-fixing': 548,\n",
              " 'Gamal': 549,\n",
              " 'Previously': 550,\n",
              " 'Monitor': 551,\n",
              " 'overflights': 552,\n",
              " 'soybeans': 553,\n",
              " 'principally': 554,\n",
              " 'replacing': 555,\n",
              " 'annals': 556,\n",
              " 'ONCE': 557,\n",
              " 'Aslam': 558,\n",
              " 'Taliban-style': 559,\n",
              " 'Sahibzada': 560,\n",
              " '2,80,000': 561,\n",
              " 'Pohamba': 562,\n",
              " 'great-granddaughter': 563,\n",
              " 'Lott': 564,\n",
              " 'H.I.V./AIDS': 565,\n",
              " 'bypassing': 566,\n",
              " 'obesity-related': 567,\n",
              " 'insupportable': 568,\n",
              " 'tormented': 569,\n",
              " 'fertilized': 570,\n",
              " 'unreasonable': 571,\n",
              " 'earthquake-ravaged': 572,\n",
              " 'visited': 573,\n",
              " 'Water': 574,\n",
              " 'nephew': 575,\n",
              " 'Tzotzil': 576,\n",
              " 'finfish': 577,\n",
              " '1889': 578,\n",
              " 'delighted': 579,\n",
              " 'quo': 580,\n",
              " 'Qalqilya': 581,\n",
              " 'forward': 582,\n",
              " 'racing': 583,\n",
              " 'infectious': 584,\n",
              " 'market-oriented': 585,\n",
              " 'sliding': 586,\n",
              " 'describing': 587,\n",
              " 'commercially-available': 588,\n",
              " 'Eckhard': 589,\n",
              " 'Zambian': 590,\n",
              " 'Amiriyah': 591,\n",
              " 'parachute': 592,\n",
              " 'unproven': 593,\n",
              " 'FDIC': 594,\n",
              " '489': 595,\n",
              " 'Frears': 596,\n",
              " 'remnants': 597,\n",
              " 'Fifth': 598,\n",
              " 'relationship': 599,\n",
              " 'Nagano': 600,\n",
              " 'Victory': 601,\n",
              " 'aliases': 602,\n",
              " 'Banco': 603,\n",
              " 'Gaza-Egypt': 604,\n",
              " 'Hainan': 605,\n",
              " '23-member': 606,\n",
              " 'enact': 607,\n",
              " 'Maumoon': 608,\n",
              " 'handler': 609,\n",
              " 'Luis': 610,\n",
              " 'Oxford': 611,\n",
              " 'deceive': 612,\n",
              " 'dwindled': 613,\n",
              " 'Retired': 614,\n",
              " 'headlines': 615,\n",
              " 'al-Hayam': 616,\n",
              " 'object': 617,\n",
              " 'assailed': 618,\n",
              " 'formalizing': 619,\n",
              " 'Artur': 620,\n",
              " 'Dervis': 621,\n",
              " 'Tornado': 622,\n",
              " 'exacerbates': 623,\n",
              " 'ineffective': 624,\n",
              " 'acknowledging': 625,\n",
              " 'proboscis': 626,\n",
              " 'Tycze': 627,\n",
              " 'Last': 628,\n",
              " 'fundamentally': 629,\n",
              " 'sake': 630,\n",
              " 'Ben': 631,\n",
              " 'southernmost': 632,\n",
              " 'typhoid': 633,\n",
              " 'Fisherman': 634,\n",
              " '20-year': 635,\n",
              " 'G-20': 636,\n",
              " 'sex': 637,\n",
              " 'He': 638,\n",
              " 'RODRIGUEZ': 639,\n",
              " 'patient': 640,\n",
              " '1,15,000': 641,\n",
              " 'Brendan': 642,\n",
              " 'accordingly': 643,\n",
              " 'bride-to-be': 644,\n",
              " 'wheels': 645,\n",
              " 'Sid-Ahmed': 646,\n",
              " 'globe': 647,\n",
              " 'Ellsworth': 648,\n",
              " 'lamb': 649,\n",
              " 'Shuai': 650,\n",
              " 'social': 651,\n",
              " 'subsurface': 652,\n",
              " 'cities': 653,\n",
              " 'Ernest': 654,\n",
              " 'Valentino': 655,\n",
              " 'Hurricane': 656,\n",
              " 'pro-business': 657,\n",
              " 'decades-old': 658,\n",
              " 'indulge': 659,\n",
              " 'Politician': 660,\n",
              " 'huts': 661,\n",
              " 'hill-top': 662,\n",
              " 'export-led': 663,\n",
              " 'stone-throwers': 664,\n",
              " 'deterrent': 665,\n",
              " 'Baptist': 666,\n",
              " 'M.H.': 667,\n",
              " 'rumbling': 668,\n",
              " 'Ceylon': 669,\n",
              " 'parental': 670,\n",
              " 'confirming': 671,\n",
              " 'factory': 672,\n",
              " 're-awakened': 673,\n",
              " 'C-130': 674,\n",
              " 'Lars': 675,\n",
              " 'change': 676,\n",
              " 'Rameez': 677,\n",
              " 'Jalal': 678,\n",
              " 'Kamin': 679,\n",
              " 'Sirens': 680,\n",
              " 'Lao': 681,\n",
              " 'Huge': 682,\n",
              " '38-year': 683,\n",
              " 'dedicated': 684,\n",
              " 'hostilities': 685,\n",
              " 'sells': 686,\n",
              " 'silencers': 687,\n",
              " 'climate': 688,\n",
              " 'Patzi': 689,\n",
              " 'dangerous': 690,\n",
              " 'displaying': 691,\n",
              " 'eels': 692,\n",
              " 'adopted': 693,\n",
              " 'Allahabad': 694,\n",
              " 'comparative': 695,\n",
              " 'MGM': 696,\n",
              " 'Seppi': 697,\n",
              " 'Weiner': 698,\n",
              " 'MIddle': 699,\n",
              " 'Modell': 700,\n",
              " 'asserts': 701,\n",
              " 'incomplete': 702,\n",
              " 'Kara-Suu': 703,\n",
              " 'Mardi': 704,\n",
              " 'offering': 705,\n",
              " 'redesigned': 706,\n",
              " 'broadcasting': 707,\n",
              " 'emphasizing': 708,\n",
              " 'performers': 709,\n",
              " 'Walt': 710,\n",
              " 'Nepal-United': 711,\n",
              " 'muted': 712,\n",
              " '1,600': 713,\n",
              " 'trap': 714,\n",
              " 'sow': 715,\n",
              " '203-4': 716,\n",
              " 'gang': 717,\n",
              " 'Qom': 718,\n",
              " 'Stan': 719,\n",
              " 'leash': 720,\n",
              " 'touchstone': 721,\n",
              " 'trouncing': 722,\n",
              " 'Pipeline': 723,\n",
              " 'SCIRI': 724,\n",
              " 'wide-opened': 725,\n",
              " 'lost': 726,\n",
              " 'ACC': 727,\n",
              " 'Polyclinic': 728,\n",
              " 'Ties': 729,\n",
              " 'Ammar': 730,\n",
              " 'standings': 731,\n",
              " 'Neighboring': 732,\n",
              " 'purifying': 733,\n",
              " 'financial': 734,\n",
              " 'strongest-ever': 735,\n",
              " 'slipped': 736,\n",
              " 'Wing': 737,\n",
              " 'raiser': 738,\n",
              " 'capable': 739,\n",
              " 'Special': 740,\n",
              " 'Nazarov': 741,\n",
              " 'Sugiura': 742,\n",
              " 'truncated': 743,\n",
              " 'Gyanendra': 744,\n",
              " '13.65': 745,\n",
              " 'Ayad': 746,\n",
              " 'EEC': 747,\n",
              " 'employment': 748,\n",
              " 'Farda': 749,\n",
              " 'Bombay': 750,\n",
              " 'recharge': 751,\n",
              " 'Iran-Iraq': 752,\n",
              " 'risked': 753,\n",
              " 'Teams': 754,\n",
              " 'Stein': 755,\n",
              " 'Dee': 756,\n",
              " 'guileful': 757,\n",
              " 'nest': 758,\n",
              " 'quarry': 759,\n",
              " 'comedy': 760,\n",
              " 'Mariam': 761,\n",
              " 'ex-diplomat': 762,\n",
              " 'Deif': 763,\n",
              " 'recognizable': 764,\n",
              " 'Persons': 765,\n",
              " '12-year-old': 766,\n",
              " 'DVDs': 767,\n",
              " 'highly-developed': 768,\n",
              " 'shotgun': 769,\n",
              " 'Yusgiantoro': 770,\n",
              " 'anti-government': 771,\n",
              " 'Fidel': 772,\n",
              " 'TB': 773,\n",
              " 'Pavarotti': 774,\n",
              " 'Montedison': 775,\n",
              " 'instance': 776,\n",
              " 'schoolgirls': 777,\n",
              " 'embankment': 778,\n",
              " '4000': 779,\n",
              " 'Gerd': 780,\n",
              " 'pre-2007': 781,\n",
              " 'personalities': 782,\n",
              " 'sidewalks': 783,\n",
              " 'sunrise': 784,\n",
              " 'nurses': 785,\n",
              " 'rock': 786,\n",
              " 'Abdel-Al': 787,\n",
              " 'tickets': 788,\n",
              " 'excerpts': 789,\n",
              " '4,000': 790,\n",
              " 'six-country': 791,\n",
              " 'twin': 792,\n",
              " 'Earth': 793,\n",
              " 'desperately': 794,\n",
              " 'humiliated': 795,\n",
              " 'rebuilding': 796,\n",
              " 'fishing-rich': 797,\n",
              " 'repressed': 798,\n",
              " 'Fats': 799,\n",
              " 'Bitarov': 800,\n",
              " 'burst': 801,\n",
              " 'Safeguards': 802,\n",
              " 'dimension': 803,\n",
              " 'Kononov': 804,\n",
              " 'Pervasive': 805,\n",
              " 'Yusuf': 806,\n",
              " 'suing': 807,\n",
              " 'stand-up': 808,\n",
              " 'Miroslav': 809,\n",
              " 'Kalachen': 810,\n",
              " 'sanction': 811,\n",
              " 'mirrors': 812,\n",
              " 'Williams': 813,\n",
              " 'bronchitis': 814,\n",
              " 'toll': 815,\n",
              " 'steer': 816,\n",
              " 'al-Sherif': 817,\n",
              " 'anti-inflammatory': 818,\n",
              " 'borrows': 819,\n",
              " 'Lyudmila': 820,\n",
              " 'remembrances': 821,\n",
              " 'expending': 822,\n",
              " 'near-earth': 823,\n",
              " 'perception': 824,\n",
              " '18,000': 825,\n",
              " 'spearheaded': 826,\n",
              " 'jar': 827,\n",
              " 'Hirayama': 828,\n",
              " 'Hongdu': 829,\n",
              " 'kwaito': 830,\n",
              " 'nausea': 831,\n",
              " 'suspending': 832,\n",
              " 'qualifiers': 833,\n",
              " 'Walikale': 834,\n",
              " 'yelled': 835,\n",
              " 'P.J.': 836,\n",
              " 'Young-sam': 837,\n",
              " '30-year-old': 838,\n",
              " 'devotion': 839,\n",
              " 'businesses': 840,\n",
              " 'Loit': 841,\n",
              " 'Pillay': 842,\n",
              " '68': 843,\n",
              " 'Braille': 844,\n",
              " 'via': 845,\n",
              " 'Mathematically': 846,\n",
              " 'Yokota': 847,\n",
              " 'enlisted': 848,\n",
              " '10-member': 849,\n",
              " 'banned': 850,\n",
              " 'congratulations': 851,\n",
              " 'Luton': 852,\n",
              " 'markets': 853,\n",
              " 'Boroujerdi': 854,\n",
              " 'wired': 855,\n",
              " 'stranding': 856,\n",
              " 'informal': 857,\n",
              " 'peshmerga': 858,\n",
              " 'Netherlands': 859,\n",
              " 'Emancipation': 860,\n",
              " 'staffing': 861,\n",
              " 'Gaspard': 862,\n",
              " 'HARIRI': 863,\n",
              " 'ODA': 864,\n",
              " 'Change': 865,\n",
              " 'Via': 866,\n",
              " 'Ibero-American': 867,\n",
              " 'Lashkar-e-Islam': 868,\n",
              " 'Fat': 869,\n",
              " 'extensive': 870,\n",
              " 'Rains': 871,\n",
              " 'Antoin': 872,\n",
              " 'Parma': 873,\n",
              " 'Lieutenant-Colonel': 874,\n",
              " 'dissolve': 875,\n",
              " 'rolling': 876,\n",
              " 'lots': 877,\n",
              " 'ablaze': 878,\n",
              " 'fairly': 879,\n",
              " 'lectures': 880,\n",
              " 'egg': 881,\n",
              " 'Arabs': 882,\n",
              " 'slaughterhouse': 883,\n",
              " 'Diyala': 884,\n",
              " 'strikeouts': 885,\n",
              " 'unilateral': 886,\n",
              " 'swimmer': 887,\n",
              " 'awkwardly': 888,\n",
              " 'blossomed': 889,\n",
              " 'distress': 890,\n",
              " 'Ohrid': 891,\n",
              " 'cafes': 892,\n",
              " 'warheads': 893,\n",
              " 'bearing': 894,\n",
              " 'inductee': 895,\n",
              " 'Gargash': 896,\n",
              " 'deaths': 897,\n",
              " 'parliamentarians': 898,\n",
              " 'government-funded': 899,\n",
              " '72-hour': 900,\n",
              " 'fathers': 901,\n",
              " 'Q.': 902,\n",
              " 'Dukeness': 903,\n",
              " 'Roswell': 904,\n",
              " 'Splendour': 905,\n",
              " 'Australians': 906,\n",
              " 'timeline': 907,\n",
              " 'Eyharts': 908,\n",
              " 'Rashad': 909,\n",
              " 'jailhouse': 910,\n",
              " 'Adriano': 911,\n",
              " 'Nourollah': 912,\n",
              " 'Palm': 913,\n",
              " 'Cabo': 914,\n",
              " 'unguarded': 915,\n",
              " 'rebounds': 916,\n",
              " 'Landslides': 917,\n",
              " 'Providence': 918,\n",
              " 'weigh': 919,\n",
              " 'Nath': 920,\n",
              " 'she': 921,\n",
              " 'spur': 922,\n",
              " 'treatment': 923,\n",
              " 'revised': 924,\n",
              " 'Avi': 925,\n",
              " '6-hour': 926,\n",
              " 'quarter-century': 927,\n",
              " 'Thirty-three': 928,\n",
              " 'Vokey': 929,\n",
              " 'Zahi': 930,\n",
              " 'party-line': 931,\n",
              " 'Astros': 932,\n",
              " 'Berger': 933,\n",
              " 'Moi': 934,\n",
              " 'modernity': 935,\n",
              " '22,000': 936,\n",
              " 'cancer-causing': 937,\n",
              " 'Singnaghi': 938,\n",
              " 'Raikkonen': 939,\n",
              " 'confiscating': 940,\n",
              " 'instruments': 941,\n",
              " 'Doug': 942,\n",
              " 'Many': 943,\n",
              " 'response': 944,\n",
              " 'Yosef': 945,\n",
              " 'Freeman': 946,\n",
              " 'Larry': 947,\n",
              " 'Univision': 948,\n",
              " 'agencies': 949,\n",
              " 'Skier': 950,\n",
              " 'veto-wielding': 951,\n",
              " 'Mara': 952,\n",
              " 'pachyderm': 953,\n",
              " 'pro-investment': 954,\n",
              " 'Oshiomole': 955,\n",
              " 'Rebelde': 956,\n",
              " '164': 957,\n",
              " '740': 958,\n",
              " '1943': 959,\n",
              " 'bloc': 960,\n",
              " 'liberty': 961,\n",
              " 'unity': 962,\n",
              " 'Naqib': 963,\n",
              " 'Churches': 964,\n",
              " 'Lebanese-Syrian': 965,\n",
              " '1-to-1': 966,\n",
              " '8.20': 967,\n",
              " 'Anambra': 968,\n",
              " 'Ulrich': 969,\n",
              " 'pains': 970,\n",
              " '422.5': 971,\n",
              " 'Laden': 972,\n",
              " '218-5': 973,\n",
              " 'plaintiffs': 974,\n",
              " 'Stethem': 975,\n",
              " 'interrogating': 976,\n",
              " 'Zurbriggen': 977,\n",
              " 'RONCO': 978,\n",
              " 'ten-man': 979,\n",
              " 'orientation': 980,\n",
              " 'Francis': 981,\n",
              " 'HARE': 982,\n",
              " 'definite': 983,\n",
              " 'combative': 984,\n",
              " '11.5': 985,\n",
              " '1978': 986,\n",
              " 'Radisson': 987,\n",
              " 'rounded': 988,\n",
              " 'expatriate': 989,\n",
              " 'Frazier': 990,\n",
              " '7,864': 991,\n",
              " 'Andriy': 992,\n",
              " 'cruised': 993,\n",
              " 'ratified': 994,\n",
              " 'eventually': 995,\n",
              " 'produced': 996,\n",
              " 'Gomoa': 997,\n",
              " 'standardized': 998,\n",
              " 'being': 999,\n",
              " '70th': 1000,\n",
              " ...}"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Padding and train test split"
      ],
      "metadata": {
        "id": "WEc6LyllSooK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ml = 50\n",
        "X= [[wi[w[0]] for w in s ] for s in s]\n",
        "X = pad_sequences(maxlen=ml,sequences = X,padding ='post',value = num_words-1)\n",
        "y= [[ti[w[2]] for w in s ] for s in s]\n",
        "y = pad_sequences(maxlen=ml,sequences = y,padding ='post',value = ti[\"O\"])\n",
        "y = [to_categorical(i,num_classes = num_tags) for i in y]\n",
        "x_train,x_test,y_train,y_test = train_test_split(X,y,test_size=0.1,random_state=1)"
      ],
      "metadata": {
        "id": "47eV3523S3ey"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Build and Compile Bidirectional LSTM Model"
      ],
      "metadata": {
        "id": "ZspJ8KzsSol7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "input_word = Input(shape=(ml,))\n",
        "m = Embedding(input_dim = num_words, output_dim = ml,input_length = ml)(input_word)\n",
        "m= SpatialDropout1D(0.1)(m)\n",
        "m = Bidirectional(LSTM(units=100, return_sequences = True, recurrent_dropout =0.1))(m)\n",
        "out= TimeDistributed(Dense(num_tags,activation = 'softmax'))(m)\n",
        "m = Model(input_word,out)\n",
        "m.compile(loss ='mse', optimizer ='adam')\n",
        "m.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gP42giW9S9Lj",
        "outputId": "0fb22166-f257-43f7-8fce-63d91f9c89c3"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_2 (InputLayer)        [(None, 50)]              0         \n",
            "                                                                 \n",
            " embedding_1 (Embedding)     (None, 50, 50)            1758950   \n",
            "                                                                 \n",
            " spatial_dropout1d_1 (Spatia  (None, 50, 50)           0         \n",
            " lDropout1D)                                                     \n",
            "                                                                 \n",
            " bidirectional_1 (Bidirectio  (None, 50, 200)          120800    \n",
            " nal)                                                            \n",
            "                                                                 \n",
            " time_distributed_1 (TimeDis  (None, 50, 17)           3417      \n",
            " tributed)                                                       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1,883,167\n",
            "Trainable params: 1,883,167\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Train the Model"
      ],
      "metadata": {
        "id": "VTIQK1ggSojZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "es= EarlyStopping(monitor='val_accuracy',patience=1,verbose=0,mode='max',restore_best_weights=False)\n",
        "cb = [PlotLossesCallback(),es]\n",
        "h = m.fit(\n",
        "    x_train, np.array(y_train),\n",
        "    validation_split = 0.2,\n",
        "    batch_size=32,\n",
        "    epochs=3,\n",
        "    verbose = 1\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OSe6UXrWTALt",
        "outputId": "7eed581f-c433-412a-edc3-92c742aff36b"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/3\n",
            "1080/1080 [==============================] - 219s 198ms/step - loss: 0.0050 - val_loss: 0.0021\n",
            "Epoch 2/3\n",
            "1080/1080 [==============================] - 211s 195ms/step - loss: 0.0016 - val_loss: 0.0014\n",
            "Epoch 3/3\n",
            "1080/1080 [==============================] - 212s 196ms/step - loss: 0.0011 - val_loss: 0.0013\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Evaluate"
      ],
      "metadata": {
        "id": "JRKcE2_2SoPZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "m.evaluate(x_test,np.array(y_test))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OeZlYJ3bTDGk",
        "outputId": "966576a8-c57e-4276-c481-00b9e52dcd68"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "150/150 [==============================] - 4s 28ms/step - loss: 0.0013\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0013421425828710198"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    }
  ]
}