code/main.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CS626-A2.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aScdhogHlaXU"
      },
      "source": [
        "# Overlap based WSD using Lesk's Algorithm with Word2Vec Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Eclmx_llS9S"
      },
      "source": [
        "## Install Dependencies\n",
        "Run this section only once"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lcUGkcGqBVkh",
        "outputId": "8e2019fd-960b-4c5b-ecb6-29387dcaf0ba"
      },
      "source": [
        "# Run this only once\n",
        "import nltk\n",
        "nltk.download(\"punkt\")\n",
        "nltk.download(\"wordnet\")\n",
        "nltk.download(\"semcor\") # downloads the .zip file, but doesn't unzip it\n",
        "nltk.download(\"stopwords\")\n",
        "nltk.download(\"averaged_perceptron_tagger\")"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/wordnet.zip.\n",
            "[nltk_data] Downloading package semcor to /root/nltk_data...\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HG_7xzwPES8h"
      },
      "source": [
        "# Unzip SemCor 3.0\n",
        "! unzip -q /root/nltk_data/corpora/semcor.zip -d /root/nltk_data/corpora # after this, data in /root/nltk_data/corpora/semcor"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dvChfNZgk4Sf",
        "outputId": "1ea946ba-d559-4471-e819-27b5f2bec9a5"
      },
      "source": [
        "# Download pre-trained word2vec embeddings\n",
        "! wget -c \"https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz\" # 1.53 GB"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-09-08 11:08:16--  https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz\n",
            "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.140.46\n",
            "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.140.46|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1647046227 (1.5G) [application/x-gzip]\n",
            "Saving to: ‘GoogleNews-vectors-negative300.bin.gz’\n",
            "\n",
            "GoogleNews-vectors- 100%[===================>]   1.53G  40.9MB/s    in 37s     \n",
            "\n",
            "2021-09-08 11:08:53 (42.6 MB/s) - ‘GoogleNews-vectors-negative300.bin.gz’ saved [1647046227/1647046227]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DR7O1V4igs4W",
        "outputId": "afdb2a5f-c715-4311-af59-da0f06c40a43"
      },
      "source": [
        "! pip install num2words"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting num2words\n",
            "  Downloading num2words-0.5.10-py3-none-any.whl (101 kB)\n",
            "\u001b[K     |████████████████████████████████| 101 kB 2.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: docopt>=0.6.2 in /usr/local/lib/python3.7/dist-packages (from num2words) (0.6.2)\n",
            "Installing collected packages: num2words\n",
            "Successfully installed num2words-0.5.10\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jDHkdaRgmIox"
      },
      "source": [
        "# Import w2v here itself as it takes time to load\n",
        "from gensim.models import KeyedVectors\n",
        "W2V = KeyedVectors.load_word2vec_format(\"GoogleNews-vectors-negative300.bin.gz\", binary = True)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4M5sZFHkIR6y"
      },
      "source": [
        "## Start\n",
        "To re-run, run from this cell"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8pMakpyaITBy"
      },
      "source": [
        "%reset_selective -f ^(?!W2V).*$ # clear everything except W2V"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F0AhVQs5Diyy"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UKRbhcLPDj7_"
      },
      "source": [
        "import nltk\n",
        "from nltk import word_tokenize\n",
        "from nltk.corpus import semcor # corpus reader: https://github.com/nltk/nltk/blob/develop/nltk/corpus/reader/semcor.py\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.corpus import wordnet as wn\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "\n",
        "import random\n",
        "import numpy as np\n",
        "from tqdm.notebook import tqdm\n",
        "from string import punctuation\n",
        "from num2words import num2words\n",
        "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t3uHxPSP14tV"
      },
      "source": [
        "## Constants"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yDdlSB8ls8dn"
      },
      "source": [
        "# Custom stopwords\n",
        "EXTRA_SW = [\n",
        "    \"''\",\n",
        "    \"'s\",\n",
        "    \"``\"\n",
        "]\n",
        "\n",
        "SW = stopwords.words(\"english\")\n",
        "SW += [p for p in punctuation]\n",
        "SW += EXTRA_SW"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "me_BQR1CFNIV"
      },
      "source": [
        "lemmatizer = WordNetLemmatizer()"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DNrgSQtYFaJ8"
      },
      "source": [
        "## Functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X5FEJTwHCKL7"
      },
      "source": [
        "def cosineSimilarity(a, b):\n",
        "    cs = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n",
        "    return cs"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iGRFnQT2q5pB"
      },
      "source": [
        "def isNumber(s):\n",
        "    try:\n",
        "        float(s)\n",
        "        return True\n",
        "    except ValueError:\n",
        "        return False"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0wXiU_kbhHNK"
      },
      "source": [
        "def n2w(w):\n",
        "    # converts given n to word form if n is numeric\n",
        "    if isNumber(w) and w.lower() != \"infinity\" and w.lower() != \"nan\":\n",
        "        w = num2words(w)\n",
        "    return w"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FNsc0CgC4qiP"
      },
      "source": [
        "def lemmatize(w, tag):\n",
        "    if tag is None:\n",
        "        return lemmatizer.lemmatize(w)\n",
        "    else:\n",
        "        return lemmatizer.lemmatize(w, tag)"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NEBmWMwVuBKU"
      },
      "source": [
        "def clean(tokens):\n",
        "    tagged = nltk.pos_tag(tokens)\n",
        "    lemmatized = [lemmatize(w, treebank2wn(tag)) for w, tag in tagged]\n",
        "    cleaned = [n2w(w) for w in lemmatized if w.lower() not in SW]\n",
        "    return cleaned"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n3U2-JoBmls2"
      },
      "source": [
        "def getVec(w):\n",
        "    # Returns (300,) shaped numpy array\n",
        "    try:\n",
        "        v = W2V[w]\n",
        "        return v\n",
        "    except KeyError:\n",
        "        return None # ignore words not in vocab"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nkY8QXnBS7Bv"
      },
      "source": [
        "def syn2sense(syn):\n",
        "    # get the sense (= lemma.postag.num) for a given synset\n",
        "    s = syn.name()\n",
        "    # n = \".\".join(s.split(\".\")[-2:]) # n.01 and v.01 are different senses (eg: ash.n.01, ash.v.01)\n",
        "    return s"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fvRoWEzRz0oi"
      },
      "source": [
        "def treebank2wn(ttag):\n",
        "    if ttag.startswith(\"J\"):\n",
        "        return wn.ADJ\n",
        "    elif ttag.startswith(\"V\"):\n",
        "        return wn.VERB\n",
        "    elif ttag.startswith(\"N\"):\n",
        "        return wn.NOUN\n",
        "    elif ttag.startswith(\"R\"):\n",
        "        return wn.ADV\n",
        "    else:\n",
        "        return None"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "swX0ECwZ76up"
      },
      "source": [
        "def sent2vec(tokens):\n",
        "\n",
        "    v = 0\n",
        "    n = 0\n",
        "\n",
        "    for w in tokens:\n",
        "\n",
        "        # Check if w is a named entity (TODO: use wordnet NE tag directly instead of below approach)\n",
        "        tkns = word_tokenize(w)\n",
        "\n",
        "        if len(tkns) > 1:\n",
        "            for t in tkns:\n",
        "                vt = getVec(t)\n",
        "                if vt is not None:\n",
        "                    n += 1\n",
        "                    v += vt\n",
        "        else:\n",
        "            vw = getVec(w)\n",
        "            if vw is not None:\n",
        "                n += 1\n",
        "                v += vw\n",
        "\n",
        "    if n == 0: # when tokens is empty or no token in word2vec\n",
        "        v = None\n",
        "    else:\n",
        "        v /= n\n",
        "\n",
        "    return v"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U27aDOsWFbdA"
      },
      "source": [
        "def parse(d):\n",
        "    # d (nltk.corpus.reader.semcor.SemcorSentence) : can have lists as elements or nltk.tree.Tree\n",
        "\n",
        "    tokens = []\n",
        "    senses = []\n",
        "\n",
        "    for e in d:\n",
        "\n",
        "        if isinstance(e, nltk.tree.Tree):\n",
        "\n",
        "            # e.label() returns a nltk.corpus.reader.wordnet.Lemma object or simply a string (of the form word.pos.num)\n",
        "            lemma = e.label()\n",
        "            \n",
        "            if isinstance(lemma, nltk.corpus.reader.wordnet.Lemma):\n",
        "                synset = lemma.synset() # nltk.corpus.reader.wordnet.Synset\n",
        "                sense = syn2sense(synset)\n",
        "            else:\n",
        "                sense = None # ignore all tagged senses which aren't in WN (i.e. present as Lemma)\n",
        "            \n",
        "            le = len(e)\n",
        "            if le == 1:\n",
        "                w = e[0]\n",
        "                if isinstance(w, nltk.tree.Tree) or isinstance(w, list):\n",
        "                    # ignore w.label()\n",
        "                    lw = len(w)\n",
        "                    w = \" \".join([w[i] for i in range(lw)])\n",
        "            else:\n",
        "                w = \" \".join([e[i] for i in range(le)])\n",
        "\n",
        "        elif isinstance(e, list):\n",
        "            w = e[0]\n",
        "            sense = None\n",
        "\n",
        "        else:\n",
        "            invtype = type(e)\n",
        "            raise TypeError(f\"Invalid type: {invtype}\")\n",
        "\n",
        "        if w:\n",
        "            tokens.append(w)\n",
        "            senses.append(sense)\n",
        "\n",
        "    return tokens, senses"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WoJRkwmc_laW"
      },
      "source": [
        "def getCandidates(w, tag):\n",
        "    # Get candidate sense vectors and labels of a word w\n",
        "\n",
        "    w = w.replace(\".\", \"\") # \"Sept.\" becomes \"Sept\"\n",
        "    w = w.replace(\"-\", \"\") # re-elected becomes \"relected\"\n",
        "\n",
        "    # Handle ngrams (like \"united states\")\n",
        "    tkns = word_tokenize(w)\n",
        "    if len(tkns) > 1:\n",
        "        tagged = nltk.pos_tag(tkns)\n",
        "        tags = [treebank2wn(p[1]) for p in tagged]\n",
        "        ltkns = [lemmatize(w, t) for w, t in zip(tkns, tags)]\n",
        "        w = \"_\".join(ltkns)\n",
        "\n",
        "    syns = wn.synsets(w, tag)\n",
        "\n",
        "    if len(syns) == 0:\n",
        "        w = \"_\".join(tkns) # cases where lemmatization doesn't help (\"agreed upon\")\n",
        "        syns = wn.synsets(w, tag)\n",
        "\n",
        "    sense_vectors = []\n",
        "    sense_labels = []\n",
        "\n",
        "    for syn in syns:\n",
        "\n",
        "        label = syn2sense(syn)\n",
        "\n",
        "        defn = syn.definition() # TODO: Implement the extended Lesk algorithm that uses related synsets as well\n",
        "\n",
        "        defn = defn.replace(\"_\", \" \")\n",
        "        defn = defn.replace(\"-\", \" \")\n",
        "\n",
        "        tkns = word_tokenize(defn)\n",
        "        if len(tkns) == 0:\n",
        "            raise ValueError(f\"0 tokens found: {defn}\")\n",
        "\n",
        "        clnd = clean(tkns)\n",
        "        if len(clnd) < 2:\n",
        "            clnd = tkns # don't remove stopwords if the sentence is almost entirely made up of them\n",
        "\n",
        "        sv = sent2vec(clnd)\n",
        "\n",
        "        if sv is None:\n",
        "            print(f\"Empty sense vector. Word: {w}, Definition: {defn}, Cleaned: {clnd}. Using a random vector as sense.\")\n",
        "            sv = np.random.rand(300,)\n",
        "        \n",
        "        sense_vectors.append(sv)\n",
        "        sense_labels.append(label)\n",
        "\n",
        "    return sense_vectors, sense_labels # returns empty lists if no synsets found"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5rZ-IsI2DkTd"
      },
      "source": [
        "## Main"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_yqFveUQDYQD"
      },
      "source": [
        "data = semcor.tagged_sents(tag = \"sem\") # 37176 sentences, 224716 tagged words, 34189 unique senses "
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vUegBwQjrUlb",
        "outputId": "61d7a56c-294e-465d-b991-217ef18ff0a8"
      },
      "source": [
        "n_total = 0\n",
        "n_correct = 0\n",
        "n_samples = 0\n",
        "\n",
        "true = []\n",
        "pred = []\n",
        "\n",
        "for d in data:\n",
        "\n",
        "    try:\n",
        "\n",
        "        tokens, senses = parse(d)\n",
        "        n_tokens = len(tokens)\n",
        "\n",
        "        # Tag and lemmatize tokens, don't remove stopwords here\n",
        "        tagged = nltk.pos_tag(tokens)\n",
        "        tags = [treebank2wn(p[1]) for p in tagged]\n",
        "        tokens = [lemmatize(w, tag) for w, tag in zip(tokens, tags)]\n",
        "\n",
        "        for i in range(n_tokens):\n",
        "\n",
        "            w = tokens[i]\n",
        "            tag = tags[i]\n",
        "            s_true = senses[i]\n",
        "\n",
        "            if not isinstance(w, str):\n",
        "                raise TypeError(f\"Invalid type: {type(w)} : {w} : {tokens}\")\n",
        "\n",
        "            # Don't predict for words that aren't sense-tagged\n",
        "            if s_true is None:\n",
        "                continue\n",
        "\n",
        "            # Get context for w (all words in the sentence except w)\n",
        "            context = tokens.copy()\n",
        "            del context[i] # more efficient than .pop(i)\n",
        "\n",
        "            # Remove stopwords and punctuation from context to reduce #elements in the context\n",
        "            # These don't contribute much to the semantic overlap anyways\n",
        "            cleaned = clean(context)\n",
        "            if len(cleaned) < 2:\n",
        "                cleaned = context # if almost all words are stopwords, don't remove any\n",
        "\n",
        "            # Get context vector by average w2v vectors for each word\n",
        "            cv = sent2vec(cleaned)\n",
        "\n",
        "            if cv is None:\n",
        "                print(f\"Empty context vector. Word: {w}, Cleaned: {cleaned}, Tokens: {tokens}. Using a random vector as context.\")\n",
        "                cv = np.random.rand(300,)\n",
        "\n",
        "            # Get WordNet candidate senses\n",
        "            sense_vectors, sense_labels = getCandidates(w, tag)\n",
        "            n_candidates = len(sense_labels)\n",
        "\n",
        "            s_pred = None\n",
        "            if n_candidates == 0:\n",
        "                # Try without pos tag\n",
        "                sense_vectors, sense_labels = getCandidates(w, None)\n",
        "                n_candidates = len(sense_labels)\n",
        "                if n_candidates == 0:\n",
        "                    # print(f\"No synsets found. Word: {w}, Sense: {s_true}\") # don't print, too many NE's in the data\n",
        "                    s_pred = random.choice([\"group.n.01\", \"person.n.01\", \"location.n.01\"]) # most likely an NE\n",
        "            \n",
        "            # Use cosine similarity to get the best senses\n",
        "            best = -1 \n",
        "            for j in range(n_candidates):\n",
        "                sv = sense_vectors[j]\n",
        "                cs = cosineSimilarity(cv, sv)\n",
        "                if cs > best:\n",
        "                    best = cs\n",
        "                    s_pred = sense_labels[j]\n",
        "\n",
        "            if s_true == s_pred:\n",
        "                n_correct += 1\n",
        "            n_total += 1\n",
        "\n",
        "            true.append(s_true)\n",
        "            pred.append(s_pred)\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Error at: {n_samples}\")\n",
        "        print(str(e))\n",
        "        raise ValueError(\"Error\")\n",
        "\n",
        "    n_samples += 1\n",
        "\n",
        "    if n_samples%200 == 0:\n",
        "        print(f\"{n_samples} sentences processed\")\n",
        "        acc = (n_correct/n_total)*100\n",
        "        print(f\"Accuracy: {acc:.4f}\")\n",
        "        print()"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "200 sentences processed\n",
            "Accuracy: 44.2681\n",
            "\n",
            "400 sentences processed\n",
            "Accuracy: 41.6832\n",
            "\n",
            "600 sentences processed\n",
            "Accuracy: 40.8078\n",
            "\n",
            "800 sentences processed\n",
            "Accuracy: 39.9124\n",
            "\n",
            "1000 sentences processed\n",
            "Accuracy: 40.2434\n",
            "\n",
            "1200 sentences processed\n",
            "Accuracy: 39.9983\n",
            "\n",
            "1400 sentences processed\n",
            "Accuracy: 40.0856\n",
            "\n",
            "1600 sentences processed\n",
            "Accuracy: 40.5734\n",
            "\n",
            "1800 sentences processed\n",
            "Accuracy: 40.5844\n",
            "\n",
            "2000 sentences processed\n",
            "Accuracy: 40.5544\n",
            "\n",
            "2200 sentences processed\n",
            "Accuracy: 40.5462\n",
            "\n",
            "Empty context vector. Word: Cancer, Cleaned: ['``', \"''\", '!'], Tokens: ['``', 'Cancer', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: By no means, Cleaned: ['.'], Tokens: ['By no means', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: For instance, Cleaned: [':'], Tokens: ['For instance', ':']. Using a random vector as context.\n",
            "Empty context vector. Word: Death, Cleaned: ['!'], Tokens: ['Death', '!']. Using a random vector as context.\n",
            "2400 sentences processed\n",
            "Accuracy: 40.8285\n",
            "\n",
            "2600 sentences processed\n",
            "Accuracy: 40.9303\n",
            "\n",
            "2800 sentences processed\n",
            "Accuracy: 41.0180\n",
            "\n",
            "3000 sentences processed\n",
            "Accuracy: 41.4333\n",
            "\n",
            "3200 sentences processed\n",
            "Accuracy: 41.6645\n",
            "\n",
            "3400 sentences processed\n",
            "Accuracy: 42.1153\n",
            "\n",
            "3600 sentences processed\n",
            "Accuracy: 42.5494\n",
            "\n",
            "3800 sentences processed\n",
            "Accuracy: 42.9463\n",
            "\n",
            "4000 sentences processed\n",
            "Accuracy: 43.1785\n",
            "\n",
            "4200 sentences processed\n",
            "Accuracy: 43.6787\n",
            "\n",
            "4400 sentences processed\n",
            "Accuracy: 43.9530\n",
            "\n",
            "4600 sentences processed\n",
            "Accuracy: 43.7741\n",
            "\n",
            "4800 sentences processed\n",
            "Accuracy: 43.7173\n",
            "\n",
            "5000 sentences processed\n",
            "Accuracy: 43.4453\n",
            "\n",
            "5200 sentences processed\n",
            "Accuracy: 43.2820\n",
            "\n",
            "Empty context vector. Word: Therefore, Cleaned: [','], Tokens: ['Therefore', ',']. Using a random vector as context.\n",
            "5400 sentences processed\n",
            "Accuracy: 43.1136\n",
            "\n",
            "5600 sentences processed\n",
            "Accuracy: 43.3265\n",
            "\n",
            "5800 sentences processed\n",
            "Accuracy: 43.2033\n",
            "\n",
            "Empty context vector. Word: Mines, Cleaned: ['.'], Tokens: ['Mines', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Mines, Cleaned: ['.'], Tokens: ['Mines', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Mines, Cleaned: ['.'], Tokens: ['Mines', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Mines, Cleaned: ['.'], Tokens: ['Mines', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Wait, Cleaned: ['``', '!'], Tokens: ['``', 'Wait', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: All right, Cleaned: ['``', '.'], Tokens: ['``', 'All right', '.']. Using a random vector as context.\n",
            "6000 sentences processed\n",
            "Accuracy: 43.0230\n",
            "\n",
            "6200 sentences processed\n",
            "Accuracy: 42.8960\n",
            "\n",
            "Empty context vector. Word: Stevie, Cleaned: ['``', '!'], Tokens: ['``', 'Stevie', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Stevie, Cleaned: [\"''\", '!'], Tokens: ['Stevie', \"''\", '!']. Using a random vector as context.\n",
            "6400 sentences processed\n",
            "Accuracy: 42.8214\n",
            "\n",
            "Empty context vector. Word: Months, Cleaned: ['.'], Tokens: ['Months', '.']. Using a random vector as context.\n",
            "6600 sentences processed\n",
            "Accuracy: 42.7548\n",
            "\n",
            "Empty context vector. Word: Brandon, Cleaned: ['.'], Tokens: ['Brandon', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Johnson, Cleaned: ['.'], Tokens: ['Johnson', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Johnson, Cleaned: ['Sleepy-eyed', 'soft-spoken'], Tokens: ['Sleepy-eyed', ',', 'soft-spoken', 'Johnson', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Knows the score, Cleaned: ['.'], Tokens: ['Knows the score', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Top dog, Cleaned: ['.'], Tokens: ['Top dog', '.']. Using a random vector as context.\n",
            "6800 sentences processed\n",
            "Accuracy: 42.6591\n",
            "\n",
            "7000 sentences processed\n",
            "Accuracy: 42.4769\n",
            "\n",
            "Empty context vector. Word: Eli Corault, Cleaned: ['!'], Tokens: ['Eli Corault', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: John, Cleaned: ['``', '?'], Tokens: ['``', 'John', '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Plot, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Plot', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Heretic, Cleaned: ['``', \"''\", '!'], Tokens: ['``', 'Heretic', \"''\", '!']. Using a random vector as context.\n",
            "7200 sentences processed\n",
            "Accuracy: 42.3363\n",
            "\n",
            "7400 sentences processed\n",
            "Accuracy: 42.2946\n",
            "\n",
            "Empty context vector. Word: Locked, Cleaned: ['.'], Tokens: ['Locked', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Workmen, Cleaned: ['``', '.'], Tokens: ['``', 'Workmen', '.']. Using a random vector as context.\n",
            "7600 sentences processed\n",
            "Accuracy: 42.2446\n",
            "\n",
            "Empty context vector. Word: Tomorrow, Cleaned: ['``', '.'], Tokens: ['``', 'Tomorrow', '.']. Using a random vector as context.\n",
            "7800 sentences processed\n",
            "Accuracy: 42.0719\n",
            "\n",
            "Empty context vector. Word: Warsaw, Cleaned: ['!'], Tokens: ['Warsaw', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Warsaw, Cleaned: ['!'], Tokens: ['Warsaw', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Warsaw, Cleaned: ['!'], Tokens: ['Warsaw', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Miss Rak, Cleaned: ['``', '.'], Tokens: ['``', 'Miss Rak', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Deportees, Cleaned: ['.'], Tokens: ['Deportees', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Jews, Cleaned: ['.'], Tokens: ['Jews', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Captain Androfski, Cleaned: ['``', \"''\", '!'], Tokens: ['``', 'Captain Androfski', \"''\", '!']. Using a random vector as context.\n",
            "8000 sentences processed\n",
            "Accuracy: 42.0485\n",
            "\n",
            "Empty context vector. Word: Precious, Cleaned: ['``', '.'], Tokens: ['``', 'Precious', '.']. Using a random vector as context.\n",
            "8200 sentences processed\n",
            "Accuracy: 41.9407\n",
            "\n",
            "8400 sentences processed\n",
            "Accuracy: 41.8269\n",
            "\n",
            "Empty context vector. Word: Byron, Cleaned: ['``', \"''\", '!'], Tokens: ['``', 'Byron', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Beckworth, Cleaned: ['-', '.'], Tokens: ['-', 'Beckworth', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Sir, Cleaned: ['-', '?'], Tokens: ['-', 'Sir', '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Exactly, Cleaned: ['!'], Tokens: ['Exactly', '!']. Using a random vector as context.\n",
            "8600 sentences processed\n",
            "Accuracy: 41.7817\n",
            "\n",
            "Empty context vector. Word: Henry, Cleaned: ['-', '?'], Tokens: ['-', 'Henry', '?']. Using a random vector as context.\n",
            "8800 sentences processed\n",
            "Accuracy: 41.6943\n",
            "\n",
            "Empty context vector. Word: Abel, Cleaned: [\"''\", '?'], Tokens: ['Abel', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Pedersen, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Pedersen', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Pedersen, Cleaned: ['.'], Tokens: ['Pedersen', '.']. Using a random vector as context.\n",
            "9000 sentences processed\n",
            "Accuracy: 41.6163\n",
            "\n",
            "Empty context vector. Word: Jorge, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'Jorge', \"''\", '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Nothing, Cleaned: ['.'], Tokens: ['Nothing', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: No, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'No', \"''\", '.']. Using a random vector as context.\n",
            "Empty context vector. Word: All right, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'All right', \"''\", '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Shut up, Cleaned: ['``', '.'], Tokens: ['``', 'Shut up', '.']. Using a random vector as context.\n",
            "9200 sentences processed\n",
            "Accuracy: 41.5522\n",
            "\n",
            "Empty context vector. Word: Cousin Ada, Cleaned: ['``', '!'], Tokens: ['``', 'Cousin Ada', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Cousin John, Cleaned: [\"''\", '!'], Tokens: ['Cousin John', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Cousin Lura, Cleaned: ['``', \"''\", '!'], Tokens: ['``', 'Cousin Lura', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Cousin Howard, Cleaned: ['``', \"''\", '!'], Tokens: ['``', 'Cousin Howard', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Howard, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'Howard', \"''\", '.']. Using a random vector as context.\n",
            "Empty sense vector. Word: ruffle, Definition: discompose, Cleaned: ['discompose']. Using a random vector as sense.\n",
            "Empty context vector. Word: All right, Cleaned: ['``', '.'], Tokens: ['``', 'All right', '.']. Using a random vector as context.\n",
            "9400 sentences processed\n",
            "Accuracy: 41.4279\n",
            "\n",
            "Empty context vector. Word: Leona, Cleaned: [\"''\", '!'], Tokens: ['Leona', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Winston, Cleaned: ['``', \"''\", '!'], Tokens: ['``', 'Winston', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Picture, Cleaned: ['``', '!'], Tokens: ['``', 'Picture', '!']. Using a random vector as context.\n",
            "9600 sentences processed\n",
            "Accuracy: 41.3512\n",
            "\n",
            "Empty context vector. Word: big chested, Cleaned: ['big-shouldered', 'heavy-armed'], Tokens: ['He', 'be', 'big chested', ',', 'big-shouldered', 'and', 'heavy-armed', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Felix, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Felix', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Felix Grubb, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Felix Grubb', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Felix Grubb, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Felix Grubb', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: No, Cleaned: ['``', '.'], Tokens: ['``', 'No', '.']. Using a random vector as context.\n",
            "9800 sentences processed\n",
            "Accuracy: 41.3297\n",
            "\n",
            "Empty context vector. Word: Reporters, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Reporters', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: No, Cleaned: ['``', '.'], Tokens: ['``', 'No', '.']. Using a random vector as context.\n",
            "10000 sentences processed\n",
            "Accuracy: 41.2848\n",
            "\n",
            "Empty context vector. Word: Hi, Cleaned: ['``', \"''\", '!'], Tokens: ['``', 'Hi', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Asleep, Cleaned: ['``', '.'], Tokens: ['``', 'Asleep', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Fine, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'Fine', \"''\", '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Mike, Cleaned: ['``', '?'], Tokens: ['``', 'Mike', '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Yes, Cleaned: ['``', '.'], Tokens: ['``', 'Yes', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Yes, Cleaned: ['``', '.'], Tokens: ['``', 'Yes', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: good, Cleaned: ['impresser', 'Pornsen'], Tokens: ['He', 'be', 'a', 'good', 'impresser', ',', 'that', 'Pornsen', '.']. Using a random vector as context.\n",
            "10200 sentences processed\n",
            "Accuracy: 41.2214\n",
            "\n",
            "Empty context vector. Word: Perhaps, Cleaned: ['.'], Tokens: ['Perhaps', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Pornsen, Cleaned: ['.'], Tokens: ['Pornsen', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Jake Carwood, Cleaned: ['``', '.'], Tokens: ['``', 'Jake Carwood', '.']. Using a random vector as context.\n",
            "10400 sentences processed\n",
            "Accuracy: 41.2015\n",
            "\n",
            "Empty context vector. Word: Judith Pierce, Cleaned: ['.'], Tokens: ['Judith Pierce', '.']. Using a random vector as context.\n",
            "10600 sentences processed\n",
            "Accuracy: 41.1543\n",
            "\n",
            "Empty context vector. Word: Good, Cleaned: ['``', \"''\", '!'], Tokens: ['``', 'Good', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Samples, Cleaned: [':'], Tokens: ['Samples', ':']. Using a random vector as context.\n",
            "10800 sentences processed\n",
            "Accuracy: 41.0702\n",
            "\n",
            "Empty context vector. Word: say, Cleaned: ['Furiouser', 'furiouser'], Tokens: ['``', 'Furiouser', 'and', 'furiouser', \"''\", ',', 'I', 'say', '.']. Using a random vector as context.\n",
            "11000 sentences processed\n",
            "Accuracy: 41.0026\n",
            "\n",
            "Empty context vector. Word: S. J. Perelman, Cleaned: ['.'], Tokens: ['S. J. Perelman', '.']. Using a random vector as context.\n",
            "11200 sentences processed\n",
            "Accuracy: 41.0595\n",
            "\n",
            "Empty context vector. Word: Note, Cleaned: ['(', ':'], Tokens: ['(', 'Note', ':']. Using a random vector as context.\n",
            "11400 sentences processed\n",
            "Accuracy: 41.1557\n",
            "\n",
            "11600 sentences processed\n",
            "Accuracy: 41.2245\n",
            "\n",
            "11800 sentences processed\n",
            "Accuracy: 41.2499\n",
            "\n",
            "12000 sentences processed\n",
            "Accuracy: 41.2372\n",
            "\n",
            "12200 sentences processed\n",
            "Accuracy: 41.2889\n",
            "\n",
            "12400 sentences processed\n",
            "Accuracy: 41.3016\n",
            "\n",
            "12600 sentences processed\n",
            "Accuracy: 41.2845\n",
            "\n",
            "12800 sentences processed\n",
            "Accuracy: 41.3701\n",
            "\n",
            "13000 sentences processed\n",
            "Accuracy: 41.3483\n",
            "\n",
            "13200 sentences processed\n",
            "Accuracy: 41.3552\n",
            "\n",
            "13400 sentences processed\n",
            "Accuracy: 41.3818\n",
            "\n",
            "Empty context vector. Word: No more, Cleaned: ['.'], Tokens: ['No more', '.']. Using a random vector as context.\n",
            "13600 sentences processed\n",
            "Accuracy: 41.3470\n",
            "\n",
            "13800 sentences processed\n",
            "Accuracy: 41.3146\n",
            "\n",
            "14000 sentences processed\n",
            "Accuracy: 41.3210\n",
            "\n",
            "14200 sentences processed\n",
            "Accuracy: 41.3136\n",
            "\n",
            "14400 sentences processed\n",
            "Accuracy: 41.3501\n",
            "\n",
            "14600 sentences processed\n",
            "Accuracy: 41.2954\n",
            "\n",
            "14800 sentences processed\n",
            "Accuracy: 41.3260\n",
            "\n",
            "15000 sentences processed\n",
            "Accuracy: 41.3954\n",
            "\n",
            "Empty context vector. Word: Ventilation, Cleaned: ['.'], Tokens: ['Ventilation', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Lighting, Cleaned: ['.'], Tokens: ['Lighting', '.']. Using a random vector as context.\n",
            "15200 sentences processed\n",
            "Accuracy: 41.4300\n",
            "\n",
            "15400 sentences processed\n",
            "Accuracy: 41.4915\n",
            "\n",
            "15600 sentences processed\n",
            "Accuracy: 41.5374\n",
            "\n",
            "15800 sentences processed\n",
            "Accuracy: 41.5757\n",
            "\n",
            "16000 sentences processed\n",
            "Accuracy: 41.5968\n",
            "\n",
            "16200 sentences processed\n",
            "Accuracy: 41.5866\n",
            "\n",
            "16400 sentences processed\n",
            "Accuracy: 41.5954\n",
            "\n",
            "Empty context vector. Word: Sure, Cleaned: ['.'], Tokens: ['Sure', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Sure, Cleaned: ['.'], Tokens: ['Sure', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Katharine, Cleaned: ['.'], Tokens: ['Katharine', '.']. Using a random vector as context.\n",
            "16600 sentences processed\n",
            "Accuracy: 41.5485\n",
            "\n",
            "Empty context vector. Word: Coffee, Cleaned: ['.'], Tokens: ['Coffee', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Maude, Cleaned: ['?'], Tokens: ['Maude', '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Maude, Cleaned: ['.'], Tokens: ['Maude', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Glendora, Cleaned: [\"''\", '-'], Tokens: ['Glendora', \"''\", '-']. Using a random vector as context.\n",
            "Empty context vector. Word: secretary, Cleaned: ['``', 'Gilborn', \"'s\", '?'], Tokens: ['``', 'Gilborn', \"'s\", 'secretary', '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Right, Cleaned: ['``', '.'], Tokens: ['``', 'Right', '.']. Using a random vector as context.\n",
            "16800 sentences processed\n",
            "Accuracy: 41.5147\n",
            "\n",
            "Empty context vector. Word: Dad, Cleaned: ['``', '.'], Tokens: ['``', 'Dad', '.']. Using a random vector as context.\n",
            "17000 sentences processed\n",
            "Accuracy: 41.4580\n",
            "\n",
            "Empty context vector. Word: Mae, Cleaned: ['``', \"''\", '-'], Tokens: ['``', 'Mae', \"''\", '-']. Using a random vector as context.\n",
            "17200 sentences processed\n",
            "Accuracy: 41.4336\n",
            "\n",
            "Empty context vector. Word: Motive, Cleaned: ['.'], Tokens: ['Motive', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Right, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Right', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Mullins, Cleaned: ['?'], Tokens: ['Mullins', '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Motive, Cleaned: ['?'], Tokens: ['Motive', '?']. Using a random vector as context.\n",
            "17400 sentences processed\n",
            "Accuracy: 41.4045\n",
            "\n",
            "Empty context vector. Word: bequest, Cleaned: ['Ten-thousand-dollar', '.'], Tokens: ['Ten-thousand-dollar', 'bequest', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Impossible, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'Impossible', \"''\", '.']. Using a random vector as context.\n",
            "Empty context vector. Word: office, Cleaned: ['Mahzeer', \"'s\", \"''\", '.'], Tokens: ['Mahzeer', \"'s\", 'office', \"''\", '.']. Using a random vector as context.\n",
            "17600 sentences processed\n",
            "Accuracy: 41.3900\n",
            "\n",
            "Empty context vector. Word: Sir, Cleaned: ['``', '.'], Tokens: ['``', 'Sir', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Jack, Cleaned: [':'], Tokens: ['Jack', ':']. Using a random vector as context.\n",
            "Empty context vector. Word: Casey, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Casey', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Tony Calenda, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'Tony Calenda', \"''\", '.']. Using a random vector as context.\n",
            "17800 sentences processed\n",
            "Accuracy: 41.3600\n",
            "\n",
            "Empty context vector. Word: Casey, Cleaned: ['``', '?'], Tokens: ['``', 'Casey', '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Tom, Cleaned: ['``', '!'], Tokens: ['``', 'Tom', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Donna, Cleaned: ['!'], Tokens: ['Donna', '!']. Using a random vector as context.\n",
            "18000 sentences processed\n",
            "Accuracy: 41.3321\n",
            "\n",
            "Empty context vector. Word: Five, Cleaned: ['?'], Tokens: ['Five', '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Ten, Cleaned: ['?'], Tokens: ['Ten', '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Key, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Key', \"''\", '?']. Using a random vector as context.\n",
            "18200 sentences processed\n",
            "Accuracy: 41.2999\n",
            "\n",
            "Empty context vector. Word: Sportin, Cleaned: ['``', \"''\", \"'\", '!'], Tokens: ['``', 'Sportin', \"''\", \"'\", '!']. Using a random vector as context.\n",
            "18400 sentences processed\n",
            "Accuracy: 41.2709\n",
            "\n",
            "Empty context vector. Word: Arbuckle, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Arbuckle', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: No, Cleaned: ['``', '.'], Tokens: ['``', 'No', '.']. Using a random vector as context.\n",
            "18600 sentences processed\n",
            "Accuracy: 41.2087\n",
            "\n",
            "18800 sentences processed\n",
            "Accuracy: 41.1570\n",
            "\n",
            "Empty context vector. Word: Now, Cleaned: ['!'], Tokens: ['Now', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Get in, Cleaned: [\"''\", '.'], Tokens: ['Get in', \"''\", '.']. Using a random vector as context.\n",
            "Empty context vector. Word: S-s-sahjunt, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'S-s-sahjunt', \"''\", '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Sure, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'Sure', \"''\", '.']. Using a random vector as context.\n",
            "19000 sentences processed\n",
            "Accuracy: 41.1103\n",
            "\n",
            "Empty context vector. Word: Splendid, Cleaned: ['``', '.'], Tokens: ['``', 'Splendid', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Murder, Cleaned: ['``', '?'], Tokens: ['``', 'Murder', '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Itch, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Itch', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: Goodbye, Cleaned: ['``', '.'], Tokens: ['``', 'Goodbye', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Softly, Cleaned: ['.'], Tokens: ['Softly', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Warmly, Cleaned: ['.'], Tokens: ['Warmly', '.']. Using a random vector as context.\n",
            "19200 sentences processed\n",
            "Accuracy: 41.0666\n",
            "\n",
            "Empty context vector. Word: Very well, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'Very well', \"''\", '.']. Using a random vector as context.\n",
            "19400 sentences processed\n",
            "Accuracy: 41.0209\n",
            "\n",
            "Empty context vector. Word: Osric Pendleton, Cleaned: ['``', \"''\", '?'], Tokens: ['``', 'Osric Pendleton', \"''\", '?']. Using a random vector as context.\n",
            "Empty context vector. Word: laugh, Cleaned: ['Askington', '.'], Tokens: ['Askington', 'laugh', '.']. Using a random vector as context.\n",
            "19600 sentences processed\n",
            "Accuracy: 40.9861\n",
            "\n",
            "Empty context vector. Word: Nothing, Cleaned: ['``', '.'], Tokens: ['``', 'Nothing', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Third, Cleaned: ['``', '!'], Tokens: ['``', 'Third', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Third base, Cleaned: [\"''\", '!'], Tokens: ['Third base', \"''\", '!']. Using a random vector as context.\n",
            "19800 sentences processed\n",
            "Accuracy: 40.9453\n",
            "\n",
            "20000 sentences processed\n",
            "Accuracy: 40.9187\n",
            "\n",
            "20200 sentences processed\n",
            "Accuracy: 40.8799\n",
            "\n",
            "20400 sentences processed\n",
            "Accuracy: 40.8224\n",
            "\n",
            "20600 sentences processed\n",
            "Accuracy: 40.7789\n",
            "\n",
            "20800 sentences processed\n",
            "Accuracy: 40.7343\n",
            "\n",
            "21000 sentences processed\n",
            "Accuracy: 40.6957\n",
            "\n",
            "21200 sentences processed\n",
            "Accuracy: 40.6486\n",
            "\n",
            "21400 sentences processed\n",
            "Accuracy: 40.6215\n",
            "\n",
            "21600 sentences processed\n",
            "Accuracy: 40.5706\n",
            "\n",
            "21800 sentences processed\n",
            "Accuracy: 40.5293\n",
            "\n",
            "22000 sentences processed\n",
            "Accuracy: 40.4984\n",
            "\n",
            "22200 sentences processed\n",
            "Accuracy: 40.4671\n",
            "\n",
            "22400 sentences processed\n",
            "Accuracy: 40.4237\n",
            "\n",
            "22600 sentences processed\n",
            "Accuracy: 40.3827\n",
            "\n",
            "22800 sentences processed\n",
            "Accuracy: 40.3292\n",
            "\n",
            "23000 sentences processed\n",
            "Accuracy: 40.2797\n",
            "\n",
            "23200 sentences processed\n",
            "Accuracy: 40.2372\n",
            "\n",
            "23400 sentences processed\n",
            "Accuracy: 40.2074\n",
            "\n",
            "23600 sentences processed\n",
            "Accuracy: 40.1751\n",
            "\n",
            "23800 sentences processed\n",
            "Accuracy: 40.1290\n",
            "\n",
            "24000 sentences processed\n",
            "Accuracy: 40.0813\n",
            "\n",
            "24200 sentences processed\n",
            "Accuracy: 40.0369\n",
            "\n",
            "24400 sentences processed\n",
            "Accuracy: 39.9888\n",
            "\n",
            "24600 sentences processed\n",
            "Accuracy: 39.9531\n",
            "\n",
            "24800 sentences processed\n",
            "Accuracy: 39.9136\n",
            "\n",
            "25000 sentences processed\n",
            "Accuracy: 39.8694\n",
            "\n",
            "25200 sentences processed\n",
            "Accuracy: 39.8180\n",
            "\n",
            "25400 sentences processed\n",
            "Accuracy: 39.7825\n",
            "\n",
            "Empty context vector. Word: understood, Cleaned: ['Leninism-Marxism', 'Exegete'], Tokens: ['Leninism-Marxism', ',', 'a', 'understood', 'by', 'Exegete', '.']. Using a random vector as context.\n",
            "25600 sentences processed\n",
            "Accuracy: 39.7382\n",
            "\n",
            "25800 sentences processed\n",
            "Accuracy: 39.6999\n",
            "\n",
            "26000 sentences processed\n",
            "Accuracy: 39.6578\n",
            "\n",
            "26200 sentences processed\n",
            "Accuracy: 39.6238\n",
            "\n",
            "26400 sentences processed\n",
            "Accuracy: 39.5847\n",
            "\n",
            "26600 sentences processed\n",
            "Accuracy: 39.5362\n",
            "\n",
            "26800 sentences processed\n",
            "Accuracy: 39.5061\n",
            "\n",
            "27000 sentences processed\n",
            "Accuracy: 39.4673\n",
            "\n",
            "27200 sentences processed\n",
            "Accuracy: 39.4273\n",
            "\n",
            "27400 sentences processed\n",
            "Accuracy: 39.3793\n",
            "\n",
            "27600 sentences processed\n",
            "Accuracy: 39.3326\n",
            "\n",
            "27800 sentences processed\n",
            "Accuracy: 39.2911\n",
            "\n",
            "28000 sentences processed\n",
            "Accuracy: 39.2315\n",
            "\n",
            "28200 sentences processed\n",
            "Accuracy: 39.1983\n",
            "\n",
            "28400 sentences processed\n",
            "Accuracy: 39.1633\n",
            "\n",
            "28600 sentences processed\n",
            "Accuracy: 39.1249\n",
            "\n",
            "28800 sentences processed\n",
            "Accuracy: 39.0836\n",
            "\n",
            "29000 sentences processed\n",
            "Accuracy: 39.0642\n",
            "\n",
            "29200 sentences processed\n",
            "Accuracy: 39.0268\n",
            "\n",
            "29400 sentences processed\n",
            "Accuracy: 38.9958\n",
            "\n",
            "Empty context vector. Word: Simmer, Cleaned: ['15', '.'], Tokens: ['Simmer', '15', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Serves, Cleaned: ['12', '.'], Tokens: ['Serves', '12', '.']. Using a random vector as context.\n",
            "29600 sentences processed\n",
            "Accuracy: 38.9645\n",
            "\n",
            "29800 sentences processed\n",
            "Accuracy: 38.9403\n",
            "\n",
            "30000 sentences processed\n",
            "Accuracy: 38.9142\n",
            "\n",
            "30200 sentences processed\n",
            "Accuracy: 38.8837\n",
            "\n",
            "Empty context vector. Word: Stop, Cleaned: ['``', '!'], Tokens: ['``', 'Stop', '!']. Using a random vector as context.\n",
            "30400 sentences processed\n",
            "Accuracy: 38.8571\n",
            "\n",
            "30600 sentences processed\n",
            "Accuracy: 38.8205\n",
            "\n",
            "30800 sentences processed\n",
            "Accuracy: 38.7898\n",
            "\n",
            "31000 sentences processed\n",
            "Accuracy: 38.7518\n",
            "\n",
            "31200 sentences processed\n",
            "Accuracy: 38.7069\n",
            "\n",
            "Empty sense vector. Word: ruffle, Definition: discompose, Cleaned: ['discompose']. Using a random vector as sense.\n",
            "31400 sentences processed\n",
            "Accuracy: 38.6848\n",
            "\n",
            "31600 sentences processed\n",
            "Accuracy: 38.6596\n",
            "\n",
            "31800 sentences processed\n",
            "Accuracy: 38.6283\n",
            "\n",
            "32000 sentences processed\n",
            "Accuracy: 38.5975\n",
            "\n",
            "32200 sentences processed\n",
            "Accuracy: 38.5578\n",
            "\n",
            "32400 sentences processed\n",
            "Accuracy: 38.5335\n",
            "\n",
            "32600 sentences processed\n",
            "Accuracy: 38.5004\n",
            "\n",
            "32800 sentences processed\n",
            "Accuracy: 38.4721\n",
            "\n",
            "33000 sentences processed\n",
            "Accuracy: 38.4400\n",
            "\n",
            "33200 sentences processed\n",
            "Accuracy: 38.4139\n",
            "\n",
            "33400 sentences processed\n",
            "Accuracy: 38.3657\n",
            "\n",
            "33600 sentences processed\n",
            "Accuracy: 38.3243\n",
            "\n",
            "33800 sentences processed\n",
            "Accuracy: 38.2977\n",
            "\n",
            "34000 sentences processed\n",
            "Accuracy: 38.2755\n",
            "\n",
            "34200 sentences processed\n",
            "Accuracy: 38.2540\n",
            "\n",
            "34400 sentences processed\n",
            "Accuracy: 38.2236\n",
            "\n",
            "Empty context vector. Word: Receiving, Cleaned: ['``', \"''\", '.'], Tokens: ['``', 'Receiving', \"''\", '.']. Using a random vector as context.\n",
            "Empty context vector. Word: ask, Cleaned: ['Ekstrohm', '.'], Tokens: ['Ekstrohm', 'ask', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: Take it easy, Cleaned: [\"''\", '.'], Tokens: ['Take it easy', \"''\", '.']. Using a random vector as context.\n",
            "Empty context vector. Word: suggest, Cleaned: ['Ekstrohm', '.'], Tokens: ['Ekstrohm', 'suggest', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: stood up, Cleaned: ['Ekstrohm', '.'], Tokens: ['Ekstrohm', 'stood up', '.']. Using a random vector as context.\n",
            "34600 sentences processed\n",
            "Accuracy: 38.2022\n",
            "\n",
            "Empty context vector. Word: demand, Cleaned: ['Ekstrohm', '.'], Tokens: ['Ekstrohm', 'demand', '.']. Using a random vector as context.\n",
            "Empty context vector. Word: scowl, Cleaned: ['Ekstrohm', '.'], Tokens: ['Ekstrohm', 'scowl', '.']. Using a random vector as context.\n",
            "34800 sentences processed\n",
            "Accuracy: 38.1849\n",
            "\n",
            "Empty context vector. Word: Get up, Cleaned: ['``', '.'], Tokens: ['``', 'Get up', '.']. Using a random vector as context.\n",
            "35000 sentences processed\n",
            "Accuracy: 38.1529\n",
            "\n",
            "35200 sentences processed\n",
            "Accuracy: 38.1341\n",
            "\n",
            "35400 sentences processed\n",
            "Accuracy: 38.1133\n",
            "\n",
            "Empty context vector. Word: Hurry, Cleaned: ['``', '!'], Tokens: ['``', 'Hurry', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Hurry, Cleaned: [\"''\", '!'], Tokens: ['Hurry', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Hustle, Cleaned: [\"''\", '!'], Tokens: ['Hustle', \"''\", '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Fort up, Cleaned: ['!'], Tokens: ['Fort up', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Lead up, Cleaned: ['``', '!'], Tokens: ['``', 'Lead up', '!']. Using a random vector as context.\n",
            "Empty context vector. Word: Lead up, Cleaned: ['!'], Tokens: ['Lead up', '!']. Using a random vector as context.\n",
            "35600 sentences processed\n",
            "Accuracy: 38.0922\n",
            "\n",
            "35800 sentences processed\n",
            "Accuracy: 38.0686\n",
            "\n",
            "36000 sentences processed\n",
            "Accuracy: 38.0396\n",
            "\n",
            "36200 sentences processed\n",
            "Accuracy: 38.0112\n",
            "\n",
            "36400 sentences processed\n",
            "Accuracy: 37.9746\n",
            "\n",
            "Empty context vector. Word: Take off, Cleaned: ['``', ',', 'fly-boy', \"''\", '!'], Tokens: ['``', 'Take off', ',', 'fly-boy', \"''\", '!']. Using a random vector as context.\n",
            "36600 sentences processed\n",
            "Accuracy: 37.9392\n",
            "\n",
            "36800 sentences processed\n",
            "Accuracy: 37.9151\n",
            "\n",
            "37000 sentences processed\n",
            "Accuracy: 37.8907\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0yKCvj0mcu_1"
      },
      "source": [
        "pred_sense_set = set(pred)\n",
        "true_sense_set = set(true)\n",
        "all_senses = sorted(list(true_sense_set.union(pred_sense_set)))\n",
        "not_predicted = true_sense_set - pred_sense_set\n",
        "extra_predicted = pred_sense_set - true_sense_set"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eS2imPibGSEe",
        "outputId": "a1fe36e2-3cdd-4960-da10-a3edf3a271c4"
      },
      "source": [
        "acc = accuracy_score(true, pred)\n",
        "prec = precision_score(true, pred, average = \"macro\")\n",
        "rec = recall_score(true, pred, average = \"macro\")\n",
        "f1 = f1_score(true, pred, average = \"macro\")\n",
        "\n",
        "print(f\"Accuracy: {acc:.4f}\")\n",
        "print(f\"Precision: {prec:.4f}\")\n",
        "print(f\"Recall: {rec:.4f}\")\n",
        "print(f\"F1-Score: {f1:.4f}\")"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 0.3786\n",
            "Precision: 0.4136\n",
            "Recall: 0.4009\n",
            "F1-Score: 0.3776\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bsb2D-5bfxE7"
      },
      "source": [
        "def predict(sent):\n",
        "\n",
        "    senses = []\n",
        "    tokens = word_tokenize(sent)\n",
        "    # Tag and lemmatize tokens, don't remove stopwords here\n",
        "    tagged = nltk.pos_tag(tokens)\n",
        "    tags = [treebank2wn(p[1]) for p in tagged]\n",
        "    tokens = [lemmatize(w, tag) for w, tag in zip(tokens, tags)]\n",
        "    n_tokens = len(tokens)\n",
        "\n",
        "    for i in range(n_tokens):\n",
        "\n",
        "        w = tokens[i]\n",
        "        tag = tags[i]\n",
        "\n",
        "        # Get context for w (all words in the sentence except w)\n",
        "        context = tokens.copy()\n",
        "        del context[i] # more efficient than .pop(i)\n",
        "\n",
        "        # Get context vector by average w2v vectors for each word\n",
        "        cv = sent2vec(context)\n",
        "\n",
        "        if cv is None:\n",
        "            print(f\"Empty context vector. Word: {w}, Tokens: {tokens}. Using a random vector as context.\")\n",
        "            cv = np.random.rand(300,)\n",
        "\n",
        "        # Get WordNet candidate senses\n",
        "        sense_vectors, sense_labels = getCandidates(w, tag)\n",
        "        n_candidates = len(sense_labels)\n",
        "\n",
        "        s_pred = None\n",
        "        if n_candidates == 0:\n",
        "            # Try without pos tag\n",
        "            sense_vectors, sense_labels = getCandidates(w, None)\n",
        "            n_candidates = len(sense_labels)\n",
        "            if n_candidates == 0:\n",
        "                print(f\"No synsets found: {w}\")\n",
        "                s_pred = None\n",
        "\n",
        "        # Use cosine similarity to get the best senses\n",
        "        best = -1 \n",
        "        for j in range(n_candidates):\n",
        "            sv = sense_vectors[j]\n",
        "            cs = cosineSimilarity(cv, sv)\n",
        "            if cs > best:\n",
        "                best = cs\n",
        "                s_pred = sense_labels[j]\n",
        "\n",
        "        senses.append(s_pred)\n",
        "\n",
        "    return senses"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ez6y5gsegqYt",
        "outputId": "bc1e4616-7270-4d8d-ca76-b0e4ebc3680b"
      },
      "source": [
        "sents = [\n",
        "    \"On combustion of coal we get ash\", \n",
        "    \"The bank is located in the city near the river\",\n",
        "    \"The stolen credit cards were found near the river bank\",\n",
        "    \"The user had to kill the computer process\",\n",
        "    \"The trees near Nuclear Power Plant were cut down\"\n",
        "]\n",
        "\n",
        "for sent in sents:\n",
        "    senses = predict(sent)\n",
        "    for s in senses:\n",
        "        if s is not None:\n",
        "            print(s, \":\", wn.synset(s).definition())\n",
        "    print()"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "No synsets found: of\n",
            "No synsets found: we\n",
            "on.r.03 : in a state required for something to function or be effective\n",
            "combustion.n.01 : a process in which a substance reacts with oxygen to give heat and light\n",
            "ember.n.01 : a hot fragment of wood or coal that is left from a fire and is glowing or smoldering\n",
            "get.v.01 : come into the possession of something concrete or abstract\n",
            "ash.n.01 : the residue that remains when something is burned\n",
            "\n",
            "No synsets found: The\n",
            "No synsets found: the\n",
            "No synsets found: the\n",
            "bank.n.07 : a slope in the turn of a road or track; the outside is higher than the inside in order to reduce the effects of centrifugal force\n",
            "be.v.03 : occupy a certain position or area; be somewhere\n",
            "situate.v.01 : determine or indicate the place, site, or limits of, as if by an instrument or by a survey\n",
            "in.r.01 : to or toward the inside of\n",
            "city.n.01 : a large and densely populated urban area; may include several independent administrative districts\n",
            "near.r.01 : near in time or place or relationship\n",
            "river.n.01 : a large natural stream of water (larger than a creek)\n",
            "\n",
            "No synsets found: The\n",
            "No synsets found: the\n",
            "steal.v.01 : take without the owner's consent\n",
            "credit.n.03 : an accounting entry acknowledging income or capital items\n",
            "card.n.01 : one of a set of small pieces of stiff paper marked in various ways and used for playing games or for telling fortunes\n",
            "be.v.03 : occupy a certain position or area; be somewhere\n",
            "find.v.03 : come upon after searching; find the location of something that was missed or lost\n",
            "near.r.01 : near in time or place or relationship\n",
            "river.n.01 : a large natural stream of water (larger than a creek)\n",
            "savings_bank.n.02 : a container (usually with a slot in the top) for keeping money at home\n",
            "\n",
            "No synsets found: The\n",
            "No synsets found: to\n",
            "No synsets found: the\n",
            "user.n.01 : a person who makes use of a thing; someone who uses or employs something\n",
            "have.v.07 : have a personal or business relationship with someone\n",
            "kill.v.07 : hit with so much force as to make a return impossible, in racket games\n",
            "computer.n.01 : a machine for performing calculations automatically\n",
            "process.n.02 : (psychology) the performance of some composite cognitive activity; an operation that affects mental contents\n",
            "\n",
            "No synsets found: The\n",
            "tree.n.01 : a tall perennial woody plant having a main trunk and branches forming a distinct elevated crown; includes both gymnosperms and angiosperms\n",
            "about.r.07 : (of actions or states) slightly short of or not quite accomplished; all but\n",
            "nuclear.a.01 : (weapons) deriving destructive energy from the release of atomic energy\n",
            "office.n.04 : (of a government or government official) holding an office means being in power\n",
            "plant.n.04 : something planted secretly for discovery by another\n",
            "be.v.05 : happen, occur, take place; this was during the visit to my parents' house\"\n",
            "edit.v.03 : cut and assemble the components of\n",
            "down.r.02 : away from a more central or a more northerly place\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TAQF8ZDlKh2h"
      },
      "source": [
        "## References\n",
        "- [NLTK Trees](https://stackoverflow.com/questions/62472606/get-the-type-of-a-nltk-tree)\n",
        "- [NLTK WordNet Lemma](https://www.nltk.org/_modules/nltk/corpus/reader/wordnet.html)  \n",
        "`Lemma` attributes, accessible via methods with the same name:\n",
        "    - name: The canonical name of this lemma.\n",
        "    - synset: The synset that this lemma belongs to.\n",
        "    - syntactic_marker: For adjectives, the WordNet string identifying the\n",
        "        syntactic position relative modified noun. See:\n",
        "        https://wordnet.princeton.edu/documentation/wninput5wn\n",
        "        For all other parts of speech, this attribute is None.\n",
        "    - count: The frequency of this lemma in wordnet."
      ]
    }
  ]
}