Flair_text_classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Flair_text_classification.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/navneetkrc/Flair_SOTA_NLP/blob/master/Flair_text_classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "N358tpRYaw7G",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This is an implementation based on this [Analytics Vidhya Blog].(https://www.analyticsvidhya.com/blog/2019/02/flair-nlp-library-python/)"
      ]
    },
    {
      "metadata": {
        "id": "CRI11WBajJnQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Dataset\n",
        "We’ll be working on the [AV Twitter Sentiment Analysis](https://datahack.analyticsvidhya.com/contest/practice-problem-twitter-sentiment-analysis/)  practice problem. Go ahead and download the dataset from there (you’ll need to register/log in first).\n",
        "\n",
        "The problem statement posed by this challenge is:\n",
        "\n",
        "The objective of this task is to detect hate speech in tweets. For the sake of simplicity, we say a tweet contains hate speech if it has a racist or sexist sentiment associated with it. So, the task is to classify racist or sexist tweets from other tweets.\n"
      ]
    },
    {
      "metadata": {
        "id": "zXkMo_cHjeiy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Steps Involved\n",
        "**1. Text Classification Using Flair Embeddings**\n",
        "\n",
        "Overview of steps:\n",
        "\n",
        "Step 1: Import the data into the local Environment of Colab:\n",
        "\n",
        "Step 2: Installing Flair\n",
        "\n",
        "Step 3: Preparing text to work with Flair\n",
        "\n",
        "Step 4: Word Embeddings with Flair\n",
        "\n",
        "Step 5: Vectorizing the text\n",
        "\n",
        "Step 6: Partitioning the data for Train and Test Sets\n",
        "\n",
        "Step 7: Time for predictions!"
      ]
    },
    {
      "metadata": {
        "id": "UofJ-GS2jxGZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 1: Import the data into the local Environment of Colab:"
      ]
    },
    {
      "metadata": {
        "id": "0RkGbxD2jwAw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Install the PyDrive wrapper & import libraries.\n",
        "# This only needs to be done once per notebook.\n",
        "\n",
        "!pip install -U -q PyDrive\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        "\n",
        "\n",
        "# Authenticate and create the PyDrive client.\n",
        "# This only needs to be done once per notebook.\n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)\n",
        "\n",
        "# Download a file based on its file ID.\n",
        "# A file ID looks like: laggVyWshwcyP6kEI-y_W3P8D26sz\n",
        "file_id = '1GhyH4k9C4uPRnMAMKhJYOqa-V9Tqt4q8' ### File ID ###\n",
        "data = drive.CreateFile({'id': file_id})\n",
        "#print('Downloaded content \"{}\"'.format(downloaded.GetContentString()))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "rxhMmSS-kA9A",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#Import Dataset in Colab Notebook\n",
        "import io\n",
        "import pandas as pd\n",
        "data = pd.read_csv(io.StringIO(data.GetContentString())) \n",
        "data.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "E6JL5aW5ktsL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###Step 2 Download Flair Library"
      ]
    },
    {
      "metadata": {
        "id": "dA8H7XH7kkGq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# download flair library #\n",
        "import torch\n",
        "!pip install flair\n",
        "import flair"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NmEKS0I8k0Ld",
        "colab_type": "code",
        "outputId": "48a9ff9a-df0a-45ba-9a76-fb59cc5f53a5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "from flair.data import Sentence\n",
        "# create a sentence #\n",
        "sentence = Sentence('Blogs of Analytics Vidhya are Awesome.')\n",
        "# print the sentence to see what’s in it. #\n",
        "print(sentence)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentence: \"Blogs of Analytics Vidhya are Awesome.\" - 6 Tokens\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "nlJTvhIEk9K4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###Step 3: Preparing text to work with Flair\n"
      ]
    },
    {
      "metadata": {
        "id": "SIacRL7Dk6lZ",
        "colab_type": "code",
        "outputId": "629908db-8a63-4404-caf6-25936d3ac1d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "cell_type": "code",
      "source": [
        "#extracting the tweet part#\n",
        "text = data['tweet'] \n",
        " ## txt is a list of tweets ##\n",
        "txt = text.tolist()\n",
        "print(txt[:10])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['  user when a father is dysfunctional and is so selfish he drags his kids into his dysfunction     run', ' user  user thanks for  lyft credit i can t use cause they don t offer wheelchair vans in pdx      disapointed  getthanked', '  bihday your majesty', ' model   i love u take with u all the time in ur                                      ', ' factsguide  society now     motivation', '      huge fan fare and big talking before they leave  chaos and pay disputes when they get there   allshowandnogo  ', '  user camping tomorrow  user  user  user  user  user  user  user danny   ', 'the next school year is the year for exams      can t think about that       school  exams    hate  imagine  actorslife  revolutionschool  girl', 'we won    love the land     allin  cavs  champions  cleveland  clevelandcavaliers      ', '  user  user welcome here    i m   it s so  gr    ']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "bqqE8Ng5lRRE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###Step 4: Word Embeddings with Flair"
      ]
    },
    {
      "metadata": {
        "id": "WxIvTYPHlFzf",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\n",
        "## Importing the Embeddings ##\n",
        "from flair.embeddings import WordEmbeddings\n",
        "from flair.embeddings import CharacterEmbeddings\n",
        "from flair.embeddings import StackedEmbeddings\n",
        "from flair.embeddings import FlairEmbeddings\n",
        "from flair.embeddings import BertEmbeddings\n",
        "from flair.embeddings import ELMoEmbeddings\n",
        "from flair.embeddings import FlairEmbeddings\n",
        "\n",
        "### Initialising embeddings (un-comment to use others) ###\n",
        "#glove_embedding = WordEmbeddings('glove')\n",
        "#character_embeddings = CharacterEmbeddings()\n",
        "flair_forward  = FlairEmbeddings('news-forward-fast')\n",
        "flair_backward = FlairEmbeddings('news-backward-fast')\n",
        "#bert_embedding = BertEmbedding()\n",
        "#elmo_embedding = ElmoEmbedding()\n",
        "\n",
        "stacked_embeddings = StackedEmbeddings( embeddings = [ \n",
        "                                                       flair_forward, \n",
        "                                                       flair_backward\n",
        "                                                      ])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ILdEvY83lh7Q",
        "colab_type": "code",
        "outputId": "75ebcb1d-384b-4576-9654-b5c47aea111f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        }
      },
      "cell_type": "code",
      "source": [
        "#Testing the stacked embeddings:\n",
        "\n",
        "# create a sentence #\n",
        "sentence = Sentence('Blogs of Analytics Vidhya are Awesome.')\n",
        "# embed words in sentence #\n",
        "stacked_embeddings.embed(sentence)\n",
        "\n",
        "for token in sentence:\n",
        "  print(token.embedding)\n",
        "# data type and size of embedding #\n",
        "print(type(token.embedding))\n",
        "# storing size (length) #\n",
        "z = token.embedding.size()[0]"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([ 3.0279e-04, -1.4077e-07,  2.6455e-06,  ..., -1.1807e-07,\n",
            "        -4.5203e-06,  3.4654e-03])\n",
            "tensor([-7.3398e-03, -4.8201e-05,  1.2195e-07,  ..., -1.3866e-08,\n",
            "        -1.9298e-04,  5.3008e-03])\n",
            "tensor([ 2.1015e-03, -5.1521e-06,  9.0945e-08,  ..., -3.9210e-09,\n",
            "         1.5152e-05,  1.3080e-02])\n",
            "tensor([-3.6214e-03, -1.4667e-06,  6.8676e-07,  ..., -3.8634e-08,\n",
            "         2.1911e-04,  1.7681e-02])\n",
            "tensor([ 2.5456e-03,  3.4033e-06,  3.1239e-06,  ..., -5.5899e-08,\n",
            "        -1.3424e-04,  6.0970e-03])\n",
            "tensor([-1.0973e-04,  6.7579e-07,  4.5737e-08,  ..., -7.0676e-09,\n",
            "        -8.7311e-04,  4.5264e-03])\n",
            "<class 'torch.Tensor'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ONgzvHv7oOda",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###Step 5: Vectorizing the text\n",
        "\n",
        "**We’ll be showcasing this using two approaches.**\n",
        "\n",
        " \n",
        "\n",
        "Mean of Word Embeddings within a Tweet\n",
        "\n",
        "We will be calculating the following in this approach:\n",
        "\n",
        "For each sentence:\n",
        "\n",
        "1.   Generate word embedding for each word\n",
        "\n",
        "2.   Calculate the mean of the embeddings of each word to obtain the embedding of the sentence\n"
      ]
    },
    {
      "metadata": {
        "id": "nn-r2-ZymP3X",
        "colab_type": "code",
        "outputId": "277a2761-a295-4ef0-ad04-bee88ff6bae4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "from tqdm import tqdm ## tracks progress of loop ##\n",
        "\n",
        "# creating a tensor for storing sentence embeddings #\n",
        "s = torch.zeros(0,z)\n",
        "\n",
        "# iterating Sentence (tqdm tracks progress) #\n",
        "for tweet in tqdm(txt):   \n",
        "  # empty tensor for words #\n",
        "  w = torch.zeros(0,z)   \n",
        "  sentence = Sentence(tweet)\n",
        "  stacked_embeddings.embed(sentence)\n",
        "  # for every word #\n",
        "  for token in sentence:\n",
        "    # storing word Embeddings of each word in a sentence #\n",
        "    w = torch.cat((w,token.embedding.view(-1,z)),0)\n",
        "  # storing sentence Embeddings (mean of embeddings of all words)   #\n",
        "  s = torch.cat((s, w.mean(dim = 0).view(-1, z)),0)\n"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 49159/49159 [1:12:17<00:00,  6.37it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "JMMB8bqHots0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###Document Embedding: Vectorizing the entire Tweet\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "jY18rAAfosI2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "9786095b-1701-48b7-d191-dfa278f221e6"
      },
      "cell_type": "code",
      "source": [
        "from flair.embeddings import DocumentPoolEmbeddings\n",
        "\n",
        "### initialize the document embeddings, mode = mean ###\n",
        "document_embeddings = DocumentPoolEmbeddings([\n",
        "                                              flair_backward,\n",
        "                                              flair_forward\n",
        "                                             ])\n",
        "# Storing Size of embedding #\n",
        "z = sentence.embedding.size()[0]  #value here was 1 in the tutorial\n",
        "\n",
        "### Vectorising text ###\n",
        "# creating a tensor for storing sentence embeddings\n",
        "s = torch.zeros(0,z)\n",
        "# iterating Sentences #\n",
        "for tweet in tqdm(txt):   \n",
        "  sentence = Sentence(tweet)\n",
        "  document_embeddings.embed(sentence)\n",
        "  # Adding Document embeddings to list #\n",
        "  s = torch.cat((s, sentence.embedding.view(-1,z)),0)\n"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 49159/49159 [1:12:11<00:00,  6.51it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "jVJFdDHXo7hG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "You can choose either approach for your model. Now that our text is vectorised, we can feed it to our machine learning model!\n"
      ]
    },
    {
      "metadata": {
        "id": "sZxvAQWao9aC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###Step 6: Partitioning the data for Train and Test Sets\n"
      ]
    },
    {
      "metadata": {
        "id": "NjkHPhP2pHKx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "## tensor to numpy array ##\n",
        "X = s.numpy()   \n",
        "\n",
        "## Test set ##\n",
        "test = X[31962:,:]\n",
        "train = X[:31962,:]\n",
        "\n",
        "# extracting labels of the training set #\n",
        "target = data['label'][data['label'].isnull()==False].values"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DyK_nibMpSp_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###Step 6: Building the Model and Defining Custom Evaluator (for F1 Score)\n"
      ]
    },
    {
      "metadata": {
        "id": "mvOzRDX7pN88",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#Defining custom F1 evaluator for XGBoost\n",
        "def custom_eval(preds, dtrain):\n",
        "    labels = dtrain.get_label().astype(np.int)\n",
        "    preds = (preds >= 0.3).astype(np.int)\n",
        "    return [('f1_score', f1_score(labels, preds))]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4_Dx-i9RpiJ5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Building XGBoost Model**"
      ]
    },
    {
      "metadata": {
        "id": "ROn36X9OpeMI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2482
        },
        "outputId": "628ebb4d-5391-4eb0-c644-8401728711e2"
      },
      "cell_type": "code",
      "source": [
        "import xgboost as xgb\n",
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import f1_score\n",
        "\n",
        "### Splitting training set ###\n",
        "x_train, x_valid, y_train, y_valid = train_test_split(train, target,  \n",
        "                                                      random_state=42, \n",
        "                                                          test_size=0.3)\n",
        "\n",
        "### XGBoost compatible data ###\n",
        "dtrain = xgb.DMatrix(x_train,y_train)         \n",
        "dvalid = xgb.DMatrix(x_valid, label = y_valid)\n",
        "\n",
        "### defining parameters ###\n",
        "params = {\n",
        "          'colsample': 0.9,\n",
        "          'colsample_bytree': 0.5,\n",
        "          'eta': 0.1,\n",
        "          'max_depth': 8,\n",
        "          'min_child_weight': 6,\n",
        "          'objective': 'binary:logistic',\n",
        "          'subsample': 0.9\n",
        "          }\n",
        "\n",
        "### Training the model ###\n",
        "xgb_model = xgb.train(\n",
        "                      params,\n",
        "                      dtrain,\n",
        "                      feval= custom_eval,\n",
        "                      num_boost_round= 1000,\n",
        "                      maximize=True,\n",
        "                      evals=[(dvalid, \"Validation\")],\n",
        "                      early_stopping_rounds=30\n",
        "                      )"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[0]\tValidation-error:0.070289\tValidation-f1_score:0.133165\n",
            "Multiple eval metrics have been passed: 'Validation-f1_score' will be used for early stopping.\n",
            "\n",
            "Will train until Validation-f1_score hasn't improved in 30 rounds.\n",
            "[1]\tValidation-error:0.064657\tValidation-f1_score:0.133165\n",
            "[2]\tValidation-error:0.063615\tValidation-f1_score:0.133165\n",
            "[3]\tValidation-error:0.063719\tValidation-f1_score:0.133165\n",
            "[4]\tValidation-error:0.06351\tValidation-f1_score:0.133165\n",
            "[5]\tValidation-error:0.063093\tValidation-f1_score:0.29055\n",
            "[6]\tValidation-error:0.06351\tValidation-f1_score:0.370826\n",
            "[7]\tValidation-error:0.063302\tValidation-f1_score:0.408986\n",
            "[8]\tValidation-error:0.063197\tValidation-f1_score:0.440418\n",
            "[9]\tValidation-error:0.063197\tValidation-f1_score:0.445313\n",
            "[10]\tValidation-error:0.062363\tValidation-f1_score:0.444076\n",
            "[11]\tValidation-error:0.06205\tValidation-f1_score:0.453826\n",
            "[12]\tValidation-error:0.062259\tValidation-f1_score:0.435897\n",
            "[13]\tValidation-error:0.061633\tValidation-f1_score:0.442577\n",
            "[14]\tValidation-error:0.061425\tValidation-f1_score:0.431149\n",
            "[15]\tValidation-error:0.061633\tValidation-f1_score:0.425243\n",
            "[16]\tValidation-error:0.06132\tValidation-f1_score:0.426456\n",
            "[17]\tValidation-error:0.061216\tValidation-f1_score:0.417252\n",
            "[18]\tValidation-error:0.061216\tValidation-f1_score:0.423881\n",
            "[19]\tValidation-error:0.06059\tValidation-f1_score:0.428287\n",
            "[20]\tValidation-error:0.06059\tValidation-f1_score:0.433566\n",
            "[21]\tValidation-error:0.060382\tValidation-f1_score:0.429719\n",
            "[22]\tValidation-error:0.060173\tValidation-f1_score:0.429288\n",
            "[23]\tValidation-error:0.060382\tValidation-f1_score:0.437751\n",
            "[24]\tValidation-error:0.060173\tValidation-f1_score:0.426694\n",
            "[25]\tValidation-error:0.060382\tValidation-f1_score:0.438632\n",
            "[26]\tValidation-error:0.059756\tValidation-f1_score:0.437059\n",
            "[27]\tValidation-error:0.059652\tValidation-f1_score:0.441118\n",
            "[28]\tValidation-error:0.059443\tValidation-f1_score:0.43996\n",
            "[29]\tValidation-error:0.059443\tValidation-f1_score:0.433468\n",
            "[30]\tValidation-error:0.05913\tValidation-f1_score:0.440849\n",
            "[31]\tValidation-error:0.059339\tValidation-f1_score:0.442886\n",
            "[32]\tValidation-error:0.05913\tValidation-f1_score:0.442211\n",
            "[33]\tValidation-error:0.059443\tValidation-f1_score:0.445783\n",
            "[34]\tValidation-error:0.059443\tValidation-f1_score:0.458458\n",
            "[35]\tValidation-error:0.059547\tValidation-f1_score:0.452\n",
            "[36]\tValidation-error:0.059443\tValidation-f1_score:0.452191\n",
            "[37]\tValidation-error:0.05913\tValidation-f1_score:0.451354\n",
            "[38]\tValidation-error:0.058609\tValidation-f1_score:0.456456\n",
            "[39]\tValidation-error:0.058922\tValidation-f1_score:0.454271\n",
            "[40]\tValidation-error:0.058922\tValidation-f1_score:0.448692\n",
            "[41]\tValidation-error:0.058922\tValidation-f1_score:0.44668\n",
            "[42]\tValidation-error:0.058505\tValidation-f1_score:0.452452\n",
            "[43]\tValidation-error:0.058192\tValidation-f1_score:0.46\n",
            "[44]\tValidation-error:0.0584\tValidation-f1_score:0.464606\n",
            "[45]\tValidation-error:0.058609\tValidation-f1_score:0.470821\n",
            "[46]\tValidation-error:0.058505\tValidation-f1_score:0.467456\n",
            "[47]\tValidation-error:0.058505\tValidation-f1_score:0.470936\n",
            "[48]\tValidation-error:0.058192\tValidation-f1_score:0.467327\n",
            "[49]\tValidation-error:0.058296\tValidation-f1_score:0.47384\n",
            "[50]\tValidation-error:0.0584\tValidation-f1_score:0.47657\n",
            "[51]\tValidation-error:0.0584\tValidation-f1_score:0.475248\n",
            "[52]\tValidation-error:0.057879\tValidation-f1_score:0.472222\n",
            "[53]\tValidation-error:0.057983\tValidation-f1_score:0.478175\n",
            "[54]\tValidation-error:0.058192\tValidation-f1_score:0.478649\n",
            "[55]\tValidation-error:0.057879\tValidation-f1_score:0.478649\n",
            "[56]\tValidation-error:0.057566\tValidation-f1_score:0.479208\n",
            "[57]\tValidation-error:0.057566\tValidation-f1_score:0.480237\n",
            "[58]\tValidation-error:0.057879\tValidation-f1_score:0.47929\n",
            "[59]\tValidation-error:0.057983\tValidation-f1_score:0.477701\n",
            "[60]\tValidation-error:0.057775\tValidation-f1_score:0.479683\n",
            "[61]\tValidation-error:0.057462\tValidation-f1_score:0.479371\n",
            "[62]\tValidation-error:0.057462\tValidation-f1_score:0.480864\n",
            "[63]\tValidation-error:0.05694\tValidation-f1_score:0.478818\n",
            "[64]\tValidation-error:0.057045\tValidation-f1_score:0.481809\n",
            "[65]\tValidation-error:0.056836\tValidation-f1_score:0.483301\n",
            "[66]\tValidation-error:0.057149\tValidation-f1_score:0.484314\n",
            "[67]\tValidation-error:0.057253\tValidation-f1_score:0.484252\n",
            "[68]\tValidation-error:0.057045\tValidation-f1_score:0.483235\n",
            "[69]\tValidation-error:0.056836\tValidation-f1_score:0.485265\n",
            "[70]\tValidation-error:0.057045\tValidation-f1_score:0.482759\n",
            "[71]\tValidation-error:0.057045\tValidation-f1_score:0.483712\n",
            "[72]\tValidation-error:0.05694\tValidation-f1_score:0.489194\n",
            "[73]\tValidation-error:0.05694\tValidation-f1_score:0.488189\n",
            "[74]\tValidation-error:0.056627\tValidation-f1_score:0.489152\n",
            "[75]\tValidation-error:0.056419\tValidation-f1_score:0.491642\n",
            "[76]\tValidation-error:0.056523\tValidation-f1_score:0.491194\n",
            "[77]\tValidation-error:0.056315\tValidation-f1_score:0.493634\n",
            "[78]\tValidation-error:0.05621\tValidation-f1_score:0.48867\n",
            "[79]\tValidation-error:0.056627\tValidation-f1_score:0.489194\n",
            "[80]\tValidation-error:0.056419\tValidation-f1_score:0.492611\n",
            "[81]\tValidation-error:0.056419\tValidation-f1_score:0.498528\n",
            "[82]\tValidation-error:0.056523\tValidation-f1_score:0.493609\n",
            "[83]\tValidation-error:0.056315\tValidation-f1_score:0.493097\n",
            "[84]\tValidation-error:0.055689\tValidation-f1_score:0.493583\n",
            "[85]\tValidation-error:0.055585\tValidation-f1_score:0.496063\n",
            "[86]\tValidation-error:0.055793\tValidation-f1_score:0.499508\n",
            "[87]\tValidation-error:0.056106\tValidation-f1_score:0.501475\n",
            "[88]\tValidation-error:0.055897\tValidation-f1_score:0.497053\n",
            "[89]\tValidation-error:0.055689\tValidation-f1_score:0.496078\n",
            "[90]\tValidation-error:0.055793\tValidation-f1_score:0.497065\n",
            "[91]\tValidation-error:0.055272\tValidation-f1_score:0.501953\n",
            "[92]\tValidation-error:0.055272\tValidation-f1_score:0.503428\n",
            "[93]\tValidation-error:0.055376\tValidation-f1_score:0.49951\n",
            "[94]\tValidation-error:0.055376\tValidation-f1_score:0.500978\n",
            "[95]\tValidation-error:0.055063\tValidation-f1_score:0.501463\n",
            "[96]\tValidation-error:0.055272\tValidation-f1_score:0.497551\n",
            "[97]\tValidation-error:0.055167\tValidation-f1_score:0.497065\n",
            "[98]\tValidation-error:0.055167\tValidation-f1_score:0.506354\n",
            "[99]\tValidation-error:0.054959\tValidation-f1_score:0.503428\n",
            "[100]\tValidation-error:0.054646\tValidation-f1_score:0.505387\n",
            "[101]\tValidation-error:0.054646\tValidation-f1_score:0.503428\n",
            "[102]\tValidation-error:0.054855\tValidation-f1_score:0.503922\n",
            "[103]\tValidation-error:0.055063\tValidation-f1_score:0.502935\n",
            "[104]\tValidation-error:0.054855\tValidation-f1_score:0.501961\n",
            "[105]\tValidation-error:0.055167\tValidation-f1_score:0.500982\n",
            "[106]\tValidation-error:0.055063\tValidation-f1_score:0.505882\n",
            "[107]\tValidation-error:0.054855\tValidation-f1_score:0.500493\n",
            "[108]\tValidation-error:0.055063\tValidation-f1_score:0.507375\n",
            "[109]\tValidation-error:0.055063\tValidation-f1_score:0.504433\n",
            "[110]\tValidation-error:0.055063\tValidation-f1_score:0.507375\n",
            "[111]\tValidation-error:0.054542\tValidation-f1_score:0.504931\n",
            "[112]\tValidation-error:0.05475\tValidation-f1_score:0.501481\n",
            "[113]\tValidation-error:0.05475\tValidation-f1_score:0.502947\n",
            "[114]\tValidation-error:0.054542\tValidation-f1_score:0.501475\n",
            "[115]\tValidation-error:0.05475\tValidation-f1_score:0.502463\n",
            "[116]\tValidation-error:0.054646\tValidation-f1_score:0.5\n",
            "[117]\tValidation-error:0.054542\tValidation-f1_score:0.50297\n",
            "[118]\tValidation-error:0.054333\tValidation-f1_score:0.503953\n",
            "[119]\tValidation-error:0.054437\tValidation-f1_score:0.5\n",
            "[120]\tValidation-error:0.054542\tValidation-f1_score:0.503469\n",
            "[121]\tValidation-error:0.054542\tValidation-f1_score:0.500994\n",
            "[122]\tValidation-error:0.054542\tValidation-f1_score:0.500495\n",
            "[123]\tValidation-error:0.054646\tValidation-f1_score:0.49901\n",
            "[124]\tValidation-error:0.05475\tValidation-f1_score:0.502473\n",
            "[125]\tValidation-error:0.05475\tValidation-f1_score:0.50297\n",
            "[126]\tValidation-error:0.05475\tValidation-f1_score:0.500495\n",
            "[127]\tValidation-error:0.054646\tValidation-f1_score:0.501487\n",
            "[128]\tValidation-error:0.054646\tValidation-f1_score:0.501487\n",
            "[129]\tValidation-error:0.05475\tValidation-f1_score:0.5\n",
            "[130]\tValidation-error:0.05475\tValidation-f1_score:0.499504\n",
            "[131]\tValidation-error:0.05475\tValidation-f1_score:0.49703\n",
            "[132]\tValidation-error:0.054437\tValidation-f1_score:0.506404\n",
            "[133]\tValidation-error:0.054333\tValidation-f1_score:0.505906\n",
            "[134]\tValidation-error:0.054437\tValidation-f1_score:0.504931\n",
            "[135]\tValidation-error:0.054646\tValidation-f1_score:0.503455\n",
            "[136]\tValidation-error:0.054437\tValidation-f1_score:0.503455\n",
            "[137]\tValidation-error:0.054125\tValidation-f1_score:0.5\n",
            "[138]\tValidation-error:0.054542\tValidation-f1_score:0.504451\n",
            "Stopping. Best iteration:\n",
            "[108]\tValidation-error:0.055063\tValidation-f1_score:0.507375\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "y6DqbJNWpwe-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###Step 7: Time for predictions!"
      ]
    },
    {
      "metadata": {
        "id": "71umNVuJpska",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\n",
        "### Reformatting test set for XGB ###\n",
        "dtest = xgb.DMatrix(test)\n",
        "\n",
        "### Predicting ###\n",
        "predict = xgb_model.predict(dtest) # predicting"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "LlpjlawCp88M",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "I uploaded the predictions to the practice problem page with 0.2 as probability threshold:\n",
        "\n",
        "Word Embedding\tF1- Score\t\n",
        "Glove\t0.53\t\n",
        "flair-forward -fast\t0.45\t\n",
        "flair-backward-fast\t0.48\t\n",
        "Stacked (flair-forward-fast + flair-backward-fast)\t0.54\t\n",
        " "
      ]
    }
  ]
}