Transformer_application_Named_Entity_Recognition (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transformer Network Application: Named-Entity Recognition\n",
    "\n",
    "Welcome to Week 4's first ungraded lab. In this notebook you'll explore one application of the transformer architecture that you built in the previous assignment.\n",
    "\n",
    "**After this assignment you'll be able to**:\n",
    "\n",
    "* Use tokenizers and pre-trained models from the HuggingFace Library.\n",
    "* Fine-tune a pre-trained transformer model for Named-Entity Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "- [Packages](#0)\n",
    "- [1 - Named-Entity Recogniton to Process Resumes](#1)\n",
    "    - [1.1 - Data Cleaning](#1-1)\n",
    "    - [1.2 - Padding and Generating Tags](#1-2)\n",
    "    - [1.3 - Tokenize and Align Labels with 🤗 Library](#1-3)\n",
    "        - [Exercise 1 - tokenize_and_align_labels](#ex-1)\n",
    "    - [1.4 - Optimization](#1-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='0'></a>\n",
    "## Packages\n",
    "\n",
    "Run the following cell to load the packages you'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import json\n",
    "import random\n",
    "import logging\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1'></a>\n",
    "## 1 - Named-Entity Recogniton to Process Resumes\n",
    "\n",
    "When faced with a large amount of unstructured text data, named-entity recognition (NER) can help you detect and classify important information in your dataset. For instance, in the running example \"Jane vists Africa in September\", NER would help you detect \"Jane\", \"Africa\", and \"September\" as named-entities and classify them as person, location, and time. \n",
    "\n",
    "* You will use a variation of the Transformer model you built in the last assignment to process a large dataset of resumes.\n",
    "* You will find and classify relavent information such as the companies the applicant worked at, skills, type of degree, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1-1'></a>\n",
    "### 1.1 - Dataset Cleaning\n",
    "\n",
    "In this assignment you will optimize a Transformer model on a dataset of resumes. Take a look at how the data you will be working with are structured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data = pd.read_json(\"ner.json\", lines=True)\n",
    "df_data = df_data.drop(['extras'], axis=1)\n",
    "df_data['content'] = df_data['content'].str.replace(\"\\n\", \" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data.iloc[0]['annotation']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mergeIntervals(intervals):\n",
    "    sorted_by_lower_bound = sorted(intervals, key=lambda tup: tup[0])\n",
    "    merged = []\n",
    "\n",
    "    for higher in sorted_by_lower_bound:\n",
    "        if not merged:\n",
    "            merged.append(higher)\n",
    "        else:\n",
    "            lower = merged[-1]\n",
    "            if higher[0] <= lower[1]:\n",
    "                if lower[2] is higher[2]:\n",
    "                    upper_bound = max(lower[1], higher[1])\n",
    "                    merged[-1] = (lower[0], upper_bound, lower[2])\n",
    "                else:\n",
    "                    if lower[1] > higher[1]:\n",
    "                        merged[-1] = lower\n",
    "                    else:\n",
    "                        merged[-1] = (lower[0], higher[1], higher[2])\n",
    "            else:\n",
    "                merged.append(higher)\n",
    "    return merged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_entities(df):\n",
    "    \n",
    "    entities = []\n",
    "    \n",
    "    for i in range(len(df)):\n",
    "        entity = []\n",
    "    \n",
    "        for annot in df['annotation'][i]:\n",
    "            try:\n",
    "                ent = annot['label'][0]\n",
    "                start = annot['points'][0]['start']\n",
    "                end = annot['points'][0]['end'] + 1\n",
    "                entity.append((start, end, ent))\n",
    "            except:\n",
    "                pass\n",
    "    \n",
    "        entity = mergeIntervals(entity)\n",
    "        entities.append(entity)\n",
    "    \n",
    "    return entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data['entities'] = get_entities(df_data)\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_dataturks_to_spacy(dataturks_JSON_FilePath):\n",
    "    try:\n",
    "        training_data = []\n",
    "        lines=[]\n",
    "        with open(dataturks_JSON_FilePath, 'r') as f:\n",
    "            lines = f.readlines()\n",
    "\n",
    "        for line in lines:\n",
    "            data = json.loads(line)\n",
    "            text = data['content'].replace(\"\\n\", \" \")\n",
    "            entities = []\n",
    "            data_annotations = data['annotation']\n",
    "            if data_annotations is not None:\n",
    "                for annotation in data_annotations:\n",
    "                    #only a single point in text annotation.\n",
    "                    point = annotation['points'][0]\n",
    "                    labels = annotation['label']\n",
    "                    # handle both list of labels or a single label.\n",
    "                    if not isinstance(labels, list):\n",
    "                        labels = [labels]\n",
    "\n",
    "                    for label in labels:\n",
    "                        point_start = point['start']\n",
    "                        point_end = point['end']\n",
    "                        point_text = point['text']\n",
    "                        \n",
    "                        lstrip_diff = len(point_text) - len(point_text.lstrip())\n",
    "                        rstrip_diff = len(point_text) - len(point_text.rstrip())\n",
    "                        if lstrip_diff != 0:\n",
    "                            point_start = point_start + lstrip_diff\n",
    "                        if rstrip_diff != 0:\n",
    "                            point_end = point_end - rstrip_diff\n",
    "                        entities.append((point_start, point_end + 1 , label))\n",
    "            training_data.append((text, {\"entities\" : entities}))\n",
    "        return training_data\n",
    "    except Exception as e:\n",
    "        logging.exception(\"Unable to process \" + dataturks_JSON_FilePath + \"\\n\" + \"error = \" + str(e))\n",
    "        return None\n",
    "\n",
    "def trim_entity_spans(data: list) -> list:\n",
    "    \"\"\"Removes leading and trailing white spaces from entity spans.\n",
    "\n",
    "    Args:\n",
    "        data (list): The data to be cleaned in spaCy JSON format.\n",
    "\n",
    "    Returns:\n",
    "        list: The cleaned data.\n",
    "    \"\"\"\n",
    "    invalid_span_tokens = re.compile(r'\\s')\n",
    "\n",
    "    cleaned_data = []\n",
    "    for text, annotations in data:\n",
    "        entities = annotations['entities']\n",
    "        valid_entities = []\n",
    "        for start, end, label in entities:\n",
    "            valid_start = start\n",
    "            valid_end = end\n",
    "            while valid_start < len(text) and invalid_span_tokens.match(\n",
    "                    text[valid_start]):\n",
    "                valid_start += 1\n",
    "            while valid_end > 1 and invalid_span_tokens.match(\n",
    "                    text[valid_end - 1]):\n",
    "                valid_end -= 1\n",
    "            valid_entities.append([valid_start, valid_end, label])\n",
    "        cleaned_data.append([text, {'entities': valid_entities}])\n",
    "    return cleaned_data  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = trim_entity_spans(convert_dataturks_to_spacy(\"ner.json\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm.notebook import tqdm\n",
    "def clean_dataset(data):\n",
    "    cleanedDF = pd.DataFrame(columns=[\"setences_cleaned\"])\n",
    "    sum1 = 0\n",
    "    for i in tqdm(range(len(data))):\n",
    "        start = 0\n",
    "        emptyList = [\"Empty\"] * len(data[i][0].split())\n",
    "        numberOfWords = 0\n",
    "        lenOfString = len(data[i][0])\n",
    "        strData = data[i][0]\n",
    "        strDictData = data[i][1]\n",
    "        lastIndexOfSpace = strData.rfind(' ')\n",
    "        for i in range(lenOfString):\n",
    "            if (strData[i]==\" \" and strData[i+1]!=\" \"):\n",
    "                for k,v in strDictData.items():\n",
    "                    for j in range(len(v)):\n",
    "                        entList = v[len(v)-j-1]\n",
    "                        if (start>=int(entList[0]) and i<=int(entList[1])):\n",
    "                            emptyList[numberOfWords] = entList[2]\n",
    "                            break\n",
    "                        else:\n",
    "                            continue\n",
    "                start = i + 1  \n",
    "                numberOfWords += 1\n",
    "            if (i == lastIndexOfSpace):\n",
    "                for j in range(len(v)):\n",
    "                        entList = v[len(v)-j-1]\n",
    "                        if (lastIndexOfSpace>=int(entList[0]) and lenOfString<=int(entList[1])):\n",
    "                            emptyList[numberOfWords] = entList[2]\n",
    "                            numberOfWords += 1\n",
    "        cleanedDF = cleanedDF.append(pd.Series([emptyList],  index=cleanedDF.columns ), ignore_index=True )\n",
    "        sum1 = sum1 + numberOfWords\n",
    "    return cleanedDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleanedDF = clean_dataset(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at your cleaned dataset and the categories the named-entities are matched to, or 'tags'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleanedDF.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1-2'></a>\n",
    "### 1.2 - Padding and Generating Tags\n",
    "\n",
    "Now, it is time to generate a list of unique tags you will match the named-entities to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_tags = set(cleanedDF['setences_cleaned'].explode().unique())#pd.unique(cleanedDF['setences_cleaned'])#set(tag for doc in cleanedDF['setences_cleaned'].values.tolist() for tag in doc)\n",
    "tag2id = {tag: id for id, tag in enumerate(unique_tags)}\n",
    "id2tag = {id: tag for tag, id in tag2id.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you will create an array of tags from your cleaned dataset. Oftentimes your input sequence will exceed the maximum length of a sequence your network can process. In this case, your sequence will be cut off, and you need to append zeroes onto the end of the shortened sequences using this [Keras padding API](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/pad_sequences)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = 512\n",
    "labels = cleanedDF['setences_cleaned'].values.tolist()\n",
    "\n",
    "tags = pad_sequences([[tag2id.get(l) for l in lab] for lab in labels],\n",
    "                     maxlen=MAX_LEN, value=tag2id[\"Empty\"], padding=\"post\",\n",
    "                     dtype=\"long\", truncating=\"post\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1-3'></a>\n",
    "### 1.3 - Tokenize and Align Labels with 🤗 Library\n",
    "\n",
    "Before feeding the texts to a Transformer model, you will need to tokenize your input using a [🤗 Transformer tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html). It is crucial that the tokenizer you use must match the Transformer model type you are using! In this exercise, you will use the 🤗 [DistilBERT fast tokenizer](https://huggingface.co/transformers/model_doc/distilbert.html), which standardizes the length of your sequence to 512 and pads with zeros. Notice this matches the maximu length you used when creating tags. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DistilBertTokenizerFast #, TFDistilBertModel\n",
    "tokenizer = DistilBertTokenizerFast.from_pretrained('tokenizer/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transformer models are often trained by tokenizers that split words into subwords. For instance, the word 'Africa' might get split into multiple subtokens. This can create some misalignment between the list of tags for the dataset and the list of labels generated by the tokenizer, since the tokenizer can split one word into several, or add special tokens. Before processing, it is important that you align the lists of tags and the list of labels generated by the selected tokenizer with a `tokenize_and_align_labels()` function.\n",
    "\n",
    "<a name='ex-1'></a>\n",
    "### Exercise 1 - tokenize_and_align_labels\n",
    "\n",
    "Implement `tokenize_and_align_labels()`. The function should perform the following:\n",
    "* The tokenizer cuts sequences that exceed the maximum size allowed by your model with the parameter `truncation=True`\n",
    "* Aligns the list of tags and labels with the tokenizer `word_ids` method returns a list that maps the subtokens to the original word in the sentence and special tokens to `None`. \n",
    "* Set the labels of all the special tokens (`None`) to -100 to prevent them from affecting the loss function. \n",
    "* Label of the first subtoken of a word and set the label for the following subtokens to -100. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_all_tokens = True\n",
    "def tokenize_and_align_labels(tokenizer, examples, tags):\n",
    "    tokenized_inputs = tokenizer(examples, truncation=True, is_split_into_words=False, padding='max_length', max_length=512)\n",
    "    labels = []\n",
    "    for i, label in enumerate(tags):\n",
    "        word_ids = tokenized_inputs.word_ids(batch_index=i)\n",
    "        previous_word_idx = None\n",
    "        label_ids = []\n",
    "        for word_idx in word_ids:\n",
    "            # Special tokens have a word id that is None. We set the label to -100 so they are automatically\n",
    "            # ignored in the loss function.\n",
    "            if word_idx is None:\n",
    "                label_ids.append(-100)\n",
    "            # We set the label for the first token of each word.\n",
    "            elif word_idx != previous_word_idx:\n",
    "                label_ids.append(label[word_idx])\n",
    "            # For the other tokens in a word, we set the label to either the current label or -100, depending on\n",
    "            # the label_all_tokens flag.\n",
    "            else:\n",
    "                label_ids.append(label[word_idx] if label_all_tokens else -100)\n",
    "            previous_word_idx = word_idx\n",
    "\n",
    "        labels.append(label_ids)\n",
    "\n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have tokenized inputs, you can create train and test datasets!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = tokenize_and_align_labels(tokenizer, df_data['content'].values.tolist(), tags)\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices((\n",
    "    test['input_ids'],\n",
    "    test['labels']\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test['labels'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1-4'></a>\n",
    "### 1.4 - Optimization\n",
    "\n",
    "Fantastic! Now you can finally feed your data into into a pretrained 🤗 model. You will optimize a DistilBERT model, which matches the tokenizer you used to preprocess your data. Try playing around with the different hyperparamters to improve your results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TFDistilBertForTokenClassification\n",
    "\n",
    "model = TFDistilBertForTokenClassification.from_pretrained('model/', num_labels=len(unique_tags))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)\n",
    "model.compile(optimizer=optimizer, loss=model.compute_loss, metrics=['accuracy']) # can also use any keras loss fn\n",
    "model.fit(train_dataset.shuffle(1000).batch(16),\n",
    "          epochs=3, \n",
    "          batch_size=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Congratulations!\n",
    "\n",
    "#### Here's what you should remember\n",
    "\n",
    "- Named-entity recognition (NER) detects and classifies named-entities, and can help process resumes, customer reviews, browsing histories, etc. \n",
    "- You must preprocess text data with the corresponding tokenizer to the pretrained model before feeding your input into your Transformer model"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}