Stance_Detection_for_the_Fake_News_Challenge_Questions.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Stance_Detection_for_the_Fake_News_Challenge_Questions.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"QI9jhXKPCFcJ","colab_type":"text"},"source":["# Stance Detection for the Fake News Challenge\n","\n","## Identifying Textual Relationships with Deep Neural Nets\n","\n","### Check the problem context [here](https://drive.google.com/open?id=1KfWaZyQdGBw8AUTacJ2yY86Yxgw2Xwq0).\n","\n","### Download files required for the project from [here](https://drive.google.com/open?id=10yf39ifEwVihw4xeJJR60oeFBY30Y5J8)."]},{"cell_type":"markdown","metadata":{"id":"vSNgdEMpenpE","colab_type":"text"},"source":["## Step1: Load the given dataset  \n","\n","1. Mount the google drive\n","\n","2. Import Glove embeddings\n","\n","3. Import the test and train datasets"]},{"cell_type":"markdown","metadata":{"id":"aPOZRohMiSpQ","colab_type":"text"},"source":["### Mount the google drive to access required project files\n","\n","Run the below commands"]},{"cell_type":"code","metadata":{"id":"7AS39z1XgFpT","colab_type":"code","colab":{}},"source":["from google.colab import drive"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"S_7yCFdzgFsH","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":121},"outputId":"34955ba5-79fe-44be-c96a-48b2678469e9","executionInfo":{"status":"ok","timestamp":1567337860986,"user_tz":-330,"elapsed":29599,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["drive.mount('/content/drive/')"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n","\n","Enter your authorization code:\n","··········\n","Mounted at /content/drive/\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"bhZdJ4zpwWzN","colab_type":"text"},"source":["#### Path for Project files on google drive\n","\n","**Note:** You need to change this path according where you have kept the files in google drive. "]},{"cell_type":"code","metadata":{"id":"Aol97RUogFuS","colab_type":"code","colab":{}},"source":["project_path = \"/content/drive/My Drive/SequenceModelsNLP/FakeNewChallenge/RequiredData/\""],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"2ly0VxAnwJ2f","colab_type":"text"},"source":["### Loading the Glove Embeddings"]},{"cell_type":"code","metadata":{"id":"xmsPn6PF-cgL","colab_type":"code","colab":{}},"source":["from zipfile import ZipFile\n","with ZipFile(project_path+'glove.6B.zip', 'r') as z:\n","  z.extractall()"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"TjLJEQ_PwcGi","colab_type":"text"},"source":["# Load the dataset [5 Marks]\n","\n","1. Using [read_csv()](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) in pandas load the given train datasets files **`train_bodies.csv`** and **`train_stances.csv`**\n","\n","2. Using [merge](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html) command in pandas merge the two datasets based on the Body ID. \n","\n","Note: Save the final merged dataset in a dataframe with name **`dataset`**."]},{"cell_type":"code","metadata":{"id":"7gXO1WZ-gFwm","colab_type":"code","colab":{}},"source":["import pandas as pd\n","import keras"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"kosAWskdOOT8","colab_type":"code","colab":{}},"source":["train_bodies = pd.read_csv(\"/content/drive/My Drive/SequenceModelsNLP/FakeNewChallenge/RequiredData/train_bodies.csv\")\n","train_stances = pd.read_csv(\"/content/drive/My Drive/SequenceModelsNLP/FakeNewChallenge/RequiredData/train_stances.csv\")"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"rG8LisHBrm9S","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"1fa5604f-3112-4d8f-db51-26552b14abb7","executionInfo":{"status":"ok","timestamp":1567340761555,"user_tz":-330,"elapsed":1272,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["train_bodies.shape"],"execution_count":33,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(1683, 2)"]},"metadata":{"tags":[]},"execution_count":33}]},{"cell_type":"code","metadata":{"id":"TvEcp3L7rp-X","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"98903710-3487-4ba1-aa53-7b7999a352f9","executionInfo":{"status":"ok","timestamp":1567340763144,"user_tz":-330,"elapsed":914,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["train_stances.shape"],"execution_count":34,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(49972, 3)"]},"metadata":{"tags":[]},"execution_count":34}]},{"cell_type":"code","metadata":{"id":"tCweGN5arsat","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":136},"outputId":"d9931c9d-4f92-4a10-9aeb-41613b126e8b","executionInfo":{"status":"ok","timestamp":1567340765342,"user_tz":-330,"elapsed":1222,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["train_bodies.head(3)"],"execution_count":35,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>Body ID</th>\n","      <th>articleBody</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0</td>\n","      <td>A small meteorite crashed into a wooded area i...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>4</td>\n","      <td>Last week we hinted at what was to come as Ebo...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>5</td>\n","      <td>(NEWSER) – Wonder how long a Quarter Pounder w...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   Body ID                                        articleBody\n","0        0  A small meteorite crashed into a wooded area i...\n","1        4  Last week we hinted at what was to come as Ebo...\n","2        5  (NEWSER) – Wonder how long a Quarter Pounder w..."]},"metadata":{"tags":[]},"execution_count":35}]},{"cell_type":"code","metadata":{"id":"4mvoDObtrv09","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":136},"outputId":"498aea8a-a32f-47dd-e918-a55a0a822dfd","executionInfo":{"status":"ok","timestamp":1567340770118,"user_tz":-330,"elapsed":1209,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["train_stances.head(3)"],"execution_count":36,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>Headline</th>\n","      <th>Body ID</th>\n","      <th>Stance</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Police find mass graves with at least '15 bodi...</td>\n","      <td>712</td>\n","      <td>unrelated</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Hundreds of Palestinians flee floods in Gaza a...</td>\n","      <td>158</td>\n","      <td>agree</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Christian Bale passes on role of Steve Jobs, a...</td>\n","      <td>137</td>\n","      <td>unrelated</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                            Headline  Body ID     Stance\n","0  Police find mass graves with at least '15 bodi...      712  unrelated\n","1  Hundreds of Palestinians flee floods in Gaza a...      158      agree\n","2  Christian Bale passes on role of Steve Jobs, a...      137  unrelated"]},"metadata":{"tags":[]},"execution_count":36}]},{"cell_type":"code","metadata":{"id":"GNTjpvZqr--U","colab_type":"code","colab":{}},"source":["dataset = pd.merge(train_bodies, train_stances, on=\"Body ID\")"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"g4ycQbBCg20S","colab_type":"text"},"source":["\n","<h2> Check1:</h2>\n","  \n","<h3> You should see the below output if you run `dataset.head()` command as given below </h3>"]},{"cell_type":"code","metadata":{"id":"mf3sLUW0sDpo","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"5218f1a0-f8ec-40a1-bd94-d524ec26e229","executionInfo":{"status":"ok","timestamp":1567340776482,"user_tz":-330,"elapsed":1318,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["dataset.shape"],"execution_count":38,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(49972, 4)"]},"metadata":{"tags":[]},"execution_count":38}]},{"cell_type":"code","metadata":{"id":"IUtF7iOmj11k","colab_type":"code","outputId":"79a0da9f-4b70-46e8-e742-6bab4ba03b94","colab":{"base_uri":"https://localhost:8080/","height":195},"executionInfo":{"status":"ok","timestamp":1567340779731,"user_tz":-330,"elapsed":1381,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["dataset.head()"],"execution_count":39,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>Body ID</th>\n","      <th>articleBody</th>\n","      <th>Headline</th>\n","      <th>Stance</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0</td>\n","      <td>A small meteorite crashed into a wooded area i...</td>\n","      <td>Soldier shot, Parliament locked down after gun...</td>\n","      <td>unrelated</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>0</td>\n","      <td>A small meteorite crashed into a wooded area i...</td>\n","      <td>Tourist dubbed ‘Spider Man’ after spider burro...</td>\n","      <td>unrelated</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0</td>\n","      <td>A small meteorite crashed into a wooded area i...</td>\n","      <td>Luke Somers 'killed in failed rescue attempt i...</td>\n","      <td>unrelated</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>0</td>\n","      <td>A small meteorite crashed into a wooded area i...</td>\n","      <td>BREAKING: Soldier shot at War Memorial in Ottawa</td>\n","      <td>unrelated</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0</td>\n","      <td>A small meteorite crashed into a wooded area i...</td>\n","      <td>Giant 8ft 9in catfish weighing 19 stone caught...</td>\n","      <td>unrelated</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   Body ID  ...     Stance\n","0        0  ...  unrelated\n","1        0  ...  unrelated\n","2        0  ...  unrelated\n","3        0  ...  unrelated\n","4        0  ...  unrelated\n","\n","[5 rows x 4 columns]"]},"metadata":{"tags":[]},"execution_count":39}]},{"cell_type":"markdown","metadata":{"id":"tjzVz2ifijmj","colab_type":"text"},"source":["## Step2: Data Pre-processing and setting some hyper parameters needed for model\n","\n","\n","#### Run the code given below to set the required parameters.\n","\n","1. `MAX_SENTS` = Maximum no.of sentences to consider in an article.\n","\n","2. `MAX_SENT_LENGTH` = Maximum no.of words to consider in a sentence.\n","\n","3. `MAX_NB_WORDS` = Maximum no.of words in the total vocabualry.\n","\n","4. `MAX_SENTS_HEADING` = Maximum no.of sentences to consider in a heading of an article."]},{"cell_type":"code","metadata":{"id":"KDXSdpvqjuqw","colab_type":"code","colab":{}},"source":["MAX_NB_WORDS = 20000\n","MAX_SENTS = 20\n","MAX_SENTS_HEADING = 1\n","MAX_SENT_LENGTH = 20\n","VALIDATION_SPLIT = 0.2"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"zwE7CPHdiDT-","colab_type":"text"},"source":["### Download the `Punkt` from nltk using the commands given below. This is for sentence tokenization.\n","\n","For more info on how to use it, read [this](https://stackoverflow.com/questions/35275001/use-of-punktsentencetokenizer-in-nltk).\n","\n"]},{"cell_type":"code","metadata":{"id":"lsiKmyJUZ-hU","colab_type":"code","outputId":"114ab418-9028-46c4-96f3-2bf3d61643c4","colab":{"base_uri":"https://localhost:8080/","height":67},"executionInfo":{"status":"ok","timestamp":1567340795255,"user_tz":-330,"elapsed":1262,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["import nltk\n","nltk.download('punkt')"],"execution_count":41,"outputs":[{"output_type":"stream","text":["[nltk_data] Downloading package punkt to /root/nltk_data...\n","[nltk_data]   Package punkt is already up-to-date!\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["True"]},"metadata":{"tags":[]},"execution_count":41}]},{"cell_type":"markdown","metadata":{"id":"Gqwm_GbwwnhX","colab_type":"text"},"source":["# Tokenizing the text and loading the pre-trained Glove word embeddings for each token  [5 marks] "]},{"cell_type":"markdown","metadata":{"id":"WfZLR24mm32k","colab_type":"text"},"source":["Keras provides [Tokenizer API](https://keras.io/preprocessing/text/) for preparing text. Read it before going any further."]},{"cell_type":"markdown","metadata":{"id":"fLSn9S-5oG4Z","colab_type":"text"},"source":["#### Import the Tokenizer from keras preprocessing text"]},{"cell_type":"code","metadata":{"id":"S-VUgh2yoMlR","colab_type":"code","colab":{}},"source":["from keras.preprocessing.text import Tokenizer"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"eml0Lge4oOuh","colab_type":"text"},"source":["#### Initialize the Tokenizer class with maximum vocabulary count as `MAX_NB_WORDS` initialized at the start of step2. "]},{"cell_type":"code","metadata":{"id":"Qm85qirPofc2","colab_type":"code","colab":{}},"source":["tokenizer = keras.preprocessing.text.Tokenizer(num_words=MAX_NB_WORDS, filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n', lower=True, split=' ', char_level=False, oov_token=None, document_count=0)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"HBe1KuXDosJ7","colab_type":"text"},"source":["#### Now, using fit_on_texts() from Tokenizer class, lets encode the data \n","\n","Note: We need to fit articleBody and Headline also to cover all the words."]},{"cell_type":"code","metadata":{"id":"Q5rk-UyBlmyA","colab_type":"code","colab":{}},"source":["tokenizer.fit_on_texts(dataset[\"articleBody\"])"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"tbyBNNrSyP1I","colab_type":"code","colab":{}},"source":["tokenizer.fit_on_texts(dataset[\"Headline\"])"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"-CA8frIVyVIv","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"9d851a85-0cbe-4156-b50d-bd5c52abafc5","executionInfo":{"status":"ok","timestamp":1567340824334,"user_tz":-330,"elapsed":1875,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["len(tokenizer.word_counts)"],"execution_count":46,"outputs":[{"output_type":"execute_result","data":{"text/plain":["27873"]},"metadata":{"tags":[]},"execution_count":46}]},{"cell_type":"code","metadata":{"id":"eoUftzuwyXsP","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"63c404f1-2b68-4d14-b56d-a46de0d0a3e0","executionInfo":{"status":"ok","timestamp":1567340826050,"user_tz":-330,"elapsed":941,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["tokenizer.document_count"],"execution_count":47,"outputs":[{"output_type":"execute_result","data":{"text/plain":["99944"]},"metadata":{"tags":[]},"execution_count":47}]},{"cell_type":"markdown","metadata":{"id":"omptHX-JpBsN","colab_type":"text"},"source":["#### fit_on_texts() gives the following attributes in the output as given [here](https://faroit.github.io/keras-docs/1.2.2/preprocessing/text/).\n","\n","* **word_counts:** dictionary mapping words (str) to the number of times they appeared on during fit. Only set after fit_on_texts was called.\n","\n","* **word_docs:** dictionary mapping words (str) to the number of documents/texts they appeared on during fit. Only set after fit_on_texts was called.\n","\n","* **word_index:** dictionary mapping words (str) to their rank/index (int). Only set after fit_on_texts was called.\n","\n","* **document_count:** int. Number of documents (texts/sequences) the tokenizer was trained on. Only set after fit_on_texts or fit_on_sequences was called.\n","\n"]},{"cell_type":"markdown","metadata":{"id":"SHnsT2sTtFAA","colab_type":"text"},"source":["### Now, tokenize the sentences using nltk sent_tokenize() and encode the senteces with the ids we got form the above `t.word_index`\n","\n","Initialise 2 lists with names `texts` and `articles`.\n","\n","```\n","texts = [] to store text of article as it is.\n","\n","articles = [] split the above text into a list of sentences.\n","```"]},{"cell_type":"code","metadata":{"id":"ctEu-d4c4EZs","colab_type":"code","colab":{}},"source":["texts = list(dataset[\"articleBody\"])\n"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"VDCdqCjcy3lk","colab_type":"code","colab":{}},"source":["articles = []\n","for text in texts:\n","  articles.append(nltk.sent_tokenize(text))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"koTVJjoO6P78","colab_type":"text"},"source":["## Check 2:\n","\n","first element of texts and articles should be as given below. "]},{"cell_type":"code","metadata":{"id":"3mWBW99p5UW9","colab_type":"code","outputId":"165073f4-92a3-464f-ee1e-e17a6e4d582e","colab":{"base_uri":"https://localhost:8080/","height":54},"executionInfo":{"status":"ok","timestamp":1567340923206,"user_tz":-330,"elapsed":1562,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["texts[0]"],"execution_count":50,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'A small meteorite crashed into a wooded area in Nicaragua\\'s capital of Managua overnight, the government said Sunday. Residents reported hearing a mysterious boom that left a 16-foot deep crater near the city\\'s airport, the Associated Press reports. \\n\\nGovernment spokeswoman Rosario Murillo said a committee formed by the government to study the event determined it was a \"relatively small\" meteorite that \"appears to have come off an asteroid that was passing close to Earth.\" House-sized asteroid 2014 RC, which measured 60 feet in diameter, skimmed the Earth this weekend, ABC News reports. \\nMurillo said Nicaragua will ask international experts to help local scientists in understanding what happened.\\n\\nThe crater left by the meteorite had a radius of 39 feet and a depth of 16 feet,  said Humberto Saballos, a volcanologist with the Nicaraguan Institute of Territorial Studies who was on the committee. He said it is still not clear if the meteorite disintegrated or was buried.\\n\\nHumberto Garcia, of the Astronomy Center at the National Autonomous University of Nicaragua, said the meteorite could be related to an asteroid that was forecast to pass by the planet Saturday night.\\n\\n\"We have to study it more because it could be ice or rock,\" he said.\\n\\nWilfried Strauch, an adviser to the Institute of Territorial Studies, said it was \"very strange that no one reported a streak of light. We have to ask if anyone has a photo or something.\"\\n\\nLocal residents reported hearing a loud boom Saturday night, but said they didn\\'t see anything strange in the sky.\\n\\n\"I was sitting on my porch and I saw nothing, then all of a sudden I heard a large blast. We thought it was a bomb because we felt an expansive wave,\" Jorge Santamaria told The Associated Press.\\n\\nThe site of the crater is near Managua\\'s international airport and an air force base. Only journalists from state media were allowed to visit it.'"]},"metadata":{"tags":[]},"execution_count":50}]},{"cell_type":"code","metadata":{"id":"WtIjO3ht5EKA","colab_type":"code","outputId":"701950c9-eee2-404e-ac4f-0e5a196445d2","colab":{"base_uri":"https://localhost:8080/","height":306},"executionInfo":{"status":"ok","timestamp":1567340936793,"user_tz":-330,"elapsed":1273,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["articles[0]"],"execution_count":51,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[\"A small meteorite crashed into a wooded area in Nicaragua's capital of Managua overnight, the government said Sunday.\",\n"," \"Residents reported hearing a mysterious boom that left a 16-foot deep crater near the city's airport, the Associated Press reports.\",\n"," 'Government spokeswoman Rosario Murillo said a committee formed by the government to study the event determined it was a \"relatively small\" meteorite that \"appears to have come off an asteroid that was passing close to Earth.\"',\n"," 'House-sized asteroid 2014 RC, which measured 60 feet in diameter, skimmed the Earth this weekend, ABC News reports.',\n"," 'Murillo said Nicaragua will ask international experts to help local scientists in understanding what happened.',\n"," 'The crater left by the meteorite had a radius of 39 feet and a depth of 16 feet,  said Humberto Saballos, a volcanologist with the Nicaraguan Institute of Territorial Studies who was on the committee.',\n"," 'He said it is still not clear if the meteorite disintegrated or was buried.',\n"," 'Humberto Garcia, of the Astronomy Center at the National Autonomous University of Nicaragua, said the meteorite could be related to an asteroid that was forecast to pass by the planet Saturday night.',\n"," '\"We have to study it more because it could be ice or rock,\" he said.',\n"," 'Wilfried Strauch, an adviser to the Institute of Territorial Studies, said it was \"very strange that no one reported a streak of light.',\n"," 'We have to ask if anyone has a photo or something.\"',\n"," \"Local residents reported hearing a loud boom Saturday night, but said they didn't see anything strange in the sky.\",\n"," '\"I was sitting on my porch and I saw nothing, then all of a sudden I heard a large blast.',\n"," 'We thought it was a bomb because we felt an expansive wave,\" Jorge Santamaria told The Associated Press.',\n"," \"The site of the crater is near Managua's international airport and an air force base.\",\n"," 'Only journalists from state media were allowed to visit it.']"]},"metadata":{"tags":[]},"execution_count":51}]},{"cell_type":"markdown","metadata":{"id":"fpuRIA7cCfcY","colab_type":"text"},"source":["# Now iterate through each article and each sentence to encode the words into ids using t.word_index  [5 marks] \n","\n","Here, to get words from sentence you can use `text_to_word_sequence` from keras preprocessing text.\n","\n","1. Import text_to_word_sequence\n","\n","2. Initialize a variable of shape (no.of articles, MAX_SENTS, MAX_SENT_LENGTH) with name `data` with zeros first (you can use numpy [np.zeros](https://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros.html) to initialize with all zeros)and then update it while iterating through the words and sentences in each article."]},{"cell_type":"code","metadata":{"id":"YVyClBULCqWj","colab_type":"code","colab":{}},"source":["from keras.preprocessing.text import text_to_word_sequence\n","import numpy as np"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"WEkg1ymyzZhh","colab_type":"code","colab":{}},"source":["data = np.zeros((len(texts),MAX_SENTS, MAX_SENT_LENGTH))\n","for art_idx, article in enumerate(articles):\n","  for sent_idx, sent in enumerate(article):\n","    if sent_idx > MAX_SENTS-1:\n","        break\n","    cur_words = text_to_word_sequence(sent)\n","    for word_idx, word in enumerate(cur_words):\n","      if word_idx > MAX_SENT_LENGTH-1:\n","        break\n","      data[art_idx][sent_idx][word_idx] = tokenizer.word_index[word]"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"bFdmiDYcE144","colab_type":"text"},"source":["### Check 3:\n","\n","Accessing first element in data should give something like given below."]},{"cell_type":"code","metadata":{"id":"TsFWW5C2Djog","colab_type":"code","outputId":"8edcd907-9841-4e08-be71-358e5213b68c","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"ok","timestamp":1567341050585,"user_tz":-330,"elapsed":1358,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["data[0, :, :]"],"execution_count":54,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([[3.0000e+00, 4.8100e+02, 4.2700e+02, 7.2110e+03, 8.1000e+01,\n","        3.0000e+00, 3.7330e+03, 3.3100e+02, 5.0000e+00, 3.8910e+03,\n","        3.5000e+02, 4.0000e+00, 1.4310e+03, 2.9580e+03, 1.0000e+00,\n","        8.9000e+01, 1.2000e+01, 4.6400e+02, 0.0000e+00, 0.0000e+00],\n","       [7.5800e+02, 9.5000e+01, 1.0470e+03, 3.0000e+00, 2.6790e+03,\n","        1.7520e+03, 7.0000e+00, 1.8900e+02, 3.0000e+00, 1.2170e+03,\n","        1.0750e+03, 2.0300e+03, 7.0000e+02, 1.5900e+02, 1.0000e+00,\n","        3.0320e+03, 4.4800e+02, 1.0000e+00, 5.5500e+02, 2.3500e+02],\n","       [8.9000e+01, 1.0670e+03, 4.1150e+03, 2.3490e+03, 1.2000e+01,\n","        3.0000e+00, 1.0920e+03, 3.3060e+03, 1.9000e+01, 1.0000e+00,\n","        8.9000e+01, 2.0000e+00, 1.7930e+03, 1.0000e+00, 5.2100e+02,\n","        2.0090e+03, 1.5000e+01, 9.0000e+00, 3.0000e+00, 3.1110e+03],\n","       [1.8100e+02, 3.6400e+03, 9.7200e+02, 2.0000e+02, 2.5560e+03,\n","        4.4000e+01, 6.7750e+03, 1.7220e+03, 1.2520e+03, 5.0000e+00,\n","        1.3317e+04, 1.7936e+04, 1.0000e+00, 7.7800e+02, 3.1000e+01,\n","        7.4000e+02, 3.9900e+03, 6.7000e+01, 8.5000e+01, 0.0000e+00],\n","       [2.3490e+03, 1.2000e+01, 1.5570e+03, 3.8000e+01, 1.0940e+03,\n","        3.5100e+02, 7.7500e+02, 2.0000e+00, 3.6700e+02, 2.6000e+02,\n","        1.7700e+03, 5.0000e+00, 4.4500e+03, 7.0000e+01, 4.9400e+02,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n","       [1.0000e+00, 7.0000e+02, 1.8900e+02, 1.9000e+01, 1.0000e+00,\n","        4.2700e+02, 3.2000e+01, 3.0000e+00, 7.4170e+03, 4.0000e+00,\n","        2.1590e+03, 1.2520e+03, 6.0000e+00, 3.0000e+00, 5.2700e+03,\n","        4.0000e+00, 1.2170e+03, 1.2520e+03, 1.2000e+01, 3.3630e+03],\n","       [1.3000e+01, 1.2000e+01, 1.5000e+01, 8.0000e+00, 1.4900e+02,\n","        2.5000e+01, 5.4300e+02, 6.4000e+01, 1.0000e+00, 4.2700e+02,\n","        3.7270e+03, 4.1000e+01, 9.0000e+00, 1.8500e+03, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n","       [3.3630e+03, 5.7330e+03, 4.0000e+00, 1.0000e+00, 5.8750e+03,\n","        6.1400e+02, 2.1000e+01, 1.0000e+00, 3.1100e+02, 3.4380e+03,\n","        7.9400e+02, 4.0000e+00, 1.5570e+03, 1.2000e+01, 1.0000e+00,\n","        4.2700e+02, 6.9000e+01, 2.3000e+01, 7.8700e+02, 2.0000e+00],\n","       [3.7000e+01, 1.7000e+01, 2.0000e+00, 1.7930e+03, 1.5000e+01,\n","        5.2000e+01, 1.2000e+02, 1.5000e+01, 6.9000e+01, 2.3000e+01,\n","        4.9210e+03, 4.1000e+01, 1.9630e+03, 1.3000e+01, 1.2000e+01,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n","       [4.7360e+03, 3.3380e+03, 2.4000e+01, 3.9690e+03, 2.0000e+00,\n","        1.0000e+00, 1.3160e+03, 4.0000e+00, 3.0720e+03, 1.6530e+03,\n","        1.2000e+01, 1.5000e+01, 9.0000e+00, 1.9500e+02, 1.4200e+03,\n","        7.0000e+00, 5.8000e+01, 4.0000e+01, 9.5000e+01, 3.0000e+00],\n","       [3.7000e+01, 1.7000e+01, 2.0000e+00, 1.0940e+03, 6.4000e+01,\n","        5.1000e+02, 2.0000e+01, 3.0000e+00, 2.5000e+02, 4.1000e+01,\n","        2.6400e+02, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n","       [2.6000e+02, 7.5800e+02, 9.5000e+01, 1.0470e+03, 3.0000e+00,\n","        1.8060e+03, 1.7520e+03, 5.3100e+02, 2.7600e+02, 2.9000e+01,\n","        1.2000e+01, 3.3000e+01, 7.0300e+02, 1.6300e+02, 8.9200e+02,\n","        1.4200e+03, 5.0000e+00, 1.0000e+00, 2.0810e+03, 0.0000e+00],\n","       [3.5000e+01, 9.0000e+00, 2.0570e+03, 1.0000e+01, 1.1600e+02,\n","        5.8250e+03, 6.0000e+00, 3.5000e+01, 5.7600e+02, 6.5600e+02,\n","        1.0400e+02, 5.9000e+01, 4.0000e+00, 3.0000e+00, 2.4100e+03,\n","        3.5000e+01, 2.4100e+02, 3.0000e+00, 5.1200e+02, 1.9110e+03],\n","       [3.7000e+01, 3.4100e+02, 1.5000e+01, 9.0000e+00, 3.0000e+00,\n","        2.0820e+03, 1.2000e+02, 3.7000e+01, 8.8100e+02, 2.4000e+01,\n","        4.4510e+03, 2.5840e+03, 4.3150e+03, 4.9220e+03, 5.5000e+01,\n","        1.0000e+00, 5.5500e+02, 2.3500e+02, 0.0000e+00, 0.0000e+00],\n","       [1.0000e+00, 2.5500e+02, 4.0000e+00, 1.0000e+00, 7.0000e+02,\n","        8.0000e+00, 1.5900e+02, 3.9610e+03, 3.5100e+02, 4.4800e+02,\n","        6.0000e+00, 2.4000e+01, 1.5500e+02, 4.6500e+02, 1.9290e+03,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n","       [1.2600e+02, 9.2100e+02, 2.2000e+01, 4.7000e+01, 1.0000e+02,\n","        3.6000e+01, 1.8330e+03, 2.0000e+00, 1.2120e+03, 1.5000e+01,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n","       [0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n","       [0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n","       [0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n","       [0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n","        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00]])"]},"metadata":{"tags":[]},"execution_count":54}]},{"cell_type":"markdown","metadata":{"id":"hTG6JySHehkT","colab_type":"text"},"source":["# Repeat the same process for the `Headings` as well. Use variables with names `texts_heading` and `articles_heading` accordingly. [5 marks] "]},{"cell_type":"code","metadata":{"id":"_CliiIhLemJV","colab_type":"code","colab":{}},"source":["texts_heading = list(dataset[\"Headline\"])\n","articles_heading = []\n","for text in texts:\n","  articles_heading.append(nltk.sent_tokenize(text))\n","  \n","\n","data_heading = np.zeros((len(texts),MAX_SENTS_HEADING, MAX_SENT_LENGTH))\n","for art_idx, article in enumerate(articles_heading):\n","  for sent_idx, sent in enumerate(article):\n","    if sent_idx > MAX_SENTS_HEADING-1:\n","        break\n","    cur_words = text_to_word_sequence(sent)\n","    for word_idx, word in enumerate(cur_words):\n","      if word_idx > MAX_SENT_LENGTH-1:\n","        break\n","      data_heading[art_idx][sent_idx][word_idx] = tokenizer.word_index[word]\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"iaH0Ey1qe_Co","colab_type":"text"},"source":["### Now the features are ready, lets make the labels ready for the model to process.\n","\n","### Convert labels into one-hot vectors\n","\n","You can use [get_dummies](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html) in pandas to create one-hot vectors."]},{"cell_type":"code","metadata":{"id":"Zq-VcgM8fat1","colab_type":"code","colab":{}},"source":["labels = pd.get_dummies(dataset[\"Stance\"])"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"JoKLPq980W-q","colab_type":"code","colab":{}},"source":["labels = np.array(labels)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"40mA8FI2fcxZ","colab_type":"text"},"source":["### Check 4:\n","\n","The shape of data and labels shoould match the given below numbers."]},{"cell_type":"code","metadata":{"id":"vpEWEnjFfnFR","colab_type":"code","outputId":"44f4d70e-702e-413c-bfa5-f8a7f262ac9d","colab":{"base_uri":"https://localhost:8080/","height":50},"executionInfo":{"status":"ok","timestamp":1567341291207,"user_tz":-330,"elapsed":1216,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["print('Shape of data tensor:', data.shape)\n","print('Shape of label tensor:', labels.shape)"],"execution_count":61,"outputs":[{"output_type":"stream","text":["Shape of data tensor: (49972, 20, 20)\n","Shape of label tensor: (49972, 4)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"sDOxHdR3frDu","colab_type":"text"},"source":["### Shuffle the data"]},{"cell_type":"code","metadata":{"id":"-Ra-yYTvfzRt","colab_type":"code","colab":{}},"source":["## get numbers upto no.of articles\n","indices = np.arange(data.shape[0])\n","## shuffle the numbers\n","np.random.shuffle(indices)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"LKnSqwIFf3Iy","colab_type":"code","colab":{}},"source":["## shuffle the data\n","data = data[indices]\n","data_heading = data_heading[indices]\n","## shuffle the labels according to data\n","labels = labels[indices]"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"JcOFVfPBf9kA","colab_type":"text"},"source":["### Split into train and validation sets. Split the train set 80:20 ratio to get the train and validation sets.\n","\n","\n","Use the variable names as given below:\n","\n","x_train, x_val - for body of articles.\n","\n","x-heading_train, x_heading_val - for heading of articles.\n","\n","y_train - for training labels.\n","\n","y_val - for validation labels.\n","\n"]},{"cell_type":"code","metadata":{"id":"2neh9Wcof8iR","colab_type":"code","colab":{}},"source":["from sklearn.model_selection import train_test_split"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"o5u3PTz3gEV-","colab_type":"code","colab":{}},"source":["x_train, x_val, y_train, y_val = train_test_split(data, labels, test_size=0.2)\n","x_heading_train, x_heading_val, y_train, y_val = train_test_split(data_heading, labels, test_size=0.2)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"UTyvoHrsgMDw","colab_type":"text"},"source":["### Check 5:\n","\n","The shape of x_train, x_val, y_train and y_val should match the below numbers."]},{"cell_type":"code","metadata":{"id":"KLEbiw2Yghe2","colab_type":"code","outputId":"924835c2-8c9d-4da1-d681-ff6197ec44a1","colab":{"base_uri":"https://localhost:8080/","height":84},"executionInfo":{"status":"ok","timestamp":1567341365495,"user_tz":-330,"elapsed":1486,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["print(x_train.shape)\n","print(y_train.shape)\n","\n","print(x_val.shape)\n","print(y_val.shape)"],"execution_count":66,"outputs":[{"output_type":"stream","text":["(39977, 20, 20)\n","(39977, 4)\n","(9995, 20, 20)\n","(9995, 4)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Pv-x-tCd5Mnn","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":84},"outputId":"5fd396e6-13a3-4432-8416-1294092a739c","executionInfo":{"status":"ok","timestamp":1567342526105,"user_tz":-330,"elapsed":1243,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["X_train = np.reshape(x_train, (x_train.shape[0], MAX_SENTS * MAX_SENT_LENGTH))\n","print(X_train.shape)\n","\n","X_val = np.reshape(x_val, (x_val.shape[0], MAX_SENTS * MAX_SENT_LENGTH))\n","print(X_val.shape)\n","\n","print(y_train.shape)\n","print(y_val.shape)"],"execution_count":76,"outputs":[{"output_type":"stream","text":["(39977, 400)\n","(9995, 400)\n","(39977, 4)\n","(9995, 4)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Y7k9emAr5Uig","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":84},"outputId":"be58a854-119e-4b73-95e3-fb30cb10bcc6","executionInfo":{"status":"ok","timestamp":1567342547649,"user_tz":-330,"elapsed":1249,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["X_heading_train = np.reshape(x_heading_train, (x_heading_train.shape[0], MAX_SENTS_HEADING * MAX_SENT_LENGTH))\n","print(X_heading_train.shape)\n","\n","X_heading_val = np.reshape(x_heading_val, (x_heading_val.shape[0], MAX_SENTS_HEADING * MAX_SENT_LENGTH))\n","print(X_heading_val.shape)\n","\n","print(y_train.shape)\n","print(y_val.shape)"],"execution_count":77,"outputs":[{"output_type":"stream","text":["(39977, 20)\n","(9995, 20)\n","(39977, 4)\n","(9995, 4)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"yNnoBtArhJ1E","colab_type":"text"},"source":["### Create embedding matrix with the glove embeddings\n","\n","\n","Run the below code to create embedding_matrix which has all the words and their glove embedding if present in glove word list."]},{"cell_type":"code","metadata":{"id":"eKqn2IL2ZF8v","colab_type":"code","outputId":"8360b347-3e46-43db-ec68-645114070a42","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1567342575072,"user_tz":-330,"elapsed":10516,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["# load the whole embedding into memory\n","embeddings_index = dict()\n","f = open('./glove.6B.100d.txt')\n","for line in f:\n","\tvalues = line.split()\n","\tword = values[0]\n","\tcoefs = np.asarray(values[1:], dtype='float32')\n","\tembeddings_index[word] = coefs\n","f.close()\n","print('Loaded %s word vectors.' % len(embeddings_index))\n","\n","# create a weight matrix for words in training docs\n","vocab_size = len(tokenizer.word_counts)\n","embedding_matrix = np.zeros((vocab_size, 100))\n","t = tokenizer\n","\n","for word, i in t.word_index.items():\n","\tembedding_vector = embeddings_index.get(word)\n","\tif embedding_vector is not None:\n","\t\tembedding_matrix[i] = embedding_vector"],"execution_count":78,"outputs":[{"output_type":"stream","text":["Loaded 400000 word vectors.\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"LRi4o3ZspDFU","colab_type":"text"},"source":["# Try the sequential model approach and report the accuracy score. [10 marks]  "]},{"cell_type":"markdown","metadata":{"id":"zSZDnPWkw2ZZ","colab_type":"text"},"source":["### Import layers from Keras to build the model"]},{"cell_type":"code","metadata":{"id":"5AgwQsfMrzAQ","colab_type":"code","colab":{}},"source":["from keras.models import Model, Sequential\n","from keras.layers import Dense, BatchNormalization, Input, Concatenate, concatenate,Activation, Flatten, Dropout, Embedding, LSTM, Bidirectional, GlobalMaxPool1D, merge\n","from keras.optimizers import SGD"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"gpkVhIbx3gr1","colab_type":"text"},"source":["### Model"]},{"cell_type":"code","metadata":{"id":"G_8QXh-rmPFq","colab_type":"code","colab":{}},"source":["input1 = Input(shape=(MAX_SENTS * MAX_SENT_LENGTH, ))\n","input2 = Input(shape=(MAX_SENTS_HEADING * MAX_SENT_LENGTH, ))\n","embd1 = Embedding(len(tokenizer.word_counts), 100, weights=[embedding_matrix])(input1)\n","embd2 = Embedding(len(tokenizer.word_counts), 100, weights=[embedding_matrix])(input2)# embed size is 100 given that we've used glove.6B.100d.txt\n","\n","biDirLSTM1 = Bidirectional(LSTM(100, return_sequences=True, dropout=0.25, recurrent_dropout=0.1))(embd1)\n","biDirLSTM2 = Bidirectional(LSTM(100, return_sequences=True, dropout=0.25, recurrent_dropout=0.1))(embd2)\n","\n","maxPoolLayer1 = GlobalMaxPool1D()(biDirLSTM1)\n","maxPoolLayer2 = GlobalMaxPool1D()(biDirLSTM2)\n","\n","dense1 = Dense(100, activation=\"relu\")(maxPoolLayer1)\n","dp1 = Dropout(0.25)(dense1)\n","\n","dense2 = Dense(100, activation=\"relu\")(maxPoolLayer2)\n","dp2 = Dropout(0.25)(dense2)\n","\n","x = concatenate([dp1, dp2])\n","x = Dense(4, activation=\"softmax\")(x)\n","model = Model(inputs=[input1,input2], outputs=x)\n","model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"B2LVpG1v3Xrk","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":638},"outputId":"ee5bee7c-e92d-4cb5-8faa-0d986877c2b7","executionInfo":{"status":"ok","timestamp":1567342684456,"user_tz":-330,"elapsed":1232,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["model.summary()"],"execution_count":82,"outputs":[{"output_type":"stream","text":["Model: \"model_2\"\n","__________________________________________________________________________________________________\n","Layer (type)                    Output Shape         Param #     Connected to                     \n","==================================================================================================\n","input_3 (InputLayer)            (None, 400)          0                                            \n","__________________________________________________________________________________________________\n","input_4 (InputLayer)            (None, 20)           0                                            \n","__________________________________________________________________________________________________\n","embedding_3 (Embedding)         (None, 400, 100)     2787300     input_3[0][0]                    \n","__________________________________________________________________________________________________\n","embedding_4 (Embedding)         (None, 20, 100)      2787300     input_4[0][0]                    \n","__________________________________________________________________________________________________\n","bidirectional_3 (Bidirectional) (None, 400, 200)     160800      embedding_3[0][0]                \n","__________________________________________________________________________________________________\n","bidirectional_4 (Bidirectional) (None, 20, 200)      160800      embedding_4[0][0]                \n","__________________________________________________________________________________________________\n","global_max_pooling1d_3 (GlobalM (None, 200)          0           bidirectional_3[0][0]            \n","__________________________________________________________________________________________________\n","global_max_pooling1d_4 (GlobalM (None, 200)          0           bidirectional_4[0][0]            \n","__________________________________________________________________________________________________\n","dense_4 (Dense)                 (None, 100)          20100       global_max_pooling1d_3[0][0]     \n","__________________________________________________________________________________________________\n","dense_5 (Dense)                 (None, 100)          20100       global_max_pooling1d_4[0][0]     \n","__________________________________________________________________________________________________\n","dropout_3 (Dropout)             (None, 100)          0           dense_4[0][0]                    \n","__________________________________________________________________________________________________\n","dropout_4 (Dropout)             (None, 100)          0           dense_5[0][0]                    \n","__________________________________________________________________________________________________\n","concatenate_2 (Concatenate)     (None, 200)          0           dropout_3[0][0]                  \n","                                                                 dropout_4[0][0]                  \n","__________________________________________________________________________________________________\n","dense_6 (Dense)                 (None, 4)            804         concatenate_2[0][0]              \n","==================================================================================================\n","Total params: 5,937,204\n","Trainable params: 5,937,204\n","Non-trainable params: 0\n","__________________________________________________________________________________________________\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"C5Xrd-JQ3id7","colab_type":"text"},"source":["### Compile and fit the model"]},{"cell_type":"code","metadata":{"id":"MlduHU2CovxC","colab_type":"code","colab":{}},"source":["from keras.utils import plot_model "],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"CM3yCmjQoCM3","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":218},"outputId":"ddb7e020-b59e-4e57-a978-e65c520afcf1","executionInfo":{"status":"ok","timestamp":1567347789378,"user_tz":-330,"elapsed":315275,"user":{"displayName":"Ezhirko Arulmozhi","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mA_nHqFdfM_zFBpb-BYHYJwL7O3uKBW_MNkY6jW=s64","userId":"16774972310128518178"}}},"source":["model.fit([X_train,X_heading_train], y_train, batch_size=64, epochs=5, validation_data=([X_val,X_heading_val], y_val))"],"execution_count":84,"outputs":[{"output_type":"stream","text":["Train on 39977 samples, validate on 9995 samples\n","Epoch 1/5\n","39977/39977 [==============================] - 1016s 25ms/step - loss: 0.3238 - acc: 0.8690 - val_loss: 0.2890 - val_acc: 0.8852\n","Epoch 2/5\n","39977/39977 [==============================] - 1025s 26ms/step - loss: 0.2725 - acc: 0.8914 - val_loss: 0.2561 - val_acc: 0.8997\n","Epoch 3/5\n","39977/39977 [==============================] - 1026s 26ms/step - loss: 0.2480 - acc: 0.9016 - val_loss: 0.2470 - val_acc: 0.9047\n","Epoch 4/5\n","39977/39977 [==============================] - 1025s 26ms/step - loss: 0.2349 - acc: 0.9057 - val_loss: 0.2408 - val_acc: 0.9048\n","Epoch 5/5\n","39977/39977 [==============================] - 998s 25ms/step - loss: 0.2290 - acc: 0.9076 - val_loss: 0.2389 - val_acc: 0.9068\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<keras.callbacks.History at 0x7fa73771a860>"]},"metadata":{"tags":[]},"execution_count":84}]},{"cell_type":"markdown","metadata":{"id":"R47A6Ysfev3l","colab_type":"text"},"source":["## Build the same model with attention layers included for better performance (Optional)"]},{"cell_type":"code","metadata":{"id":"ZQ3TWuiAe1Uu","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"wivJ-eVkfEOm","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"olqo5ytRe7eq","colab_type":"text"},"source":["## Fit the model and report the accuracy score for the model with attention layer (Optional)"]},{"cell_type":"code","metadata":{"id":"1zgxPrhzfBkv","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"z8507P94fDuX","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}