extractive_summarizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Summarization in Python\n",
    "\n",
    "## Motivation: \n",
    "The length of textual data is increasing and people have less time. Often the newspaper articles run into a long text of, say 1000 -1200 words. As wearable devices leap to prominence (Google Glass, Apple Watch, to name a few), content must adapt to the limited screen space available on these devices.\n",
    "The task of generating intelligent and accurate summaries for long pieces of text has become a popular research as well as industry problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach: \n",
    "Extractive text summarization is all about finding the more important sentences from a document as a summary of that document.\n",
    "Our approach is using the TextRank algorithm to find these 'important' sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 1. Importing important libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# numpy library helps in working with arrays: array creation and manipulation\n",
    "# this implementation uses array for storing the matrices generated as 2-D arrays\n",
    "# PyPDF2 is a library used for reading the PDF files\n",
    "# docx2txt is the library used for reading Word documents \n",
    "# sys library has been used for printing the size of data structures used in the program\n",
    "import numpy as np\n",
    "import PyPDF2\n",
    "import docx2txt\n",
    "import sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# matplotlib is a library that is used to visualize the data by drawing graphs of matrix inputs\n",
    "# we will use it for drawing the matrices generated later in the program \n",
    "# %matplotlib inline is a command used to show the graphs in the jupyter notebook\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# networkx library helps in working with graphs ...\n",
    "# and later performing the PageRank algorithm ...\n",
    "# which is the crux of this implementation to find ...\n",
    "# the importance of each sentence using their 'rank' as a metric ...\n",
    "# rank, the output of the method textrank, is a measure of importance of sentences\n",
    "# this library has been used in the cell no. ()\n",
    "\n",
    "import networkx as nx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# the PunktSentenceTokenizer library is being imported from the file punkt.py contained in package nltk.tokenize \n",
    "# this is used to tokenize the document into sentences\n",
    "\n",
    "# Tokenization: Tokenization is the process of demarcating and possibly classifying.. \n",
    "# sections of a string of input characters. \n",
    "# The resulting tokens are then passed on to some other form of processing. \n",
    "\n",
    "from nltk.tokenize.punkt import PunktSentenceTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TfidfTransformer and CountVectorizer libraries are being imported\n",
    "\n",
    "# CountVectorizer: In this implementation, a CountVectorizer object is being created that ..\n",
    "# will be used for creating the document-term matrix\n",
    "\n",
    "# tFidTransformer: In this implementation,TfidfTransformer is used for executing the method fit_transform()... \n",
    "# which provides the output as a document-term matrix normalized (value 0-1) according to the TF-IDF\n",
    "# TF(Term Frequency): the no. of times a term(a word here) appears in the current document(single sentence here)\n",
    "# IDF(Inverse Document Frequency): the no. of times a term(a word here) appears in the entire corpus\n",
    "# Corpus: set of all sentences\n",
    "\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfTransformer, CountVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.  Function to read the document from user\n",
    "Supported formats: .txt, .pdf \n",
    "\n",
    "Input: Takes the name of the file as input. \n",
    "\n",
    "Output: Returns a string output containing the contents of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# we are going to show an example of how the method is working\n",
    "# first let's take the document as an input\n",
    "def readDoc():\n",
    "    name = input('Please input a file name: ') \n",
    "    print('You have asked for the document {}'.format(name))\n",
    "\n",
    "    # now read the type of document\n",
    "    if name.lower().endswith('.txt'):\n",
    "        choice = 1\n",
    "    elif name.lower().endswith('.pdf'):\n",
    "        choice = 2\n",
    "    else:\n",
    "        choice = 3\n",
    "        # print(name)\n",
    "    print(choice)\n",
    "    # Case 1: if it is a .txt file\n",
    "        \n",
    "    if choice == 1:\n",
    "        f = open(name, 'r')\n",
    "        document = f.read()\n",
    "        f.close()\n",
    "            \n",
    "    # Case 2: if it is a .pdf file\n",
    "    elif choice == 2:\n",
    "        pdfFileObj = open(name, 'rb')\n",
    "        pdfReader = PyPDF2.PdfFileReader(pdfFileObj)\n",
    "        pageObj = pdfReader.getPage(0)\n",
    "        document = pageObj.extractText()\n",
    "        pdfFileObj.close()\n",
    "    \n",
    "    # Case 3: none of the format\n",
    "    else:\n",
    "        print('Failed to load a valid file')\n",
    "        print('Returning an empty string')\n",
    "        document = ''\n",
    "    \n",
    "    print(type(document))\n",
    "    return document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Function to tokenize the document\n",
    "Input: String of text document\n",
    "\n",
    "Output: A list containing sentences as its elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# the function used for tokenizing the sentences\n",
    "# tokenization of a sentence: '''provided in cell() above'''\n",
    "\n",
    "def tokenize(document):\n",
    "    # We are tokenizing using the PunktSentenceTokenizer\n",
    "    # we call an instance of this class as sentence_tokenizer\n",
    "    doc_tokenizer = PunktSentenceTokenizer()\n",
    "    \n",
    "    # tokenize() method: takes our document as input and returns a list of all the sentences in the document\n",
    "    \n",
    "    # sentences is a list containing each sentence of the document as an element\n",
    "    sentences_list = doc_tokenizer.tokenize(document)\n",
    "    return sentences_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Read the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please input a file name: story1.txt\n",
      "You have asked for the document story1.txt\n",
      "1\n",
      "<class 'str'>\n",
      "The length of the file is: 7127\n"
     ]
    }
   ],
   "source": [
    "# reading a file and \n",
    "# printing the size of the file\n",
    "document = readDoc()\n",
    "print('The length of the file is:', end=' ')\n",
    "print(len(document))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Generate a list of sentences in the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The size of the list in Bytes is: 1016\n",
      "The size of the item 0 in Bytes is: 147\n"
     ]
    }
   ],
   "source": [
    "# we want to tokenize the document for further processing\n",
    "# tokenizing the sentence means that we are creating a list of all the sentences of the document.\n",
    "# Need of tokenizing the document: Initially the document is in just a string format.\n",
    "# if we want to process the document, we need to store it in a data structure.\n",
    "# Tokenization of document into words is also possible, but we will go with the tokenizing with the sentences\n",
    "# Since we want to choose the most relevant sentences, we need to generate tokens of sentences only\n",
    "sentences_list = tokenize(document)\n",
    "\n",
    "# let us print the size of memory used by the list sentences\n",
    "print('The size of the list in Bytes is: {}'.format(sys.getsizeof(sentences_list)))\n",
    "\n",
    "# the size of one of the element of the list\n",
    "print('The size of the item 0 in Bytes is: {}'.format(sys.getsizeof(sentences_list[0])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "# let us see the data type of sentences_list\n",
    "# It will be list\n",
    "print(type(sentences_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The size of the list \"sentences\" is: 101\n"
     ]
    }
   ],
   "source": [
    "# let us analyse the elements of the sentences\n",
    "# len() method applies on the list and provides the number of elements in the list\n",
    "print('The size of the list \"sentences\" is: {}'.format(len(sentences_list)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " In Banaras District there is a village called Bira in which an old, childless\n",
      "widow used to live.\n",
      "She was a Gond woman named Bhungi and she didn't\n",
      "own either a scrap of land or a house to live in.\n",
      "Her only source of livelihood\n",
      "was a parching oven.\n",
      "The village folk customarily have one meal a day of\n",
      "parched grains, so there was always a crowd around Bhungi's oven.\n",
      "Whatever grain she was paid for parching she would grind or fry and eat it.\n",
      "She slept in a corner of the same little shack that sheltered the oven.\n",
      "As soon\n",
      "as it was light she'd get up and go out to gather dry leaves from all around to\n",
      "make her fire.\n",
      "She would stack the leaves right next to the oven, and after\n",
      "twelve, light the fire.\n",
      "But on the days when she had to parch grain for Pandit\n",
      "Udaybhan Pandey, the owner of the village, she went to bed hungry.\n",
      "She\n",
      "was obliged to work without pay for Pandit Udaybhan Pandey She also had\n",
      "to fetch water for his house.\n",
      "And, for this reason, from time to time the oven\n",
      "was not lit.\n",
      "She lived in the Pandit's village, therefore he had full authority to\n",
      "make her do any sort of odd job.\n",
      "In his opinion if she received food for\n",
      "working from him, how could it be considered as work done without pay?\n",
      "He was doing her a favour, in fact, by letting her live in the village at all.\n",
      "It was spring, a day on which the fresh grain was fried and eaten and given\n",
      "as a gift.\n",
      "No fire was lit in the houses Bhungi's oven was being put to good\n",
      "use today.\n",
      "There was a crowd worthy of a village fair around her.\n",
      "She had\n",
      "scarcely opportunity to draw a breath.\n",
      "Because of the customer's impatience,\n",
      "squabbles kept breaking out.\n",
      "Then two servants arrived, each carrying a\n",
      "heaped basket of grain from Pandit Udaybhan with the order to parch it right\n",
      "away.\n",
      "When Bhungi saw the two baskets she was alarmed.\n",
      "It was already\n",
      "after twelve and even by sunset, she would not have time to parch so much\n",
      "grain.\n",
      "Now she would have to stay at the oven parching until after dark for\n",
      "no payment.\n",
      "In despair she took the two baskets.\n",
      "One of the flunkeys said\n",
      "menacingly, 'Don't waste any time or you'll be sorry.'\n",
      "With this command the servants went away and Bhungi began to parch the\n",
      "grain.\n",
      "It's no laughing matter to parch a whole maund of grain.\n",
      "She had to\n",
      "keep stopping from the parching in order to keep the oven fire going.\n",
      "So by\n",
      "sundown not even half the work was done.\n",
      "She was afraid Panditji's men\n",
      "would be coming.\n",
      "She began to move her hands all the more frantically.\n",
      "Soon the servants returned and said, 'Well, is the grain parched?'\n",
      "Feeling bold, Bhungi said, 'Can't you see?\n",
      "I'm parching it now.'\n",
      "'The whole day's gone and you haven't finished any more grain than this!\n",
      "Have you been roasting it or spoiling it?\n",
      "This is completely uncooked!\n",
      "How's it going to be used for food?\n",
      "It's the ruin of us!\n",
      "You’ll see what\n",
      "Panditji does to you for this.'\n",
      "The result was that that night the oven was dug up and Bhungi was left\n",
      "without a means of livelihood.\n",
      "Bhungi now had no means of support.\n",
      "The villagers suffered a good deal\n",
      "too from the destruction of the oven.\n",
      "In many houses even at noon, cooked\n",
      "cereal was no longer available.\n",
      "People went to Panditji and asked him to\n",
      "give the order for the old woman's oven to be rebuilt and the fire once more\n",
      "lighted, but he paid no attention to them.\n",
      "He could not suffer a loss of face.\n",
      "A few people who wished her well urged her to move to another village.\n",
      "But\n",
      "her heart would not accept this suggestion.\n",
      "She had spent her fifty miserable\n",
      "years in this village and she loved every leaf on every tree.\n",
      "Here she had\n",
      "known the sorrows and pleasures of life; she could not give it up now in the\n",
      "last days.\n",
      "The very idea of moving distressed her.\n",
      "Sorrow in this village was\n",
      "preferable to happiness in another.\n",
      "A month went by.\n",
      "Very early one morning Pandit Udaybhan, taking his\n",
      "little band of servants with him, went out to collect his rents.\n",
      "Now when he\n",
      "looked toward the old woman's oven he fell into a violent rage: it was being\n",
      "made again.\n",
      "Bhungi was energetically rebuilding it with balls of clay Most\n",
      "likely she'd spent the night at this work and wanted to finish it before the sun\n",
      "was high.\n",
      "She knew that she was going against the Pandit's wishes, but she\n",
      "hoped that he had forgotten his anger by then.\n",
      "But alas, the poor creature had\n",
      "gown old without growing wise.\n",
      "Suddenly Panditji shouted, 'By whose order?'\n",
      "Bewildered, Bhungi saw that he was standing before her.\n",
      "He demanded once again, 'By whose order are you building it?'\n",
      "In a flight\n",
      "she said, 'Everybody said I should build it and so I'm building it.'\n",
      "'I'll have it smashed again.\n",
      "'With this he kicked the oven.\n",
      "The wet clay\n",
      "collapsed in a heap.\n",
      "He kicked at the trough again but she ran in front of it\n",
      "and took the kick in her side.\n",
      "Rubbing her ribs she said, 'Maharaj, you're not\n",
      "afraid of anybody but you ought to fear God.\n",
      "What good does it do you to\n",
      "ruin me like this!\n",
      "Do you think gold is going to grow out of this small piece\n",
      "of land!\n",
      "For your own good, I'm telling you, don't torment poor people, don't\n",
      "be the death of me.\n",
      "'You're not going to build any oven here again.\n",
      "'If I don't how am I going to be able to eat!'\n",
      "'I'm not responsible for your belly.'\n",
      "'But if I do nothing except chores for you where will I go for food!'\n",
      "'If you’re going to stay in the village you'll have to do my chores.\n",
      "'I'll do them when I've built my over?.\n",
      "I can't do your work just for the\n",
      "sake of staying in the village.\n",
      "'Then don't, just get out of the village.\n",
      "'How can I!\n",
      "I've grown old in this hut.\n",
      "My in-laws and their grandparents\n",
      "lived in this same hut.\n",
      "Except for Yama, king of death, nobody's going to\n",
      "force me out of it now.\n",
      "'Excellent, now you're quoting Scripture!'\n",
      "Pandit Udaybhan said.\n",
      "'lf you'd\n",
      "worked hard I might have let you stay, but after this I won't rest until I've\n",
      "had you thrown out.\n",
      "‘To his attendants he said, 'Go get a pile of leaves right\n",
      "away and set fire to the whole thing; we'll show her how to make an oven.\n",
      "In a moment there was a tremendous racket.\n",
      "The names leapt towards the\n",
      "sky, the blaze spread wildly in all directions till the villagers came clustering \n",
      "around this mountain of fire.\n",
      "Hopelessly, Bhungi stood by her oven\n",
      "watching the conflagration.\n",
      "Suddenly, with a violent dash, she hurled herself\n",
      "into the names.\n",
      "They came running from everywhere but no one had the\n",
      "courage to go into the mouth of the blaze.\n",
      "In a matter of seconds her\n",
      "withered body was completely consumed.\n",
      "At that moment the wind rose with a gust.\n",
      "The liberated flames began to\n",
      "race toward the east.\n",
      "There were some peasants' huts near the oven which\n",
      "were engulfed by the fierce flames.\n",
      "Fed in this way, the blaze spread even\n",
      "further.\n",
      "Panditji's barn was in its path and it pounced upon it.\n",
      "By now the\n",
      "whole village was in a panic.\n",
      "They began to band together to put out the fire\n",
      "but the sprinkle of water acted like oil on it and the flames kept mounting\n",
      "higher.\n",
      "Pandit Udaybhan's splendid mansion was swallowed up; while he\n",
      "watched, it tossed like a ship amid wild waves and disappeared in the sea of\n",
      "fire.\n",
      "The sound of lamentation that broke out amidst the ashes was even\n",
      "more pitiful than Bhungi's grievous cries.\n"
     ]
    }
   ],
   "source": [
    "# print the elements of the list\n",
    "# If the input document is long, which on realistically will be wrong, we would not like to print the entire document\n",
    "for i in sentences_list:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Generate term-document matrix (TD matrix) of the data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Convert a collection of text documents to a matrix of token counts\n",
    "# fit_transform method of CountVectorizer() class \n",
    "# Learn the vocabulary dictionary and return term-document matrix. \n",
    "# I/p: An iterable which yields either str, unicode or file objects.\n",
    "# O/p: The term-document matrix named cv_matrix\n",
    "cv = CountVectorizer()\n",
    "cv_matrix = cv.fit_transform(sentences_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**So what does CountVectorizer.fit_transform() do?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result demo array is [[0 1 1 1 1 1 0 1]\n",
      " [1 0 0 1 0 0 1 0]]\n",
      "Feature list: ['am', 'are', 'ashish', 'bad', 'good', 'is', 'not', 'you']\n"
     ]
    }
   ],
   "source": [
    "# a demo of what CountVectorizer().fit_transform(text) does\n",
    "cv_demo = CountVectorizer() # a demo object of class CountVectorizer\n",
    "\n",
    "# I have repeated the words to make a non-ambiguous array of the document text matrix \n",
    "\n",
    "text_demo = [\"Ashish is good, you are bad\", \"I am not bad\"] \n",
    "res_demo = cv_demo.fit_transform(text_demo)\n",
    "print('Result demo array is {}'.format(res_demo.toarray()))\n",
    "\n",
    "# Result is 2-d matrix containing document text matrix\n",
    "# Notice that in the second row, there is 2.\n",
    "# also, bad is repeated twice in that sentence.\n",
    "# so we can infer that 2 is corresponding to the word 'bad'\n",
    "print('Feature list: {}'.format(cv_demo.get_feature_names()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The data type of bow matrix <class 'scipy.sparse.csr.csr_matrix'>\n",
      "Shape of the matrix <bound method spmatrix.get_shape of <101x492 sparse matrix of type '<class 'numpy.int64'>'\n",
      "\twith 1223 stored elements in Compressed Sparse Row format>>\n",
      "Size of the matrix is: 56\n",
      "['able', 'accept', 'acted', 'afraid', 'after', 'again', 'against', 'alarmed', 'alas', 'all', 'already', 'also', 'always', 'am', 'amid', 'amidst', 'an', 'and', 'anger', 'another', 'any', 'anybody', 'are', 'around', 'arrived', 'as', 'ashes', 'asked', 'at', 'attendants', 'attention', 'authority', 'available', 'away', 'balls', 'banaras', 'band', 'barn', 'basket', 'baskets', 'be', 'because', 'bed', 'been', 'before', 'began', 'being', 'belly', 'bewildered', 'bhungi', 'bira', 'blaze', 'body', 'bold', 'breaking', 'breath', 'broke', 'build', 'building', 'built', 'but', 'by', 'called', 'came', 'can', 'carrying', 'cereal', 'childless', 'chores', 'clay', 'clustering', 'collapsed', 'collect', 'coming', 'command', 'completely', 'conflagration', 'considered', 'consumed', 'cooked', 'corner', 'could', 'courage', 'creature', 'cries', 'crowd', 'customarily', 'customer', 'dark', 'dash', 'day', 'days', 'deal', 'death', 'demanded', 'despair', 'destruction', 'didn', 'directions', 'disappeared', 'distressed', 'district', 'do', 'does', 'doing', 'don', 'done', 'draw', 'dry', 'dug', 'each', 'early', 'east', 'eat', 'eaten', 'either', 'energetically', 'engulfed', 'even', 'every', 'everybody', 'everywhere', 'excellent', 'except', 'face', 'fact', 'fair', 'favour', 'fear', 'fed', 'feeling', 'fell', 'fetch', 'few', 'fierce', 'fifty', 'finish', 'finished', 'fire', 'flames', 'flight', 'flunkeys', 'folk', 'food', 'for', 'force', 'forgotten', 'frantically', 'fresh', 'fried', 'from', 'front', 'fry', 'full', 'further', 'gather', 'get', 'gift', 'give', 'given', 'go', 'god', 'going', 'gold', 'gond', 'gone', 'good', 'gown', 'grain', 'grains', 'grandparents', 'grievous', 'grind', 'grow', 'growing', 'grown', 'gust', 'had', 'half', 'hands', 'happiness', 'hard', 'have', 'haven', 'he', 'heap', 'heaped', 'heart', 'her', 'here', 'herself', 'high', 'higher', 'him', 'his', 'hoped', 'hopelessly', 'house', 'houses', 'how', 'hungry', 'hurled', 'hut', 'huts', 'idea', 'if', 'impatience', 'in', 'into', 'is', 'it', 'its', 'job', 'just', 'keep', 'kept', 'kick', 'kicked', 'king', 'knew', 'known', 'lamentation', 'land', 'last', 'laughing', 'laws', 'leaf', 'leapt', 'leaves', 'left', 'let', 'letting', 'lf', 'liberated', 'life', 'light', 'lighted', 'like', 'likely', 'lit', 'little', 'live', 'lived', 'livelihood', 'll', 'longer', 'looked', 'loss', 'loved', 'made', 'maharaj', 'make', 'mansion', 'many', 'matter', 'maund', 'me', 'meal', 'means', 'men', 'menacingly', 'might', 'miserable', 'moment', 'month', 'more', 'morning', 'most', 'mountain', 'mounting', 'mouth', 'move', 'moving', 'much', 'my', 'named', 'names', 'near', 'next', 'night', 'no', 'nobody', 'noon', 'not', 'nothing', 'now', 'obliged', 'odd', 'of', 'oil', 'old', 'on', 'once', 'one', 'only', 'opinion', 'opportunity', 'or', 'order', 'ought', 'out', 'oven', 'over', 'own', 'owner', 'paid', 'pandey', 'pandit', 'panditji', 'panic', 'parch', 'parched', 'parching', 'path', 'pay', 'payment', 'peasants', 'people', 'piece', 'pile', 'pitiful', 'pleasures', 'poor', 'pounced', 'preferable', 'put', 'quoting', 'race', 'racket', 'rage', 'ran', 're', 'reason', 'rebuilding', 'rebuilt', 'received', 'rents', 'responsible', 'rest', 'result', 'returned', 'ribs', 'right', 'roasting', 'rose', 'rubbing', 'ruin', 'running', 'said', 'sake', 'same', 'saw', 'scarcely', 'scrap', 'scripture', 'sea', 'seconds', 'see', 'servants', 'set', 'shack', 'she', 'sheltered', 'ship', 'should', 'shouted', 'show', 'side', 'sky', 'slept', 'small', 'smashed', 'so', 'some', 'soon', 'sorrow', 'sorrows', 'sorry', 'sort', 'sound', 'source', 'spent', 'splendid', 'spoiling', 'spread', 'spring', 'sprinkle', 'squabbles', 'stack', 'standing', 'stay', 'staying', 'stood', 'stopping', 'suddenly', 'suffer', 'suffered', 'suggestion', 'sun', 'sundown', 'sunset', 'support', 'swallowed', 'taking', 'telling', 'than', 'that', 'the', 'their', 'them', 'then', 'there', 'therefore', 'they', 'thing', 'think', 'this', 'thrown', 'till', 'time', 'to', 'today', 'together', 'too', 'took', 'torment', 'tossed', 'toward', 'towards', 'tree', 'tremendous', 'trough', 'twelve', 'two', 'udaybhan', 'uncooked', 'until', 'up', 'upon', 'urged', 'us', 'use', 'used', 've', 'very', 'village', 'villagers', 'violent', 'wanted', 'was', 'waste', 'watched', 'watching', 'water', 'waves', 'way', 'we', 'well', 'went', 'were', 'wet', 'what', 'whatever', 'when', 'where', 'which', 'while', 'who', 'whole', 'whose', 'widow', 'wild', 'wildly', 'will', 'wind', 'wise', 'wished', 'wishes', 'with', 'withered', 'without', 'woman', 'won', 'work', 'worked', 'working', 'worthy', 'would', 'yama', 'years', 'you', 'your']\n",
      "[[0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " ...\n",
      " [0 0 1 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]\n",
      " [0 0 0 ... 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "# printing the cv_matrix type\n",
    "# and how it is being stored in memory?\n",
    "# it is stored in the compressed row format\n",
    "# compressed row format: \n",
    "print('The data type of bow matrix {}'.format(type(cv_matrix)))\n",
    "print('Shape of the matrix {}'.format(cv_matrix.get_shape))\n",
    "print('Size of the matrix is: {}'.format(sys.getsizeof(cv_matrix)))\n",
    "print(cv.get_feature_names())\n",
    "print(cv_matrix.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.         0.         0.         ... 0.         0.         0.        ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " ...\n",
      " [0.         0.         0.24450772 ... 0.         0.         0.        ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]\n",
      " [0.         0.         0.         ... 0.         0.         0.        ]]\n"
     ]
    }
   ],
   "source": [
    "# Tnormalized: document-term matrix normalized (value 0-1) according to the TF-IDF\n",
    "# TF(Term Frequency): the no. of times a term(a word here) appears in the current document(single sentence here)\n",
    "# IDF(Inverse Document Frequency): the no. of times a term(a word here) appears in the entire corpus\n",
    "# Corpus: set of all sentences\n",
    "\n",
    "normal_matrix = TfidfTransformer().fit_transform(cv_matrix)\n",
    "print(normal_matrix.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<bound method _cs_matrix.toarray of <492x101 sparse matrix of type '<class 'numpy.float64'>'\n",
      "\twith 1223 stored elements in Compressed Sparse Column format>>\n"
     ]
    }
   ],
   "source": [
    "print(normal_matrix.T.toarray)\n",
    "res_graph = normal_matrix * normal_matrix.T\n",
    "# plt.spy(res_graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ashish/anaconda3/lib/python3.6/site-packages/networkx/drawing/nx_pylab.py:126: MatplotlibDeprecationWarning: pyplot.hold is deprecated.\n",
      "    Future behavior will be consistent with the long-time default:\n",
      "    plot commands add elements without first clearing the\n",
      "    Axes and/or Figure.\n",
      "  b = plt.ishold()\n",
      "/home/ashish/anaconda3/lib/python3.6/site-packages/networkx/drawing/nx_pylab.py:522: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.\n",
      "  if not cb.is_string_like(edge_color) \\\n",
      "/home/ashish/anaconda3/lib/python3.6/site-packages/networkx/drawing/nx_pylab.py:543: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.\n",
      "  if cb.is_string_like(edge_color) or len(edge_color) == 1:\n",
      "/home/ashish/anaconda3/lib/python3.6/site-packages/networkx/drawing/nx_pylab.py:138: MatplotlibDeprecationWarning: pyplot.hold is deprecated.\n",
      "    Future behavior will be consistent with the long-time default:\n",
      "    plot commands add elements without first clearing the\n",
      "    Axes and/or Figure.\n",
      "  plt.hold(b)\n",
      "/home/ashish/anaconda3/lib/python3.6/site-packages/matplotlib/__init__.py:911: MatplotlibDeprecationWarning: axes.hold is deprecated. Please remove it from your matplotlibrc and/or style files.\n",
      "  mplDeprecation)\n",
      "/home/ashish/anaconda3/lib/python3.6/site-packages/matplotlib/rcsetup.py:156: MatplotlibDeprecationWarning: axes.hold is deprecated, will be removed in 3.0\n",
      "  mplDeprecation)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of edges 3319\n",
      "Number of vertices 101\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeEAAAFCCAYAAADGwmVOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHZ9JREFUeJzt3V+MXVd59/HvOTPO/MlFSIjdIuGk\ntOKFQGIX4XABFyTQt1VStVHS3idIqcAlft8LcuNKvAEV5Bu4aUytVGrj3L6qK1dVE1UkTXOLHcAO\ngSDegoi5CGOTPxfxzHj+nPdie+PjkzNnzsxea6+99v5+JCuxM9mzMfH8Zj3redbqDQaDAZIkqXb9\n1C8gSVJXGcKSJCViCEuSlIghLElSIoawJEmJGMKSJCViCEuSlIghLElSIoawJEmJGMKSJCViCEuS\nlIghLElSIoawJEmJGMKSJCViCEuSlIghLElSIoawJEmJGMKSJCViCEuSlIghLElSIoawJEmJGMKS\nJCViCEuSlIghLElSIoawJEmJzKZ+AUkTLC3ByZNw/jy88w7cdBMcOABf+ALs3du8j5W0I73BYDBI\n/RKSRpw5A8eOwXPPFT9fWbn2zxYWYDCA++6Do0eLX0v9sXffXfy6gS3tiCEs1W27oDpxAh5/HJaX\ni6DbSq8He/YUH7O+nu5jFxbgscfgZz8zsKUdMoSlEKYJlGlWtx/5CLz2Gqyu1v4/IboqgS21lCEs\nVTFt2fjDH4bjx7df3apQBva3vgWHD7tqVmsZwtJWQpaN/WO2O/PzRXXgpz8tfu6qWS1jCEujul42\nzo2rZmXMEFb3TPoi/c//PN3qVs3jqlkZMoTVHdutcNfWYHOz+KF2Gl01S4kZwuqGafdv1Q2Li/DE\nE8XfW7ZWQoaw2mOrMjPA178Oly8nfT010A03wJUr135u2Vo1M4SVv0ll5rk5m6e0c5atVRNDWHmz\nzKyYLFsrMkNYzWeZWalZtlYkhrCayzKzms6ytSoyhNVMlpmVE8vW2iVDWM1TBnDHy8wDoDfhn28A\n61f/fhaYSfCxm1ffcdJ7dopla+2QIax0xu313nwz/OM/FivgDlsGfgLcQRHGi0P/7DJF6D0LHLv6\na0eB+xN97E+B/w3MY2BvybK1tmAIq36T9nr7/daeWDW4+qM/4WM2gBXgK8BTwK3Aw8AB4H3A28B5\n4Bng0si/m/JjP0nYwN6uCpCtxUWDWNcxhFWvFu71TlM2XgH+DvgfTLcKfTnKm8YXKrDLKsBCLW9d\ns4UFePRRePNN945lCKtGLdzr3UnZuAzWnaxC22y734cvAt+mpavm0YqPe8edZQgrjtH93vV1eP75\n4q8tUKVsrOl1btXs3nHnGMIKa9J+b0bWKRqJ1mln2Tg3oVbN2XDvuDMMYYXTgv3e4RXuKVzd5mTS\nqnkFmCOzsvWePfD5z8PsrPvGLWYIK4yM9nsHwCrFqqnkCrc9tlo1A3wduDHRe1XmvnErGcLauYz3\ne98Frp5r5Aq3g1pRtnbfuFUMYU0v4/3e0UYqdVdrytbuG7eCIazpZLDfa5lZO9GKsvXiIrz0Ehw6\nlPpNtEuGsLaXwX6vZWaFlFXZ+nd/t2jgsnErS4awJjtzBu65p7EBbJlZsWRXtrZxK0uGsK432nR1\n7hxcuJD0lQYUYTs79GuWmVWX7MrWNm5lxRBWocFNV5eBfwBuwTKzmqXRZWsbt7JgCKvRTVfvYqlZ\nzTapbL1OEc7JytY2bjWeIdx1DW26cq9XuRlXtn4LeJSEJeteDz74QTh40FO3GsoQ7rIGNF2N3oDj\nXq/apixZN2Lv2OatxjGEu2a48eqFF+CNN5K+zhXguxQrX/d61VaN2zu2easxDOGuaGDjlfu96pJJ\ne8fJ7kS2eSs5Q7gLGtZ45X6vumx073gG+COKueMkbN5KyhBuu4SNV+73StNJvm+8fz889phNWwkY\nwm2WuPHK/V5pesn3jeevnrpu01atDOE2e+ghOH06SQna/V5p5ybtG9fGpq1aGcJttLQEx4/DN78J\nm5u1fmr3e6XqhveN/yfwO0C/7pewaasWhnCbDHdAr68XP2rifq8UxyHgv0i0X2zTVnSGcFvU3AG9\nCVwAzuF+rxRbssatXg8efBBOnar7M3eGIdwGCTqg3wU+iyteqS7JGrf6/SKIP/Upu6cjMIRzl6AD\n2qYrKY2kjVseeRmFIZyr8vjJ48dru+/XpiupGUYP/DgI7Kem5i27p4MyhHOT4PhJm66kZkvSvGX3\ndBCGcE5qbL7aBN4AnsemKykHSZq37J6uzBDORc3NVzZeSflJ0rzlkZeVGMI5qLn5ysYrKV/DzVsA\nC3V8Uo+83DVDOAc1HT9p45XUHmXz1hFs2moyQ7jJajp+8gpFANt4JbWPTVvNZgg3UY3HT64Dfwv8\nPTZeSW1l01ZzGcJNU2MH9AZwGvjLqJ9FUhPU3rTlkZdTMYSbxA5oSRHVfuLW/Dy8/rpd0xPUfjuW\ntnDmTO0B/BUMYKlLXqaofN0G/B/g/1JUxKJZX4fvfCfmZ8ieK+GmsANaUgKngAeIWKLu9+GBBxxd\n2oIh3ARLS3D77VGPoPToSUnj1NI97ejSlixHN8Hx47C2FuXRm8AvKUpPt1GUogxgSaWzFJWxd2N+\nksGg2Gp7/PGi90W/5Uo4pXIU6V//NdocsM1XkqZRW/e0o0vXcSWcyokTxVGUp09HDWCbryRN4ymK\nb9hPA8tXf0SxvFwsPgS4Ek4j8iiSzVeSqoh+5OXcXHEPuqNLhnDtIl7G4PGTkkKK2rT18Y/D0093\nvmPacnTdjh0ryjGBrQPfxOYrSeFEbdp69dViQdLxRi1XwnVaWoLbboPV1aCP9fhJSTFFbdrq+GUP\nroTrcuYMfO5zwQMYiv1f2xwkxTLctBX8OplydOns2dBPzoIhXIeyE/rVV4M/2g5oSXUoj7z8BkX/\nSVAd7pi2HB1bpE5oO6AlpbCX4gCghdAP7uhlD66EY4p0KcM6RVnosxjAkup1EXiOSBc/nDwZ46mN\nZgjHFKET+grwt9gBLSmdYxSVuKBWVuDJJ4vFS4cYwrEsLcFzzwW/FWkD+PugT5SknYk2unThQufG\nlgzhWE6ejBLAzwKXgj5VSmNmZobPfOYzqV9Du/QU14I46MG7HbvowcasGM6cKe4H/tWvgj7Wyxgk\nNc0ngaeBOymuSw2mIxc9uBIOrRxHihDAjiJJapqXgc8TYY+4I2NLroRDijCOtElxm4mjSJKa7BTw\nAIFP1OrARQ+uhEOJMI40AH6Eo0iSmi9Kx/TqKtx7b6s7pg3hUCKMI61SlHksQUtqumgd0y2/6GE2\n9Qu0QoRxpA3g37ETWlI+yordtylO1Aq2yis7pqF1Fz24Eg4hwikvXsogKUflZQ8XQj+4pRc9GMIh\nvPhicdpLIHZCS8rZy8BxIOyBvbSyY9ru6KpOnIAjR2Cj+kmqdkJLagsvepiOK+EqypGkAAEMRfnG\nTmhJbRDtooder1UXPRjCuxV4JGkZeBJL0JLaI8rY0vIyvPJK6KcmYwjvVoSRpGeCPk2S0oo2tvSD\nH4R+YjKG8G4EHknyYgZJbRXlooef/KQ1c8OG8G4E3o9wHElSm5VjS69SnARY2cZGa8aVDOHdOH8+\n2EiS40iSuiD4RQ8tGVcyhHfj3Lkgj1nHcSRJ3RG0Y3owgGefhYsXQzwtGUN4p06cKPYjAvgPDGBJ\n3RK0Y7oF40qG8E6UY0kB5oIvAy9WfyNJykrZMb0e4mEtGFcyhHci4FhSD0eSJHXTU8CPQz0s83El\nQ3haAceSHEmS8tDr9VK/QmuF6awh+3ElQ3haAfcdHEnSNPp9/3im5tH68Zwn0AUPmY8r+ad8WoHG\nktZwJEnT2dwMdrSB1DjPUGzLBZHxuJIhPK1AY0k/wY5oSXJcqWAITyPgWNIPgzxFkvLnuJIhvL3A\nY0nnq7+RFNzMzEzqV1AHOa5kCG/PsSR1wEagO7GlnQo6rvTWW6GeVBtDeBLHkiQpumDjSktLoZ5U\nG0N4EseSJCm6YONK3/9+djPDhvAkjiVJUnTBxpXW17ObGTaEJ3nnnSCPcSxJkrYWdFwps5lhQ3iS\nX/86yGMcS5KkyYKNK2U2M2wIb+XMGfhh9fh0LEmStleOK10J8bCMZoYN4a0cO1bsL1TkWJJUv36/\n7+ULGXoK+H6IB2U0Mzyb+gUaKdBokmNJUhqeu52vYEXkTGaGXQmPE6iMsY5jSVJqrojz8naoB2Uy\nM2wIjxNoNOkHOJak5uj3+9xyyy2pX6N2XkeYl6AzwxmMKhnC4wQaTcrj+zB1xebmJm+++Wbq15Am\nCjoznMGokiE8zk03BXlMsLKKNAUvYSjMz8/7e5GxoDPDGYwqGcLjHDgAs9V61hxNUt28hKGwsrLi\n70XmjhEohDMYVTKExzl4sPJ4kqNJUlqzFb+RVjpn6c6okiE8zlPVDpncxNEkpXPDDTekfoVGWA8w\n5690ujKqZAiPKmeEK+gBed3jod1o6ujLlStBzhySkgrWU3PzzaGeFIUhPCrA/sEa8IeVn6Kmc/RF\niifYqNL73hfiKdEYwqMCzAjfABwI8zaS1EnBRpV+/vMQT4nGEB4VaEa42d97SVKzXQT+E6hcb3rh\nhUaPKRnCo5wRlqRG+H8hHtLwMSVDeFSATXxnhJvNgxwKc3NzqV9BmugWApSkGz6mZAiP+u//rvwI\nZ4SbzYMcCqurq7v69xYWFgK/iTResG29Bo8pGcLDlpbgxRcrPWIAPI8zwmqv5eXl1K+gjujCmJIh\nPCzQvkH1tbQkKciY0sIC3HVXgLeJwxAeFmA8qUexjyFJquYZoHIHx8YGPPJI9ZeJxBAe5niSJuj3\n8/3jYjOacnSRYmuv0pjSrbfC3r1hXiiCfL+qxOB4kibY3Nys/XOGCn6b0ZSjvcD7qdghfemSc8LZ\nOHAA5ucrPcLxJIWUIvilpniY4kKcSmZmnBPOxiOPQMUven0cT1Jz7d27l8Fg8Nsf41baTS27W1Lv\nngPAYtWHOCeckX374P3v3/W/PgB+g+NJaq5Lly5x4cIFAL73ve+NXWk3dfVtSb17ujAn7K3Xw5aW\n4De/2fW/3gNuvfrDIFYTDQYDbrvtttSvIU3FOeGuOXkSKpbiNij2MSRJ1ZwHqg2NAnNzzglnI8Cc\n8CJeY5ibLuw19npFf+lW+73lP5ea5BlgT9WHrK05J5wN54Q7qQt7jYNBMWm51X5v+c+nNekbl6Y2\ndklN5J+WYc4JS1OZ9I1LUxu7lJ+HgbWqD9mzxxGlbASYEx4Ab4Z5G6k2/X7fkrQa5wBQ7SsysLrq\niFI2Au0b/EGQp0j12dzcpNfrtXZ/fM+eyjuLSqALI0qG8LB9++Bzn6v0iB7wRxRjSlJONjc3W7s/\nvrZWuaipBBxR6qLf//3KjxjgmJIkVXUeuFL1IV5lmJkAZQvHlCSpunMEGFEaDBxRyopjSpLUCF+i\n4jWGAPff71WGWXFMSZKS2wvcR4CQ+tKXqr9MRIbwKMeUpFo5GqVxHibAKnh2Fn74wwBvE48hPMox\nJalW407rqnLqVhnqi4uVL8FTQkGuMVxfb/SMMBjC7+WYkhJq65zuTlU5dasM9cuXL4d6HSXQhRlh\nMITHc0xJibR1TlfaqX2hHtTgGWEwhMdzTEn6LS9kUN0OAX8Y4kENnxEGQ3g8x5Sk3/JCBtXtKDAb\n4kENnxEGQ3g8x5QkKYlyNKlyd0Sv1/gZYTCExws0pnRjmLeRpM4IMpoExXjS0aMhnhSVITxOgPJF\nj+K7uU9WfpIkdUeQ0SSAT3wCDh0K8aSoDOFx9u2D++4ryhkVzFHsbagdcjpUwmYq5epgqAftC9Zf\nHZV/Urdy9GjlkvQMcD/OC7fFuEMlUpjmmwGbqZSjLwIfC/Wwho8mlQzhrdx9N/zJn1R+jPPCCq2u\nbwaqrPzLQ0dyqh4orUPAtwnUFZ3BaFLJEJ7kxuqtVc4LhzEzM8N8xcpELDFLvylP0KoS9uWhI02p\nHqj5jgLB/oRnMJpUCvJNR2sFmhcOtsfRYRsbG409TSpm6bep/5tj6fV6BncHBRtLgmxGk0quhCcJ\nNC/8cYq9DkmTTQpgS9vtFWwsCYpSdAajSSVDeJIA88JQlBu+jeNK0lb27Nmz7ce4Qm6vYGNJMzPw\nrW9lMZpU6g38L3trS0tw++2wslL5URvAaeAvKz9JXWBZtuDvQzecI1DvzJ13Nv7qwlGuhCcJNC8M\njitpZwyegr8P7Rd0LOkTnwj1pNoYwts5erTYYwjAcSVJuqarY0nDDOHt3H13sccQYFTEcSVpvF6v\nl3QcS2l0dSxpmCE8jcOH4Y47gjzKcSXpvQaDQefGsbquy2NJwwzhaR0ME5+OK0nbm52d5UMf+lDq\n11BEXR5LGmYIT8txJbVMk+du19fX+cUvfpH6NRRRl8eShjmiNC3HlaQs9ft9L7RooC6PJQ1zJTwt\nx5WkLBnAzdP1saRhhvBOOK4kSZU4lnQ9Q3gnAo8r3Vv9jZSJXq8XZQ/2Ax/4QPBnSjE5lnQ994R3\n46674Ec/qvyYdeAx4KnKT1KbeFSj2mov8EsgSD2x14MHH4RTp0I8LRlXwrsRaFyp7JR2ZEnDDGC1\nVdAtuIzHkoYZwrsRaFwJ4EYcWUptPtD/l3Vo8liRNMkh4AiBVsGZjyUNsxy9GwHHlcCRJUnt9kWK\nxcYCgVZ+mY8lDXMlvBsBx5XAkSVdz9Wu2qQM4BsJGDiZjyUNM4R3K+C4EhTdgv9JUbJRNdNcEN9k\nFqfUFuU40o0hH9qCsaRhhvBuleNKi0EOXqNHca70f2GjVlVra2upX0ESgceRSi0YSxoWZF66sw4f\nLv762GMQ4FSePtcatcDRJUn5CnpLUinj25K24kq4qsOH4Y//OOgj7ZhWyf1h5SrKiYAtGUsaZgiH\ncO+9wUaWSvMUpRx12273hw1vpRR0HKm0uNiasaRhjiiFEHhkqbQM3AZcCvpUdYGnbimV4ONIcC2A\nyy3AFnElHELgkaWSlzxotwxgpRBlHGn/fnjppVYGMBjC4QQeWYLikocjOLYkqfmijCPNz8ORI60r\nQQ8zhEMJPLJU2o9jS5KaL8o4ErRqHGkcQzikw4eDB/Hw2JJBLKmJ9lKc+uc40s7ZmBXD2bPwhS8E\nue5w2LvAZ4GXgz5VknbvEPBPwJ0Uhw4Fs7hY7AW3uBQNroTjOHQIXngB5uaCPtaxpbz1+9f+uDlC\npDb4IsV22ceJEMAtHEcaxxCOZd++opQS8IvtDPDnwEeCPVF12hw6VW2rApThrFxE6YSGVo8jjWM5\nOqYzZ+Cee+Dy5WCPHADrwL8Bx4CzwZ6sOvT7/evCWMrRIYoVcNBOaCjGkf7lXzqxAi65Eo4pQsd0\nD9gDPIBd0zkygNUGUTqhOzCONI4r4TqcOAGPPx50RVx6F/gKXvag5pmbm2NmZobLEf67Vzp3AOco\nFgNBzc/D66+3vht6lCvhOhw+XHT57d8f/NFe9qCmWl1dNYBb5BBwiiKAg1+/15FxpHEM4bocOlRc\neRj4ogewa7pLbrrpptSvoA4qu6AfoFgBB28fbOHtSNMyhOsU6eSXGYo/HE8At0b5DGqKd955J/Ur\nVDY76zXmORnugg56GEepQ+NI4xjCdYp00QMU5aG/AV6nKBl18z9n5WB9fT31K2hKUc6DHtaxcaRx\nbMyqW4SxpVEbwAo2bEmq5hRFlS3KCvjOO+Hppzu7Ai4ZwilE7JYeZue0pN3YC/w18FUiBfDcHFy4\n0MlGrFGWo1MYvugh4glJdk5L2omyA/qXFNtbUQK414M//VMD+CpDOJVybOnBB2Emyn/qgJ3TkqYz\n3AG9ANwQ6xN1uBN6HMvRTfDjH8PBgxCpYWUd+AbwHeBSlM8gKWfDHdBR2Yj1Hq6Em+BjH4M/+7No\npWk7pyVtJXoHNBRf2wzgsQzhpjh6tCjTRHIDRYnJM6clDYtyDvSw2dli2+2llwzgMSxHN0lNXdNg\n57TUZXuBh4FPAQ8RqQELin6XV16BO+6I9RmyZwg3TRnEy8sQ+f+ad4HPAi9H/SySmuIQxcr3Popr\nUcPd7zZGr1esgE+divlZsmcIN9HZs3DsGDz7bNGsFalhaxO4ABwHTmLTltLq9Xr45Siesvlqnogr\n32GLi0UJuuOHcWzHEG6yixfh+HH4xjcg4j20y1f/+hxwDDgb7TNJSqG27ueSTVhTM4Rz8NBDcPp0\n9PK0x11K7XOIohmzlgDu9YoGUwN4anZH5yBy53RphmunbNk9LeVtL/A4xVhi9K8es7PFNa12Qe+Y\nK+Fc1Ng5DTZtSbmqtfkKoN+Hr34Vvvxlj6LcBUM4JzV2Tm8CvwaeB85j45bGs5mqWWpvvrIDujJD\nODc1dU4Puwz0sHFLarLam6/ADugADOFc1dQ5PczGLamZam2+KtkBHYQhnLuaOqeHedqWlFZ54tUB\n4H3AQeCD1NRpawd0UIZw7s6cgXvuqa1hq2TjllS/2puuhi0sFN/s339/MbFhCToIQ7gNau6chmun\nbZ0H3sbmLSm22puuSv0+/MVfwN13wyOP2AEdmCHcFjV2To9j85YUT5KmK7D7uQaGcJsMd04DrKzU\n/go2b0nVDe/53g58muJe8NrZ/RydIdxGFy/CyZPw5JNw4UKSV7B5S9q5pHu+o+x+roUh3GaJmrZK\nNm9J00u25zvK7udaGcJtl6Bpq2TzljSdZHu+w+bni7/a/VwrQ7gLEjdtlWzekt4ryUEbo/bvhyNH\n7H5OwBDuiuGmrV6vCOREbN5SlyU9aGMcm6+SMoS7pmzaeuUV+O534Y03kr2KzVvqkkY1XZVsvkrO\nEO6yxI1bAKvACxSrY/eN1VaNaboq2XzVGIZw1yVs3IJiRdAb+rn7xsrVaJm5/KZyAHydxHu+JY+e\nbBxDWI1p3BrmvrFyManMvALMcf03mrXbvx8OHoSbb4a77rL5qmEMYRUa1Lg17F3ga1f/fnSFcRLL\n1kqrcWXmUTZdNZ4hrOsNN2699RacOwe/+lXSFfKAYu94fujXLFuH0ev18EvA7jRitncSm66yYAhr\nsgY0b01i2VqxjdvrfRP4KxrS4TzKpqusGMLaXuLmrWlYtlZok/Z61ynKz0n3ekfZdJUlQ1jTaWDz\n1ijL1gql8Xu9pZkZ+PSn4fd+z6arTBnCml5Dm7emYdlao7IYKZrEPd9WMIS1cw1s3pqWZWs1fqRo\nO+75toohrOoa3rw1yrJ1d2VTZh7HPd9WMoQVRgbNW9OwbJ2/7MvMwzxoo/UMYYWTQfPWtCxb5yf7\nMvMoD9roBENYYWXcvDXKsnVzbLW6PUnxDVHWZeZxbLrqDENYcYw2b21swPPPw9pa6jcLYrRsvV1I\n6L2m+T2btLotvyF6DfgosFDHS8dm01XnGMKqT0v2jYctcy0EtgqJ0VVz1wN7mmB9Dvgp8L/YfnU7\nehNXFubmYHX12s9tuuosQ1j1atG+cWm7EChXzX8HfITtw6dpgb2TdwhVNt6k+P3ILly3s7gITzxR\nrHjLKpFNV51mCKt+k/aNR1cILTK4+qM/4WPqCuxQpeDyHZjiY1tVNt4py8zagiGsdEb3jcsVAcDX\nvtaqsvVOxQpsCFsK3qA4RxlgdpuPzbJsvFP9PmxuXvu5ZWZtwxBWM7WwbB3LtIE9bVi2thQc28IC\nPPoovP22ZWZNzRBWc3W0bK0MOVKkXTKE1XyWrdVU7vWqIkNYebNsrTo4UqRIDGHlz7K1YnGkSJEZ\nwmoPy9YKxTKzamIIqxssW2scy8xKzBBWd1i2VskysxrCEFb3WLbuLsvMahhDWBpm2Tpfc3Nwxx3w\n2mvvrXRYZlZDGcLSqEll6/KL+Uc/WnyxX1lJ955d0etN/oZodHW7VaXDMrMayBCWtrLdF3NXzfGU\nwXrkCPzsZ5O/IXJ1q4wZwlIV06ya778fPvxhePJJAxt2VzZ2dauWMoSlEKYJiRwDe5pS8Oxs8de1\nNcvG0g4ZwlLdUgT2bsJyJ6VgmO59LRtL1zGEpSYLGdi7DcudrFhd3Uo7YghLbWFYStkxhCVJSqSf\n+gUkSeoqQ1iSpEQMYUmSEjGEJUlKxBCWJCkRQ1iSpEQMYUmSEjGEJUlKxBCWJCkRQ1iSpEQMYUmS\nEjGEJUlKxBCWJCkRQ1iSpEQMYUmSEjGEJUlKxBCWJCkRQ1iSpEQMYUmSEjGEJUlKxBCWJCkRQ1iS\npEQMYUmSEjGEJUlKxBCWJCkRQ1iSpEQMYUmSEjGEJUlKxBCWJCkRQ1iSpEQMYUmSEjGEJUlKxBCW\nJCkRQ1iSpEQMYUmSEjGEJUlKxBCWJCkRQ1iSpET+PycZk1NVxtp6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The memory used by the graph in Bytes is: 56\n"
     ]
    }
   ],
   "source": [
    "# drawing a graph to proceed for the textrank algorithm\n",
    "# nx_graph is a graph developed using the networkx library\n",
    "# each node represents a sentence\n",
    "# an edge represents that they have words in common\n",
    "# the edge weight is the number of words that are common in both of the sentences(nodes)\n",
    "# nx.draw() method is used to draw the graph created\n",
    "\n",
    "nx_graph = nx.from_scipy_sparse_matrix(res_graph)\n",
    "nx.draw_circular(nx_graph)\n",
    "print('Number of edges {}'.format(nx_graph.number_of_edges()))\n",
    "print('Number of vertices {}'.format(nx_graph.number_of_nodes()))\n",
    "plt.show()\n",
    "print('The memory used by the graph in Bytes is: {}'.format(sys.getsizeof(nx_graph)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#  note that the graph above is dense and therefor it resembles a circle\n",
    "# if a shorter document is taken, a beautiful circular graph can be seen "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Getting the rank of every sentence using textrank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'dict'>\n",
      "The size used by the dictionary in Bytes is: 4704\n",
      "0 0.008473050396994242\n",
      "1 0.011136728928068607\n",
      "2 0.00938375811319975\n",
      "3 0.010305311473077007\n",
      "4 0.011387878003998013\n",
      "5 0.01145380655394846\n",
      "6 0.011956616615231171\n",
      "7 0.013007582297781905\n",
      "8 0.016462878398656445\n",
      "9 0.01187037539473231\n",
      "10 0.012805637163062412\n",
      "11 0.012625721662310347\n",
      "12 0.01008180942791786\n",
      "13 0.012615817098175844\n",
      "14 0.00995100419836381\n",
      "15 0.01231286689821747\n",
      "16 0.009055631543515862\n",
      "17 0.0071143391550211425\n",
      "18 0.006747114017027335\n",
      "19 0.011078547313517732\n",
      "20 0.010166769608001894\n",
      "21 0.012780859072392963\n",
      "22 0.01264130278530918\n",
      "23 0.009348558444248862\n",
      "24 0.010022831759849008\n",
      "25 0.013720926664709495\n",
      "26 0.009675233532910613\n",
      "27 0.01390199223542955\n",
      "28 0.009475796944343168\n",
      "29 0.008121737705990758\n",
      "30 0.010128784557098501\n",
      "31 0.010061180193377263\n",
      "32 0.007628895359126889\n",
      "33 0.008648669935446916\n",
      "34 0.00960744614822794\n",
      "35 0.008321167851894653\n",
      "36 0.005865449854506731\n",
      "37 0.010850655936218324\n",
      "38 0.010169896790154655\n",
      "39 0.011316273575827011\n",
      "40 0.012806608183832113\n",
      "41 0.008721584132312046\n",
      "42 0.010471498183578102\n",
      "43 0.006931016241031088\n",
      "44 0.015471208931766295\n",
      "45 0.006769574754561522\n",
      "46 0.008470468718021342\n",
      "47 0.007920488123990668\n",
      "48 0.009307304242218498\n",
      "49 0.013422349587962502\n",
      "50 0.008003773071767153\n",
      "51 0.010972977593541678\n",
      "52 0.005638736527026146\n",
      "53 0.008382997267176074\n",
      "54 0.010529607891291115\n",
      "55 0.01388066751249088\n",
      "56 0.012126702206111424\n",
      "57 0.006784615529807728\n",
      "58 0.006089601429223818\n",
      "59 0.00858664264079694\n",
      "60 0.008907144628014937\n",
      "61 0.010300210131439289\n",
      "62 0.007498144929122574\n",
      "63 0.011431342109109783\n",
      "64 0.007033608471842126\n",
      "65 0.014620307759349067\n",
      "66 0.0109522754887766\n",
      "67 0.01054428564842724\n",
      "68 0.010429795820400788\n",
      "69 0.009541861732144701\n",
      "70 0.010525969363956916\n",
      "71 0.008928640660743127\n",
      "72 0.006048548423719144\n",
      "73 0.007978918360190731\n",
      "74 0.014936237724528288\n",
      "75 0.005785113176196985\n",
      "76 0.012830794863519965\n",
      "77 0.009853549019121234\n",
      "78 0.005429708840559133\n",
      "79 0.007610180828201691\n",
      "80 0.008306177787750089\n",
      "81 0.010575200346765663\n",
      "82 0.006302922702336775\n",
      "83 0.006533326937930439\n",
      "84 0.00877071630720654\n",
      "85 0.013144181304936985\n",
      "86 0.007410276777927928\n",
      "87 0.010919714810479552\n",
      "88 0.009071467904553756\n",
      "89 0.00732432311428914\n",
      "90 0.011678174623675309\n",
      "91 0.008559039093945649\n",
      "92 0.007144824222961519\n",
      "93 0.00875468244361112\n",
      "94 0.007575234843209657\n",
      "95 0.008323733804457837\n",
      "96 0.009463817503427972\n",
      "97 0.013055845277031975\n",
      "98 0.01255544208900953\n",
      "99 0.010079441906251044\n",
      "100 0.009695491842487964\n"
     ]
    }
   ],
   "source": [
    "# ranks is a dictionary with key=node(sentences) and value=textrank (the rank of each of the sentences)\n",
    "ranks = nx.pagerank(nx_graph)\n",
    "\n",
    "# analyse the data type of ranks\n",
    "print(type(ranks))\n",
    "print('The size used by the dictionary in Bytes is: {}'.format(sys.getsizeof(ranks)))\n",
    "\n",
    "# print the dictionary\n",
    "for i in ranks:\n",
    "    print(i, ranks[i])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Finding important sentences and generating summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# enumerate method: returns an enumerate object\n",
    "# Use of list Comprehensions\n",
    "# O/p: sentence_array is the sorted(descending order w.r.t. score value) 2-d array of ranks[sentence] and sentence \n",
    "# For example, if there are two sentences: S1 (with a score of S1 = s1) and S2 with score s2, with s2>s1\n",
    "# then sentence_array is [[s2, S2], [s1, S1]]\n",
    "sentence_array = sorted(((ranks[i], s) for i, s in enumerate(sentences_list)), reverse=True)\n",
    "sentence_array = np.asarray(sentence_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# as sentence_array is in descending order wrt score value\n",
    "# fmax is the largest score value(the score of first element)\n",
    "# fmin is the smallest score value(the score of last element)\n",
    "\n",
    "rank_max = float(sentence_array[0][0])\n",
    "rank_min = float(sentence_array[len(sentence_array) - 1][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.016462878398656445\n",
      "0.005429708840559133\n"
     ]
    }
   ],
   "source": [
    "# print the largest and smallest value of scores of the sentence\n",
    "print(rank_max)\n",
    "print(rank_min)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "101\n"
     ]
    }
   ],
   "source": [
    "# Normalization of the scores\n",
    "# so that it comes out in the range 0-1\n",
    "# fmax becomes 1\n",
    "# fmin becomes 0\n",
    "# store the normalized values in the list temp_array\n",
    "\n",
    "temp_array = []\n",
    "\n",
    "# if all sentences have equal ranks, means they are all the same\n",
    "# taking any sentence will give the summary, say the first sentence\n",
    "flag = 0\n",
    "if rank_max - rank_min == 0:\n",
    "    temp_array.append(0)\n",
    "    flag = 1\n",
    "\n",
    "# If the sentence has different ranks\n",
    "if flag != 1:\n",
    "    for i in range(0, len(sentence_array)):\n",
    "        temp_array.append((float(sentence_array[i][0]) - rank_min) / (rank_max - rank_min))\n",
    "\n",
    "print(len(temp_array))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Calculation of threshold:\n",
    "# We take the mean value of normalized scores\n",
    "# any sentence with the normalized score 0.2 more than the mean value is considered to be \n",
    "threshold = (sum(temp_array) / len(temp_array)) + 0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Separate out the sentences that satiasfy the criteria of having a score above the threshold\n",
    "sentence_list = []\n",
    "if len(temp_array) > 1:\n",
    "    for i in range(0, len(temp_array)):\n",
    "        if temp_array[i] > threshold:\n",
    "                sentence_list.append(sentence_array[i][1])\n",
    "else:\n",
    "    sentence_list.append(sentence_array[0][1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = sentence_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. Writing the summary to a new file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "But on the days when she had to parch grain for Pandit\n",
      "Udaybhan Pandey, the owner of the village, she went to bed hungry. People went to Panditji and asked him to\n",
      "give the order for the old woman's oven to be rebuilt and the fire once more\n",
      "lighted, but he paid no attention to them. 'If you’re going to stay in the village you'll have to do my chores. He kicked at the trough again but she ran in front of it\n",
      "and took the kick in her side. She had to\n",
      "keep stopping from the parching in order to keep the oven fire going. Bhungi was energetically rebuilding it with balls of clay Most\n",
      "likely she'd spent the night at this work and wanted to finish it before the sun\n",
      "was high. With this command the servants went away and Bhungi began to parch the\n",
      "grain. Here she had\n",
      "known the sorrows and pleasures of life; she could not give it up now in the\n",
      "last days. ‘To his attendants he said, 'Go get a pile of leaves right\n",
      "away and set fire to the whole thing; we'll show her how to make an oven. By now the\n",
      "whole village was in a panic. She would stack the leaves right next to the oven, and after\n",
      "twelve, light the fire. I can't do your work just for the\n",
      "sake of staying in the village. The result was that that night the oven was dug up and Bhungi was left\n",
      "without a means of livelihood. And, for this reason, from time to time the oven\n",
      "was not lit. It was already\n",
      "after twelve and even by sunset, she would not have time to parch so much\n",
      "grain. Now she would have to stay at the oven parching until after dark for\n",
      "no payment. She lived in the Pandit's village, therefore he had full authority to\n",
      "make her do any sort of odd job. He was doing her a favour, in fact, by letting her live in the village at all. They began to band together to put out the fire\n",
      "but the sprinkle of water acted like oil on it and the flames kept mounting\n",
      "higher. No fire was lit in the houses Bhungi's oven was being put to good\n",
      "use today. She knew that she was going against the Pandit's wishes, but she\n",
      "hoped that he had forgotten his anger by then.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<function TextIOWrapper.close>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print(sentence_list)\n",
    "summary = \" \".join(str(x) for x in sentence_list)\n",
    "print(summary)\n",
    "# save the data in another file, names sum.txt\n",
    "f = open('final3.txt', 'a+')\n",
    "#print(type(f))\n",
    "f.write('\\n')\n",
    "f.write(summary)\n",
    "f.close\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "But on the days when she had to parch grain for Pandit\n",
      "Udaybhan Pandey, the owner of the village, she went to bed hungry.\n",
      "People went to Panditji and asked him to\n",
      "give the order for the old woman's oven to be rebuilt and the fire once more\n",
      "lighted, but he paid no attention to them.\n",
      "'If you’re going to stay in the village you'll have to do my chores.\n",
      "He kicked at the trough again but she ran in front of it\n",
      "and took the kick in her side.\n",
      "She had to\n",
      "keep stopping from the parching in order to keep the oven fire going.\n",
      "Bhungi was energetically rebuilding it with balls of clay Most\n",
      "likely she'd spent the night at this work and wanted to finish it before the sun\n",
      "was high.\n",
      "With this command the servants went away and Bhungi began to parch the\n",
      "grain.\n",
      "Here she had\n",
      "known the sorrows and pleasures of life; she could not give it up now in the\n",
      "last days.\n",
      "‘To his attendants he said, 'Go get a pile of leaves right\n",
      "away and set fire to the whole thing; we'll show her how to make an oven.\n",
      "By now the\n",
      "whole village was in a panic.\n",
      "She would stack the leaves right next to the oven, and after\n",
      "twelve, light the fire.\n",
      "I can't do your work just for the\n",
      "sake of staying in the village.\n",
      "The result was that that night the oven was dug up and Bhungi was left\n",
      "without a means of livelihood.\n",
      "And, for this reason, from time to time the oven\n",
      "was not lit.\n",
      "It was already\n",
      "after twelve and even by sunset, she would not have time to parch so much\n",
      "grain.\n",
      "Now she would have to stay at the oven parching until after dark for\n",
      "no payment.\n",
      "She lived in the Pandit's village, therefore he had full authority to\n",
      "make her do any sort of odd job.\n",
      "He was doing her a favour, in fact, by letting her live in the village at all.\n",
      "They began to band together to put out the fire\n",
      "but the sprinkle of water acted like oil on it and the flames kept mounting\n",
      "higher.\n",
      "No fire was lit in the houses Bhungi's oven was being put to good\n",
      "use today.\n",
      "She knew that she was going against the Pandit's wishes, but she\n",
      "hoped that he had forgotten his anger by then.\n"
     ]
    }
   ],
   "source": [
    "for lines in sentence_list:\n",
    "    print(lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please feel free to contribue for any improvements."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}