MCQ_Q_A_gen_.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "rWu-o6dpDNP0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download pretrained GloVe word embeddings\n",
      "100% [......................................................................] 862182613 / 862182613"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'C:/Users/USER/Desktop/glove.6B.zip'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import wget\n",
    "\n",
    "print('download pretrained GloVe word embeddings')\n",
    "\n",
    "url = 'http://nlp.stanford.edu/data/glove.6B.zip'\n",
    "wget.download(url,'C:/Users/USER/Desktop/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract all the files into a folder named glove\n",
    "import zipfile\n",
    "with zipfile.ZipFile('C:/Users/USER/Desktop/glove.6B.zip', 'r') as zip_ref:\n",
    "    zip_ref.extractall('C:/Users/USER/Desktop/glove/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "gE7yj9k8DNIu"
   },
   "outputs": [],
   "source": [
    "#BEGINNING\n",
    "#tokenizer\n",
    "from nltk.corpus import stopwords\n",
    "#from removing stopwords\n",
    "from nltk.stem import PorterStemmer,wordnet\n",
    "#Stemmoing and lammetizer\n",
    "from nltk.tokenize import word_tokenize,sent_tokenize\n",
    "#Vectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "#BOW-bag of words\n",
    "#Tf=Term freqeuncy\n",
    "#IDF=inverse document frequency\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer,TfidfTransformer\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8NBxFcIT2Urz",
    "outputId": "030297f9-d2c5-40aa-ba80-a15889bd68f8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Our editors will review what you’ve submitted and determine whether to revise the article.Entropy,  the measure of a system’s thermal energy per unit temperature that is unavailable for doing useful work. Because work is obtained from ordered molecular motion, the amount of entropy is also a measure of the molecular disorder, or randomness, of a system. The concept of entropy provides deep insight into the direction of spontaneous change for many everyday phenomena. Its introduction by the German physicist Rudolf Clausius in 1850 is a highlight of 19th-century physics.The idea of entropy provides a mathematical way to encode the intuitive notion of which processes are impossible, even though they would not violate the fundamental law of conservation of energy. For example, a block of ice placed on a hot stove surely melts, while the stove grows cooler. Such a process is called irreversible because no slight change will cause the melted water to turn back into ice while the stove grows hotter. In contrast, a block of ice placed in an ice-water bath will either thaw a little more or freeze a little more, depending on whether a small amount of heat is added to or subtracted from the system. Such a process is reversible because only an infinitesimal amount of heat is needed to change its direction from progressive freezing to progressive thawing. Similarly, compressed gas confined in a cylinder could either expand freely into the atmosphere if a valve were opened (an irreversible process), or it could do useful work by pushing a moveable piston against the force needed to confine the gas. The latter process is reversible because only a slight increase in the restraining force could reverse the direction of the process from expansion to compression. For reversible processes the system is in equilibrium with its environment, while for irreversible processes it is not.To provide a quantitative measure for the direction of spontaneous change, Clausius introduced the concept of entropy as a precise way of expressing the second law of thermodynamics. The Clausius form of the second law states that spontaneous change for an irreversible process in an isolated system (that is, one that does not exchange heat or work with its surroundings) always proceeds in the direction of increasing entropy. For example, the block of ice and the stove constitute two parts of an isolated system for which total entropy increases as the ice melts.By the Clausius definition, if an amount of heat Q  flows into a large heat reservoir at temperature T above absolute zero, then the entropy increase is ΔS = Q/T. This equation effectively gives an alternate definition of temperature that agrees with the usual definition. Assume that there are two heat reservoirs R1 and R2 at temperatures T1 and T2 (such as the stove and the block of ice). If an amount of heat Q flows from R1 to R2, then the net entropy change for the two reservoirs is \n",
      "\n",
      " which is positive provided that T1 > T2.  Thus, the observation that heat never flows spontaneously from cold to hot is equivalent to requiring the net entropy change to be positive for a spontaneous flow of heat.  If T1 = T2, then the reservoirs are in equilibrium, no heat flows, and ΔS = 0.The condition ΔS ≥ 0 determines the maximum possible efficiency of heat engines—that is, systems such as gasoline or steam engines that can do work in a cyclic fashion. Suppose a heat engine absorbs heat Q1 from R1 and exhausts heat Q2 to R2 for each complete cycle. By conservation of energy, the work done per cycle is W = Q1 – Q2, and the net entropy change is \n",
      "\n",
      " To make W as large as possible, Q2 should be as small as possible relative to Q1.  However, Q2 cannot be zero, because this would make ΔS negative and so violate the second law. The smallest possible value of Q2 corresponds to the condition ΔS = 0, yielding\n",
      "\n",
      " as the fundamental equation limiting the efficiency of all heat engines. A process for which ΔS = 0 is reversible because an infinitesimal change would be sufficient to make the heat engine run backward as a refrigerator.The same reasoning can also determine the entropy change for the working substance in the heat engine, such as a gas in a cylinder with a movable piston. If the gas absorbs an incremental amount of heat dQ from a heat reservoir at temperature T and expands reversibly against the maximum possible restraining pressure P, then it does the maximum work dW = P dV, where dV is the change in volume. The internal energy of the gas might also change by an amount dU as it expands. Then by conservation of energy, dQ = dU + P dV.  Because the net entropy change for the system plus reservoir is zero when maximum work is done and the entropy of the reservoir decreases by an amount dSreservoir = −dQ/T, this must be counterbalanced by an entropy increase of\n",
      "\n",
      " for the working gas so that dSsystem + dSreservoir = 0.  For any real process, less than the maximum work would be done (because of friction, for example), and so the actual amount of heat dQ′ absorbed from the heat reservoir would be less than the maximum amount dQ. For example, the gas could be allowed to expand freely into a vacuum and do no work at all. Therefore, it can be stated that\n",
      "\n",
      " with dQ′ = dQ in the case of maximum work corresponding to a reversible process.This equation defines Ssystem as a thermodynamic state variable, meaning that its value is completely determined by the current state of the system and not by how the system reached that state. Entropy is an extensive property in that its magnitude depends on the amount of material in the system.In one statistical interpretation of entropy, it is found that for a very large system in thermodynamic equilibrium, entropy S is proportional to the natural logarithm of a quantity Ω representing the maximum number of microscopic ways in which the macroscopic state corresponding to S can be realized; that is, S = k ln Ω, in which k is the Boltzmann constant that is related to molecular energy.All spontaneous processes are irreversible; hence, it has been said that the entropy of the universe is increasing: that is, more and more energy becomes unavailable for conversion into work. Because of this, the universe is said to be “running down.”\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "#get the text column \n",
    "\n",
    "\n",
    "\n",
    "#create a vocabulary of words, \n",
    "#ignore words that appear in 85% of documents, \n",
    "#eliminate stop words\n",
    "\n",
    "import bs4 as bs\n",
    "import urllib.request\n",
    "import re\n",
    "test_data= urllib.request.urlopen(\"https://www.britannica.com/science/entropy-physics\")\n",
    "#test_data= urllib.request.urlopen('https://en.wikiversity.org/wiki/Fundamentals_of_chemistry')\n",
    "\n",
    "art = test_data.read()\n",
    "test_article = bs.BeautifulSoup(art,'html')\n",
    "\n",
    "\n",
    "\n",
    "para = test_article.find_all('p')\n",
    "\n",
    "test_text = \"\"\n",
    "\n",
    "for p in para:\n",
    "    test_text+=(p.text)\n",
    "print(test_text)\n",
    "#get the text column \n",
    "#docs=df['text'].tolist()\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5HnogfgYHx2V",
    "outputId": "f2bd6321-e696-4663-b76b-eca0aa406093"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\USER\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\USER\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\USER\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TfidfTransformer(norm='l2', smooth_idf=True, sublinear_tf=False, use_idf=True)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 3. stemming and lemmatizing\n",
    "import nltk\n",
    "nltk.download('wordnet')\n",
    "nltk.download('stopwords')\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import re\n",
    "nltk.download('punkt')\n",
    "from nltk.corpus import stopwords\n",
    "stop_words = stopwords.words('english')\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "\n",
    "\n",
    "ps=PorterStemmer()\n",
    "#for word in words:\n",
    "    #print('Actual:{}------ Stem:{}'.format(word,ps.stem(word)))\n",
    "\n",
    "lemmatizer=wordnet.WordNetLemmatizer()\n",
    "\n",
    "\n",
    "\n",
    "article_text = []\n",
    "\n",
    "for p in para:\n",
    "    article_text.append(p.text)\n",
    "\n",
    "import pandas as pd\n",
    "df=pd.DataFrame(article_text,columns=['text'])\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "for i in range(len(df)):\n",
    "  word_list=nltk.word_tokenize(df['text'][i])\n",
    "\n",
    "  df['text'][i] = ' '.join([lemmatizer.lemmatize(w) for w in word_list])\n",
    "  \n",
    "\n",
    "\n",
    "#get the text column \n",
    "docs=df['text'].tolist()\n",
    "\n",
    "\n",
    "#create a vocabulary of words, \n",
    "#ignore words that appear in 85% of documents, \n",
    "#eliminate stop words\n",
    "\n",
    "cv=CountVectorizer(max_df=0.85,stop_words=stop_words,max_features=10000)\n",
    "word_count_vector=cv.fit_transform(docs)\n",
    "word_count_vector.shape\n",
    "\n",
    "\n",
    "\n",
    "tfidf_transformer=TfidfTransformer(smooth_idf=True,use_idf=True)\n",
    "tfidf_transformer.fit(word_count_vector)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WsTusqIg2UHS",
    "outputId": "0f388730-9985-4cee-b450-88313f4f17dc"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\USER\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import nltk\n",
    "nltk.download('punkt') # one time execution\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "o_RBakc2i51a"
   },
   "outputs": [],
   "source": [
    "from nltk.tokenize import sent_tokenize\n",
    "sentences = []\n",
    "sentences.append(sent_tokenize(test_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "Ze3TUaAA5lp6"
   },
   "outputs": [],
   "source": [
    "sentences = [y for x in sentences for y in x]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "zIOCoQtI5yhr"
   },
   "outputs": [],
   "source": [
    "clean_sentences = pd.Series(sentences).str.replace(\"[^a-zA-Z]\", \" \")\n",
    "clean_sentences = [s.lower() for s in sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "qCUi382D52zz"
   },
   "outputs": [],
   "source": [
    "# function to remove stopwords\n",
    "def remove_stopwords(sen):\n",
    "  sen_new = \" \".join([i for i in sen if i not in stop_words])\n",
    "  return sen_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "mi5VZ_u36BsL"
   },
   "outputs": [],
   "source": [
    "# remove stopwords from the sentences\n",
    "clean_sentences = [remove_stopwords(r.split()) for r in clean_sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "za3_Wjds6CeG",
    "outputId": "d64eb873-3ba9-4a34-e83e-05dd34094ffa"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['editors review you’ve submitted determine whether revise article.entropy, measure system’s thermal energy per unit temperature unavailable useful work.',\n",
       " 'work obtained ordered molecular motion, amount entropy also measure molecular disorder, randomness, system.',\n",
       " 'concept entropy provides deep insight direction spontaneous change many everyday phenomena.',\n",
       " 'introduction german physicist rudolf clausius 1850 highlight 19th-century physics.the idea entropy provides mathematical way encode intuitive notion processes impossible, even though would violate fundamental law conservation energy.',\n",
       " 'example, block ice placed hot stove surely melts, stove grows cooler.',\n",
       " 'process called irreversible slight change cause melted water turn back ice stove grows hotter.',\n",
       " 'contrast, block ice placed ice-water bath either thaw little freeze little more, depending whether small amount heat added subtracted system.',\n",
       " 'process reversible infinitesimal amount heat needed change direction progressive freezing progressive thawing.',\n",
       " 'similarly, compressed gas confined cylinder could either expand freely atmosphere valve opened (an irreversible process), could useful work pushing moveable piston force needed confine gas.',\n",
       " 'latter process reversible slight increase restraining force could reverse direction process expansion compression.',\n",
       " 'reversible processes system equilibrium environment, irreversible processes not.to provide quantitative measure direction spontaneous change, clausius introduced concept entropy precise way expressing second law thermodynamics.',\n",
       " 'clausius form second law states spontaneous change irreversible process isolated system (that is, one exchange heat work surroundings) always proceeds direction increasing entropy.',\n",
       " 'example, block ice stove constitute two parts isolated system total entropy increases ice melts.by clausius definition, amount heat q flows large heat reservoir temperature absolute zero, entropy increase δs = q/t.',\n",
       " 'equation effectively gives alternate definition temperature agrees usual definition.',\n",
       " 'assume two heat reservoirs r1 r2 temperatures t1 t2 (such stove block ice).',\n",
       " 'amount heat q flows r1 r2, net entropy change two reservoirs positive provided t1 > t2.',\n",
       " 'thus, observation heat never flows spontaneously cold hot equivalent requiring net entropy change positive spontaneous flow heat.',\n",
       " 't1 = t2, reservoirs equilibrium, heat flows, δs = 0.the condition δs ≥ 0 determines maximum possible efficiency heat engines—that is, systems gasoline steam engines work cyclic fashion.',\n",
       " 'suppose heat engine absorbs heat q1 r1 exhausts heat q2 r2 complete cycle.',\n",
       " 'conservation energy, work done per cycle w = q1 – q2, net entropy change make w large possible, q2 small possible relative q1.',\n",
       " 'however, q2 cannot zero, would make δs negative violate second law.',\n",
       " 'smallest possible value q2 corresponds condition δs = 0, yielding fundamental equation limiting efficiency heat engines.',\n",
       " 'process δs = 0 reversible infinitesimal change would sufficient make heat engine run backward refrigerator.the reasoning also determine entropy change working substance heat engine, gas cylinder movable piston.',\n",
       " 'gas absorbs incremental amount heat dq heat reservoir temperature expands reversibly maximum possible restraining pressure p, maximum work dw = p dv, dv change volume.',\n",
       " 'internal energy gas might also change amount du expands.',\n",
       " 'conservation energy, dq = du + p dv.',\n",
       " 'net entropy change system plus reservoir zero maximum work done entropy reservoir decreases amount dsreservoir = −dq/t, must counterbalanced entropy increase working gas dssystem + dsreservoir = 0.',\n",
       " 'real process, less maximum work would done (because friction, example), actual amount heat dq′ absorbed heat reservoir would less maximum amount dq.',\n",
       " 'example, gas could allowed expand freely vacuum work all.',\n",
       " 'therefore, stated dq′ = dq case maximum work corresponding reversible process.this equation defines ssystem thermodynamic state variable, meaning value completely determined current state system system reached state.',\n",
       " 'entropy extensive property magnitude depends amount material system.in one statistical interpretation entropy, found large system thermodynamic equilibrium, entropy proportional natural logarithm quantity ω representing maximum number microscopic ways macroscopic state corresponding realized; is, = k ln ω, k boltzmann constant related molecular energy.all spontaneous processes irreversible; hence, said entropy universe increasing: is, energy becomes unavailable conversion work.',\n",
       " 'this, universe said “running down.”']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wsxBQ3WUJrqG",
    "outputId": "7a729524-d4a5-48f4-cc83-ab49f94b4f43"
   },
   "outputs": [],
   "source": [
    "# download pretrained GloVe word embeddings\n",
    "#! wget http://nlp.stanford.edu/data/glove.6B.zip\n",
    "#! unzip glove*.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "4iNx8PPS6EEO"
   },
   "outputs": [],
   "source": [
    "# Extract word vectors\n",
    "word_embeddings = {}\n",
    "f = open('C:/Users/USER/Desktop/glove/glove.6B.100d.txt', encoding='utf-8')\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    word_embeddings[word] = coefs\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "dmm8vCv87K4z"
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "sentence_vectors = []\n",
    "for i in clean_sentences:\n",
    "  if len(i) != 0:\n",
    "    v = sum([word_embeddings.get(w, np.zeros((100,))) for w in i.split()])/(len(i.split())+0.001)\n",
    "  else:\n",
    "    v = np.zeros((100,))\n",
    "  sentence_vectors.append(v)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "eW8zupYL7bYU"
   },
   "outputs": [],
   "source": [
    "# similarity matrix\n",
    "sim_mat = np.zeros([len(sentences), len(sentences)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "zIYCRsie7lJE"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "fkP6mamG7qMv"
   },
   "outputs": [],
   "source": [
    "for i in range(len(sentences)):\n",
    "  for j in range(len(sentences)):\n",
    "    if i != j:\n",
    "      sim_mat[i][j] = cosine_similarity(sentence_vectors[i].reshape(1,100), sentence_vectors[j].reshape(1,100))[0,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "6JfWvFiO7w65"
   },
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "\n",
    "nx_graph = nx.from_numpy_array(sim_mat)\n",
    "scores = nx.pagerank(nx_graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "stbiTg4P73Py"
   },
   "outputs": [],
   "source": [
    "ranked_sentences = sorted(((scores[i],s) for i,s in enumerate(sentences)), reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ii997RNFAHON",
    "outputId": "f193e107-6a0a-42d2-9551-f8bf8449c5eb"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0.03464561581998735,\n",
       "  'A process for which ΔS = 0 is reversible because an infinitesimal change would be sufficient to make the heat engine run backward as a refrigerator.The same reasoning can also determine the entropy change for the working substance in the heat engine, such as a gas in a cylinder with a movable piston.'),\n",
       " (0.03372898469929629,\n",
       "  'The Clausius form of the second law states that spontaneous change for an irreversible process in an isolated system (that is, one that does not exchange heat or work with its surroundings) always proceeds in the direction of increasing entropy.'),\n",
       " (0.03367345907884624,\n",
       "  'Because the net entropy change for the system plus reservoir is zero when maximum work is done and the entropy of the reservoir decreases by an amount dSreservoir = −dQ/T, this must be counterbalanced by an entropy increase of\\n\\n for the working gas so that dSsystem + dSreservoir = 0.'),\n",
       " (0.0336304758257186,\n",
       "  'For example, the block of ice and the stove constitute two parts of an isolated system for which total entropy increases as the ice melts.By the Clausius definition, if an amount of heat Q  flows into a large heat reservoir at temperature T above absolute zero, then the entropy increase is ΔS = Q/T.'),\n",
       " (0.03358104668698713,\n",
       "  'Thus, the observation that heat never flows spontaneously from cold to hot is equivalent to requiring the net entropy change to be positive for a spontaneous flow of heat.'),\n",
       " (0.033527834724489176,\n",
       "  'For any real process, less than the maximum work would be done (because of friction, for example), and so the actual amount of heat dQ′ absorbed from the heat reservoir would be less than the maximum amount dQ.'),\n",
       " (0.03347335593150849,\n",
       "  'Therefore, it can be stated that\\n\\n with dQ′ = dQ in the case of maximum work corresponding to a reversible process.This equation defines Ssystem as a thermodynamic state variable, meaning that its value is completely determined by the current state of the system and not by how the system reached that state.'),\n",
       " (0.03327923773501304,\n",
       "  'If an amount of heat Q flows from R1 to R2, then the net entropy change for the two reservoirs is \\n\\n which is positive provided that T1 > T2.'),\n",
       " (0.03296475604175705,\n",
       "  'If the gas absorbs an incremental amount of heat dQ from a heat reservoir at temperature T and expands reversibly against the maximum possible restraining pressure P, then it does the maximum work dW = P dV, where dV is the change in volume.'),\n",
       " (0.03284514692821154,\n",
       "  'Entropy is an extensive property in that its magnitude depends on the amount of material in the system.In one statistical interpretation of entropy, it is found that for a very large system in thermodynamic equilibrium, entropy S is proportional to the natural logarithm of a quantity Ω representing the maximum number of microscopic ways in which the macroscopic state corresponding to S can be realized; that is, S = k ln Ω, in which k is the Boltzmann constant that is related to molecular energy.All spontaneous processes are irreversible; hence, it has been said that the entropy of the universe is increasing: that is, more and more energy becomes unavailable for conversion into work.'),\n",
       " (0.032796570588992546,\n",
       "  'By conservation of energy, the work done per cycle is W = Q1 – Q2, and the net entropy change is \\n\\n To make W as large as possible, Q2 should be as small as possible relative to Q1.'),\n",
       " (0.03277907804900803,\n",
       "  'The internal energy of the gas might also change by an amount dU as it expands.'),\n",
       " (0.032540371890841496,\n",
       "  'In contrast, a block of ice placed in an ice-water bath will either thaw a little more or freeze a little more, depending on whether a small amount of heat is added to or subtracted from the system.'),\n",
       " (0.03243832530498372,\n",
       "  'Such a process is reversible because only an infinitesimal amount of heat is needed to change its direction from progressive freezing to progressive thawing.'),\n",
       " (0.03231654599124843,\n",
       "  'If T1 = T2, then the reservoirs are in equilibrium, no heat flows, and ΔS = 0.The condition ΔS ≥ 0 determines the maximum possible efficiency of heat engines—that is, systems such as gasoline or steam engines that can do work in a cyclic fashion.'),\n",
       " (0.032313352975726975,\n",
       "  'The smallest possible value of Q2 corresponds to the condition ΔS = 0, yielding\\n\\n as the fundamental equation limiting the efficiency of all heat engines.'),\n",
       " (0.03227201768666916,\n",
       "  'Similarly, compressed gas confined in a cylinder could either expand freely into the atmosphere if a valve were opened (an irreversible process), or it could do useful work by pushing a moveable piston against the force needed to confine the gas.'),\n",
       " (0.0321787485999555,\n",
       "  'Our editors will review what you’ve submitted and determine whether to revise the article.Entropy,  the measure of a system’s thermal energy per unit temperature that is unavailable for doing useful work.'),\n",
       " (0.03193713935398228,\n",
       "  'The latter process is reversible because only a slight increase in the restraining force could reverse the direction of the process from expansion to compression.'),\n",
       " (0.0317645098885073,\n",
       "  'Such a process is called irreversible because no slight change will cause the melted water to turn back into ice while the stove grows hotter.'),\n",
       " (0.03165157389369818,\n",
       "  'For reversible processes the system is in equilibrium with its environment, while for irreversible processes it is not.To provide a quantitative measure for the direction of spontaneous change, Clausius introduced the concept of entropy as a precise way of expressing the second law of thermodynamics.'),\n",
       " (0.03133011206445018,\n",
       "  'The concept of entropy provides deep insight into the direction of spontaneous change for many everyday phenomena.'),\n",
       " (0.0310994015303884,\n",
       "  'Because work is obtained from ordered molecular motion, the amount of entropy is also a measure of the molecular disorder, or randomness, of a system.'),\n",
       " (0.031041274441123496,\n",
       "  'This equation effectively gives an alternate definition of temperature that agrees with the usual definition.'),\n",
       " (0.03100038909523005,\n",
       "  'For example, the gas could be allowed to expand freely into a vacuum and do no work at all.'),\n",
       " (0.030598384748214833,\n",
       "  'However, Q2 cannot be zero, because this would make ΔS negative and so violate the second law.'),\n",
       " (0.029726828627512625,\n",
       "  'Its introduction by the German physicist Rudolf Clausius in 1850 is a highlight of 19th-century physics.The idea of entropy provides a mathematical way to encode the intuitive notion of which processes are impossible, even though they would not violate the fundamental law of conservation of energy.'),\n",
       " (0.027098059396403616,\n",
       "  'Assume that there are two heat reservoirs R1 and R2 at temperatures T1 and T2 (such as the stove and the block of ice).'),\n",
       " (0.026206707155611988,\n",
       "  'For example, a block of ice placed on a hot stove surely melts, while the stove grows cooler.'),\n",
       " (0.0257465431139799,\n",
       "  'Suppose a heat engine absorbs heat Q1 from R1 and exhausts heat Q2 to R2 for each complete cycle.'),\n",
       " (0.025589360880026533,\n",
       "  'Because of this, the universe is said to be “running down.”'),\n",
       " (0.020224791251629928, 'Then by conservation of energy, dQ = dU + P dV.')]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ranked_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nwgFoPol7-6x",
    "outputId": "fdbfaf27-8e65-4957-a610-98e2c0d59003"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A process for which ΔS = 0 is reversible because an infinitesimal change would be sufficient to make the heat engine run backward as a refrigerator.The same reasoning can also determine the entropy change for the working substance in the heat engine, such as a gas in a cylinder with a movable piston.\n",
      "The Clausius form of the second law states that spontaneous change for an irreversible process in an isolated system (that is, one that does not exchange heat or work with its surroundings) always proceeds in the direction of increasing entropy.\n",
      "Because the net entropy change for the system plus reservoir is zero when maximum work is done and the entropy of the reservoir decreases by an amount dSreservoir = −dQ/T, this must be counterbalanced by an entropy increase of\n",
      "\n",
      " for the working gas so that dSsystem + dSreservoir = 0.\n",
      "For example, the block of ice and the stove constitute two parts of an isolated system for which total entropy increases as the ice melts.By the Clausius definition, if an amount of heat Q  flows into a large heat reservoir at temperature T above absolute zero, then the entropy increase is ΔS = Q/T.\n",
      "Thus, the observation that heat never flows spontaneously from cold to hot is equivalent to requiring the net entropy change to be positive for a spontaneous flow of heat.\n",
      "For any real process, less than the maximum work would be done (because of friction, for example), and so the actual amount of heat dQ′ absorbed from the heat reservoir would be less than the maximum amount dQ.\n",
      "Therefore, it can be stated that\n",
      "\n",
      " with dQ′ = dQ in the case of maximum work corresponding to a reversible process.This equation defines Ssystem as a thermodynamic state variable, meaning that its value is completely determined by the current state of the system and not by how the system reached that state.\n",
      "If an amount of heat Q flows from R1 to R2, then the net entropy change for the two reservoirs is \n",
      "\n",
      " which is positive provided that T1 > T2.\n",
      "If the gas absorbs an incremental amount of heat dQ from a heat reservoir at temperature T and expands reversibly against the maximum possible restraining pressure P, then it does the maximum work dW = P dV, where dV is the change in volume.\n",
      "Entropy is an extensive property in that its magnitude depends on the amount of material in the system.In one statistical interpretation of entropy, it is found that for a very large system in thermodynamic equilibrium, entropy S is proportional to the natural logarithm of a quantity Ω representing the maximum number of microscopic ways in which the macroscopic state corresponding to S can be realized; that is, S = k ln Ω, in which k is the Boltzmann constant that is related to molecular energy.All spontaneous processes are irreversible; hence, it has been said that the entropy of the universe is increasing: that is, more and more energy becomes unavailable for conversion into work.\n"
     ]
    }
   ],
   "source": [
    "# Specify number of sentences to form the summary\n",
    "sn = 10\n",
    "\n",
    "new=[]\n",
    "# Generate summary\n",
    "for i in range(sn):\n",
    "  print(ranked_sentences[i][1])\n",
    "  s=ranked_sentences[i][1]\n",
    "  new.append(ranked_sentences[i][1])\n",
    " # new += \"\".join (s)  \n",
    "#print(type(new))\n",
    "#print(new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "QYDPfFoT8S7Z"
   },
   "outputs": [],
   "source": [
    "df_test=pd.DataFrame(new,columns=['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 343
    },
    "id": "lElRxpB88jqt",
    "outputId": "aa024104-fc14-44c1-b6e6-2f2b73ea564d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A process for which ΔS = 0 is reversible becau...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Clausius form of the second law states tha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Because the net entropy change for the system ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>For example, the block of ice and the stove co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thus, the observation that heat never flows sp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>For any real process, less than the maximum wo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Therefore, it can be stated that\\n\\n with dQ′ ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>If an amount of heat Q flows from R1 to R2, th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>If the gas absorbs an incremental amount of he...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Entropy is an extensive property in that its m...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text\n",
       "0  A process for which ΔS = 0 is reversible becau...\n",
       "1  The Clausius form of the second law states tha...\n",
       "2  Because the net entropy change for the system ...\n",
       "3  For example, the block of ice and the stove co...\n",
       "4  Thus, the observation that heat never flows sp...\n",
       "5  For any real process, less than the maximum wo...\n",
       "6  Therefore, it can be stated that\\n\\n with dQ′ ...\n",
       "7  If an amount of heat Q flows from R1 to R2, th...\n",
       "8  If the gas absorbs an incremental amount of he...\n",
       "9  Entropy is an extensive property in that its m..."
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "tqU5gi1_8kmV"
   },
   "outputs": [],
   "source": [
    "docs_test=df_test['text'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "KibZqYIx9aEL"
   },
   "outputs": [],
   "source": [
    "def sort_coo(coo_matrix):\n",
    "    tuples = zip(coo_matrix.col, coo_matrix.data)\n",
    "\n",
    "    return sorted(tuples, key=lambda x: (x[1], x[0]), reverse=True)\n",
    "\n",
    "def extract_topn_from_vector(feature_names, sorted_items, topn=10):\n",
    "    \"\"\"get the feature names and tf-idf score of top n items\"\"\"\n",
    "    \n",
    "    #use only topn items from vector\n",
    "    sorted_items = sorted_items[:topn]\n",
    "\n",
    "    score_vals = []\n",
    "    feature_vals = []\n",
    "\n",
    "    for idx, score in sorted_items:\n",
    "        fname = feature_names[idx]\n",
    "        \n",
    "        #keep track of feature name and its corresponding score\n",
    "        score_vals.append(round(score, 3))\n",
    "        feature_vals.append(feature_names[idx])\n",
    "\n",
    "    #create a tuples of feature,score\n",
    "    #results = zip(feature_vals,score_vals)\n",
    "    results= {}\n",
    "    for idx in range(len(feature_vals)):\n",
    "        results[feature_vals[idx]]=score_vals[idx]\n",
    "    \n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5TKtemkD89oc",
    "outputId": "8fafb851-6f6d-439f-e12e-da8e88b0295f"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     C:\\Users\\USER\\AppData\\Roaming\\nltk_data...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['1850', '19th', 'absolute', 'absorbed', 'absorbs', 'actual', 'added', 'agrees', 'allowed', 'also', 'alternate', 'always', 'amount', 'article', 'assume', 'atmosphere', 'back', 'backward', 'bath', 'becomes', 'block', 'boltzmann', 'called', 'case', 'cause', 'century', 'change', 'clausius', 'cold', 'complete', 'completely', 'compressed', 'compression', 'concept', 'condition', 'confine', 'confined', 'conservation', 'constant', 'constitute', 'contrast', 'conversion', 'cooler', 'corresponding', 'corresponds', 'could', 'counterbalanced', 'current', 'cycle', 'cyclic', 'cylinder', 'decrease', 'deep', 'defines', 'definition', 'depending', 'depends', 'determine', 'determined', 'determines', 'direction', 'disorder', 'doe', 'done', 'dq', 'dsreservoir', 'dssystem', 'du', 'dv', 'dw', 'editor', 'effectively', 'efficiency', 'either', 'encode', 'energy', 'engine', 'engines', 'environment', 'equation', 'equilibrium', 'equivalent', 'even', 'everyday', 'example', 'exchange', 'exhaust', 'expand', 'expands', 'expansion', 'expressing', 'extensive', 'fashion', 'flow', 'force', 'form', 'found', 'freely', 'freeze', 'freezing', 'friction', 'fundamental', 'gas', 'gasoline', 'german', 'give', 'grows', 'ha', 'heat', 'hence', 'highlight', 'hot', 'hotter', 'however', 'ice', 'idea', 'impossible', 'increase', 'increasing', 'incremental', 'infinitesimal', 'insight', 'internal', 'interpretation', 'introduced', 'introduction', 'intuitive', 'irreversible', 'isolated', 'large', 'latter', 'law', 'le', 'limiting', 'little', 'ln', 'logarithm', 'macroscopic', 'magnitude', 'make', 'many', 'material', 'mathematical', 'maximum', 'meaning', 'measure', 'melt', 'melted', 'microscopic', 'might', 'molecular', 'motion', 'movable', 'moveable', 'must', 'natural', 'needed', 'negative', 'net', 'never', 'notion', 'number', 'observation', 'obtained', 'one', 'opened', 'ordered', 'part', 'per', 'phenomenon', 'physic', 'physicist', 'piston', 'placed', 'plus', 'positive', 'possible', 'precise', 'pressure', 'proceeds', 'process', 'progressive', 'property', 'proportional', 'provide', 'provided', 'provides', 'pushing', 'q1', 'q2', 'quantitative', 'quantity', 'r1', 'r2', 'randomness', 'reached', 'real', 'realized', 'reasoning', 'refrigerator', 'related', 'relative', 'representing', 'requiring', 'reservoir', 'restraining', 'reverse', 'reversible', 'reversibly', 'review', 'revise', 'rudolf', 'run', 'running', 'said', 'second', 'similarly', 'slight', 'small', 'smallest', 'spontaneous', 'spontaneously', 'ssystem', 'state', 'stated', 'statistical', 'steam', 'stove', 'submitted', 'substance', 'subtracted', 'sufficient', 'suppose', 'surely', 'surroundings', 'system', 't1', 't2', 'temperature', 'thaw', 'thawing', 'therefore', 'thermal', 'thermodynamic', 'thermodynamics', 'though', 'thus', 'total', 'turn', 'two', 'unavailable', 'unit', 'universe', 'useful', 'usual', 'vacuum', 'value', 'valve', 'variable', 'violate', 'volume', 'water', 'way', 'whether', 'work', 'working', 'would', 'yielding', 'zero', 'δs']\n",
      "  (0, 269)\t0.18563190312246824\n",
      "  (0, 266)\t0.16240597966196807\n",
      "  (0, 265)\t0.21836700366788758\n",
      "  (0, 231)\t0.21836700366788758\n",
      "  (0, 229)\t0.21836700366788758\n",
      "  (0, 212)\t0.21836700366788758\n",
      "  (0, 207)\t0.16240597966196807\n",
      "  (0, 199)\t0.21836700366788758\n",
      "  (0, 198)\t0.21836700366788758\n",
      "  (0, 180)\t0.12967087911654873\n",
      "  (0, 172)\t0.18563190312246824\n",
      "  (0, 152)\t0.21836700366788758\n",
      "  (0, 139)\t0.21836700366788758\n",
      "  (0, 120)\t0.18563190312246824\n",
      "  (0, 108)\t0.25934175823309746\n",
      "  (0, 102)\t0.18563190312246824\n",
      "  (0, 76)\t0.3712638062449365\n",
      "  (0, 57)\t0.18563190312246824\n",
      "  (0, 50)\t0.18563190312246824\n",
      "  (0, 26)\t0.23445114297481043\n",
      "  (0, 17)\t0.21836700366788758\n",
      "  (0, 9)\t0.18563190312246824\n",
      "  (1, 264)\t0.152837999881876\n",
      "  (1, 235)\t0.13878229735679623\n",
      "  (1, 234)\t0.28470559501076115\n",
      "  :\t:\n",
      "  (9, 155)\t0.16601579070456585\n",
      "  (9, 150)\t0.14112858929795313\n",
      "  (9, 148)\t0.16601579070456585\n",
      "  (9, 143)\t0.12347083888982362\n",
      "  (9, 141)\t0.16601579070456585\n",
      "  (9, 138)\t0.16601579070456585\n",
      "  (9, 137)\t0.16601579070456585\n",
      "  (9, 136)\t0.16601579070456585\n",
      "  (9, 135)\t0.16601579070456585\n",
      "  (9, 129)\t0.12347083888982362\n",
      "  (9, 127)\t0.12347083888982362\n",
      "  (9, 123)\t0.16601579070456585\n",
      "  (9, 118)\t0.14112858929795313\n",
      "  (9, 109)\t0.16601579070456585\n",
      "  (9, 96)\t0.16601579070456585\n",
      "  (9, 91)\t0.16601579070456585\n",
      "  (9, 80)\t0.12347083888982362\n",
      "  (9, 75)\t0.17824392526697574\n",
      "  (9, 56)\t0.16601579070456585\n",
      "  (9, 43)\t0.14112858929795313\n",
      "  (9, 41)\t0.16601579070456585\n",
      "  (9, 38)\t0.16601579070456585\n",
      "  (9, 21)\t0.16601579070456585\n",
      "  (9, 19)\t0.16601579070456585\n",
      "  (9, 12)\t0.0985836374832109\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data]   Unzipping taggers\\averaged_perceptron_tagger.zip.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "import nltk\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "\n",
    "#tfidf_transformer.idf_\n",
    "\n",
    "feature_names=cv.get_feature_names()\n",
    "print(feature_names)\n",
    "\n",
    "#generate tf-idf for all documents in your list. docs_test has 500 documents\n",
    "tf_idf_vector=tfidf_transformer.transform(cv.transform(docs_test))\n",
    "print(tf_idf_vector)\n",
    "results=[]\n",
    "for i in range(tf_idf_vector.shape[0]):\n",
    "    \n",
    "    # get vector for a single document\n",
    "    curr_vector=tf_idf_vector[i]\n",
    "    \n",
    "    #sort the tf-idf vector by descending order of scores\n",
    "    sorted_items=sort_coo(curr_vector.tocoo())\n",
    "\n",
    "    #extract only the top n; n here is 10\n",
    "    keywords=extract_topn_from_vector(feature_names,sorted_items,10)\n",
    "    \n",
    "   \n",
    "    results.append(keywords)\n",
    "\n",
    "    \n",
    "  \n",
    "\n",
    "   # nouns = [word for word,pos in tagged_sent if pos == 'NNP' and pos =='NN']\n",
    "\n",
    "  #  tagged.append(nouns)\n",
    "\n",
    "df_f=pd.DataFrame(zip(docs_test,results),columns=['doc','keywords'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 343
    },
    "id": "Kwpo5vxHCOLQ",
    "outputId": "babcf657-fdb5-4db1-c826-b8784c6e8a7a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc</th>\n",
       "      <th>keywords</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A process for which ΔS = 0 is reversible becau...</td>\n",
       "      <td>{'engine': 0.371, 'heat': 0.259, 'change': 0.2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Clausius form of the second law states tha...</td>\n",
       "      <td>{'surroundings': 0.285, 'proceeds': 0.285, 'is...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Because the net entropy change for the system ...</td>\n",
       "      <td>{'dsreservoir': 0.477, 'reservoir': 0.405, 'wo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>For example, the block of ice and the stove co...</td>\n",
       "      <td>{'ice': 0.376, 'heat': 0.3, 'total': 0.252, 'i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thus, the observation that heat never flows sp...</td>\n",
       "      <td>{'heat': 0.337, 'thus': 0.284, 'spontaneously'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>For any real process, less than the maximum wo...</td>\n",
       "      <td>{'dq': 0.473, 'would': 0.352, 'maximum': 0.352...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Therefore, it can be stated that\\n\\n with dQ′ ...</td>\n",
       "      <td>{'state': 0.443, 'dq': 0.397, 'variable': 0.19...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>If an amount of heat Q flows from R1 to R2, th...</td>\n",
       "      <td>{'t2': 0.36, 't1': 0.36, 'provided': 0.36, 'po...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>If the gas absorbs an incremental amount of he...</td>\n",
       "      <td>{'dv': 0.45, 'maximum': 0.335, 'heat': 0.267, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Entropy is an extensive property in that its m...</td>\n",
       "      <td>{'energy': 0.178, 'universe': 0.166, 'statisti...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 doc  \\\n",
       "0  A process for which ΔS = 0 is reversible becau...   \n",
       "1  The Clausius form of the second law states tha...   \n",
       "2  Because the net entropy change for the system ...   \n",
       "3  For example, the block of ice and the stove co...   \n",
       "4  Thus, the observation that heat never flows sp...   \n",
       "5  For any real process, less than the maximum wo...   \n",
       "6  Therefore, it can be stated that\\n\\n with dQ′ ...   \n",
       "7  If an amount of heat Q flows from R1 to R2, th...   \n",
       "8  If the gas absorbs an incremental amount of he...   \n",
       "9  Entropy is an extensive property in that its m...   \n",
       "\n",
       "                                            keywords  \n",
       "0  {'engine': 0.371, 'heat': 0.259, 'change': 0.2...  \n",
       "1  {'surroundings': 0.285, 'proceeds': 0.285, 'is...  \n",
       "2  {'dsreservoir': 0.477, 'reservoir': 0.405, 'wo...  \n",
       "3  {'ice': 0.376, 'heat': 0.3, 'total': 0.252, 'i...  \n",
       "4  {'heat': 0.337, 'thus': 0.284, 'spontaneously'...  \n",
       "5  {'dq': 0.473, 'would': 0.352, 'maximum': 0.352...  \n",
       "6  {'state': 0.443, 'dq': 0.397, 'variable': 0.19...  \n",
       "7  {'t2': 0.36, 't1': 0.36, 'provided': 0.36, 'po...  \n",
       "8  {'dv': 0.45, 'maximum': 0.335, 'heat': 0.267, ...  \n",
       "9  {'energy': 0.178, 'universe': 0.166, 'statisti...  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wo0mzk1HErlq",
    "outputId": "aaa0fe6c-3041-40e6-87f1-abfea151c045"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'surroundings': 0.285,\n",
       " 'proceeds': 0.285,\n",
       " 'isolated': 0.285,\n",
       " 'form': 0.285,\n",
       " 'exchange': 0.285,\n",
       " 'always': 0.285,\n",
       " 'second': 0.242,\n",
       " 'one': 0.242,\n",
       " 'increasing': 0.242,\n",
       " 'law': 0.212}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_f['keywords'][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "id": "4h0vHU5i_NtU"
   },
   "outputs": [],
   "source": [
    "from nltk.tag import pos_tag\n",
    "\n",
    "k=[]\n",
    "for d in df_f['keywords']:\n",
    "  k.append(pos_tag(d))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WEIJvRz_dFUd",
    "outputId": "854e4850-0fd7-497e-f88b-f117ce12d84b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[('engine', 'NN'),\n",
       "  ('heat', 'NN'),\n",
       "  ('change', 'NN'),\n",
       "  ('working', 'VBG'),\n",
       "  ('sufficient', 'JJ'),\n",
       "  ('substance', 'NN'),\n",
       "  ('run', 'VB'),\n",
       "  ('refrigerator', 'NN'),\n",
       "  ('reasoning', 'VBG'),\n",
       "  ('movable', 'JJ')],\n",
       " [('surroundings', 'NNS'),\n",
       "  ('proceeds', 'NNS'),\n",
       "  ('isolated', 'VBD'),\n",
       "  ('form', 'JJ'),\n",
       "  ('exchange', 'NN'),\n",
       "  ('always', 'RB'),\n",
       "  ('second', 'JJ'),\n",
       "  ('one', 'CD'),\n",
       "  ('increasing', 'VBG'),\n",
       "  ('law', 'NN')],\n",
       " [('dsreservoir', 'NN'),\n",
       "  ('reservoir', 'NN'),\n",
       "  ('working', 'VBG'),\n",
       "  ('plus', 'CC'),\n",
       "  ('must', 'MD'),\n",
       "  ('dssystem', 'VB'),\n",
       "  ('dq', 'NN'),\n",
       "  ('counterbalanced', 'VBN'),\n",
       "  ('gas', 'NN'),\n",
       "  ('done', 'VBN')],\n",
       " [('ice', 'NN'),\n",
       "  ('heat', 'NN'),\n",
       "  ('total', 'JJ'),\n",
       "  ('isolated', 'JJ'),\n",
       "  ('definition', 'NN'),\n",
       "  ('constitute', 'NN'),\n",
       "  ('absolute', 'NN'),\n",
       "  ('δs', 'CD'),\n",
       "  ('two', 'CD'),\n",
       "  ('reservoir', 'NN')],\n",
       " [('heat', 'NN'),\n",
       "  ('thus', 'RB'),\n",
       "  ('spontaneously', 'RB'),\n",
       "  ('requiring', 'VBG'),\n",
       "  ('positive', 'JJ'),\n",
       "  ('observation', 'NN'),\n",
       "  ('never', 'RB'),\n",
       "  ('flow', 'JJ'),\n",
       "  ('equivalent', 'NN'),\n",
       "  ('cold', 'NN')],\n",
       " [('dq', 'NN'),\n",
       "  ('would', 'MD'),\n",
       "  ('maximum', 'VB'),\n",
       "  ('heat', 'NN'),\n",
       "  ('amount', 'NN'),\n",
       "  ('real', 'JJ'),\n",
       "  ('friction', 'NN'),\n",
       "  ('actual', 'JJ'),\n",
       "  ('absorbed', 'VBD'),\n",
       "  ('reservoir', 'NN')],\n",
       " [('state', 'NN'),\n",
       "  ('dq', 'NN'),\n",
       "  ('variable', 'JJ'),\n",
       "  ('therefore', 'RB'),\n",
       "  ('stated', 'JJ'),\n",
       "  ('ssystem', 'NN'),\n",
       "  ('reached', 'VBD'),\n",
       "  ('meaning', 'NN'),\n",
       "  ('determined', 'VBN'),\n",
       "  ('defines', 'NNS')],\n",
       " [('t2', 'NN'),\n",
       "  ('t1', 'NN'),\n",
       "  ('provided', 'VBD'),\n",
       "  ('positive', 'JJ'),\n",
       "  ('two', 'CD'),\n",
       "  ('r2', 'JJ'),\n",
       "  ('r1', 'FW'),\n",
       "  ('net', 'JJ'),\n",
       "  ('heat', 'NN'),\n",
       "  ('amount', 'NN')],\n",
       " [('dv', 'NN'),\n",
       "  ('maximum', 'JJ'),\n",
       "  ('heat', 'NN'),\n",
       "  ('volume', 'NN'),\n",
       "  ('reversibly', 'RB'),\n",
       "  ('pressure', 'NN'),\n",
       "  ('incremental', 'JJ'),\n",
       "  ('expands', 'VBZ'),\n",
       "  ('dw', 'JJ'),\n",
       "  ('dq', 'NN')],\n",
       " [('energy', 'NN'),\n",
       "  ('universe', 'JJ'),\n",
       "  ('statistical', 'NN'),\n",
       "  ('said', 'VBD'),\n",
       "  ('representing', 'VBG'),\n",
       "  ('related', 'VBN'),\n",
       "  ('realized', 'JJ'),\n",
       "  ('quantity', 'NN'),\n",
       "  ('proportional', 'JJ'),\n",
       "  ('property', 'NN')]]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-gNlaOL9UWlT",
    "outputId": "2c4ca3f9-2a07-43f3-bcfa-8b89786cd3b8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('engine', 'NN')\n",
      "('heat', 'NN')\n",
      "('change', 'NN')\n",
      "('working', 'VBG')\n",
      "('sufficient', 'JJ')\n",
      "('substance', 'NN')\n",
      "('run', 'VB')\n",
      "('refrigerator', 'NN')\n",
      "('reasoning', 'VBG')\n",
      "('movable', 'JJ')\n",
      "('surroundings', 'NNS')\n",
      "('proceeds', 'NNS')\n",
      "('isolated', 'VBD')\n",
      "('form', 'JJ')\n",
      "('exchange', 'NN')\n",
      "('always', 'RB')\n",
      "('second', 'JJ')\n",
      "('one', 'CD')\n",
      "('increasing', 'VBG')\n",
      "('law', 'NN')\n",
      "('dsreservoir', 'NN')\n",
      "('reservoir', 'NN')\n",
      "('working', 'VBG')\n",
      "('plus', 'CC')\n",
      "('must', 'MD')\n",
      "('dssystem', 'VB')\n",
      "('dq', 'NN')\n",
      "('counterbalanced', 'VBN')\n",
      "('gas', 'NN')\n",
      "('done', 'VBN')\n",
      "('ice', 'NN')\n",
      "('heat', 'NN')\n",
      "('total', 'JJ')\n",
      "('isolated', 'JJ')\n",
      "('definition', 'NN')\n",
      "('constitute', 'NN')\n",
      "('absolute', 'NN')\n",
      "('δs', 'CD')\n",
      "('two', 'CD')\n",
      "('reservoir', 'NN')\n",
      "('heat', 'NN')\n",
      "('thus', 'RB')\n",
      "('spontaneously', 'RB')\n",
      "('requiring', 'VBG')\n",
      "('positive', 'JJ')\n",
      "('observation', 'NN')\n",
      "('never', 'RB')\n",
      "('flow', 'JJ')\n",
      "('equivalent', 'NN')\n",
      "('cold', 'NN')\n",
      "('dq', 'NN')\n",
      "('would', 'MD')\n",
      "('maximum', 'VB')\n",
      "('heat', 'NN')\n",
      "('amount', 'NN')\n",
      "('real', 'JJ')\n",
      "('friction', 'NN')\n",
      "('actual', 'JJ')\n",
      "('absorbed', 'VBD')\n",
      "('reservoir', 'NN')\n",
      "('state', 'NN')\n",
      "('dq', 'NN')\n",
      "('variable', 'JJ')\n",
      "('therefore', 'RB')\n",
      "('stated', 'JJ')\n",
      "('ssystem', 'NN')\n",
      "('reached', 'VBD')\n",
      "('meaning', 'NN')\n",
      "('determined', 'VBN')\n",
      "('defines', 'NNS')\n",
      "('t2', 'NN')\n",
      "('t1', 'NN')\n",
      "('provided', 'VBD')\n",
      "('positive', 'JJ')\n",
      "('two', 'CD')\n",
      "('r2', 'JJ')\n",
      "('r1', 'FW')\n",
      "('net', 'JJ')\n",
      "('heat', 'NN')\n",
      "('amount', 'NN')\n",
      "('dv', 'NN')\n",
      "('maximum', 'JJ')\n",
      "('heat', 'NN')\n",
      "('volume', 'NN')\n",
      "('reversibly', 'RB')\n",
      "('pressure', 'NN')\n",
      "('incremental', 'JJ')\n",
      "('expands', 'VBZ')\n",
      "('dw', 'JJ')\n",
      "('dq', 'NN')\n",
      "('energy', 'NN')\n",
      "('universe', 'JJ')\n",
      "('statistical', 'NN')\n",
      "('said', 'VBD')\n",
      "('representing', 'VBG')\n",
      "('related', 'VBN')\n",
      "('realized', 'JJ')\n",
      "('quantity', 'NN')\n",
      "('proportional', 'JJ')\n",
      "('property', 'NN')\n"
     ]
    }
   ],
   "source": [
    "tagged=[]\n",
    "for i in k:\n",
    "  #print(i)\n",
    "  tag=[]\n",
    "  for j in i:\n",
    "    print(j)\n",
    "    if j[1]=='NN'or j[1]=='NNS':\n",
    "      tag.append(j[0])\n",
    "\n",
    "  tagged.append(tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Lt3HP-d_VDBJ",
    "outputId": "cb16dfad-5235-427e-9984-e3e099d26d01"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['engine', 'heat', 'change', 'substance', 'refrigerator'],\n",
       " ['surroundings', 'proceeds', 'exchange', 'law'],\n",
       " ['dsreservoir', 'reservoir', 'dq', 'gas'],\n",
       " ['ice', 'heat', 'definition', 'constitute', 'absolute', 'reservoir'],\n",
       " ['heat', 'observation', 'equivalent', 'cold'],\n",
       " ['dq', 'heat', 'amount', 'friction', 'reservoir'],\n",
       " ['state', 'dq', 'ssystem', 'meaning', 'defines'],\n",
       " ['t2', 't1', 'heat', 'amount'],\n",
       " ['dv', 'heat', 'volume', 'pressure', 'dq'],\n",
       " ['energy', 'statistical', 'quantity', 'property']]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "id": "3HOjiN_ZYkii"
   },
   "outputs": [],
   "source": [
    "df_f['pos']=tagged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 343
    },
    "id": "F0Tecewth8--",
    "outputId": "a20e5a6d-5601-41ca-f5d4-770a50fca673"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc</th>\n",
       "      <th>keywords</th>\n",
       "      <th>pos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A process for which ΔS = 0 is reversible becau...</td>\n",
       "      <td>{'engine': 0.371, 'heat': 0.259, 'change': 0.2...</td>\n",
       "      <td>[engine, heat, change, substance, refrigerator]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Clausius form of the second law states tha...</td>\n",
       "      <td>{'surroundings': 0.285, 'proceeds': 0.285, 'is...</td>\n",
       "      <td>[surroundings, proceeds, exchange, law]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Because the net entropy change for the system ...</td>\n",
       "      <td>{'dsreservoir': 0.477, 'reservoir': 0.405, 'wo...</td>\n",
       "      <td>[dsreservoir, reservoir, dq, gas]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>For example, the block of ice and the stove co...</td>\n",
       "      <td>{'ice': 0.376, 'heat': 0.3, 'total': 0.252, 'i...</td>\n",
       "      <td>[ice, heat, definition, constitute, absolute, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thus, the observation that heat never flows sp...</td>\n",
       "      <td>{'heat': 0.337, 'thus': 0.284, 'spontaneously'...</td>\n",
       "      <td>[heat, observation, equivalent, cold]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>For any real process, less than the maximum wo...</td>\n",
       "      <td>{'dq': 0.473, 'would': 0.352, 'maximum': 0.352...</td>\n",
       "      <td>[dq, heat, amount, friction, reservoir]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Therefore, it can be stated that\\n\\n with dQ′ ...</td>\n",
       "      <td>{'state': 0.443, 'dq': 0.397, 'variable': 0.19...</td>\n",
       "      <td>[state, dq, ssystem, meaning, defines]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>If an amount of heat Q flows from R1 to R2, th...</td>\n",
       "      <td>{'t2': 0.36, 't1': 0.36, 'provided': 0.36, 'po...</td>\n",
       "      <td>[t2, t1, heat, amount]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>If the gas absorbs an incremental amount of he...</td>\n",
       "      <td>{'dv': 0.45, 'maximum': 0.335, 'heat': 0.267, ...</td>\n",
       "      <td>[dv, heat, volume, pressure, dq]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Entropy is an extensive property in that its m...</td>\n",
       "      <td>{'energy': 0.178, 'universe': 0.166, 'statisti...</td>\n",
       "      <td>[energy, statistical, quantity, property]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 doc  \\\n",
       "0  A process for which ΔS = 0 is reversible becau...   \n",
       "1  The Clausius form of the second law states tha...   \n",
       "2  Because the net entropy change for the system ...   \n",
       "3  For example, the block of ice and the stove co...   \n",
       "4  Thus, the observation that heat never flows sp...   \n",
       "5  For any real process, less than the maximum wo...   \n",
       "6  Therefore, it can be stated that\\n\\n with dQ′ ...   \n",
       "7  If an amount of heat Q flows from R1 to R2, th...   \n",
       "8  If the gas absorbs an incremental amount of he...   \n",
       "9  Entropy is an extensive property in that its m...   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  {'engine': 0.371, 'heat': 0.259, 'change': 0.2...   \n",
       "1  {'surroundings': 0.285, 'proceeds': 0.285, 'is...   \n",
       "2  {'dsreservoir': 0.477, 'reservoir': 0.405, 'wo...   \n",
       "3  {'ice': 0.376, 'heat': 0.3, 'total': 0.252, 'i...   \n",
       "4  {'heat': 0.337, 'thus': 0.284, 'spontaneously'...   \n",
       "5  {'dq': 0.473, 'would': 0.352, 'maximum': 0.352...   \n",
       "6  {'state': 0.443, 'dq': 0.397, 'variable': 0.19...   \n",
       "7  {'t2': 0.36, 't1': 0.36, 'provided': 0.36, 'po...   \n",
       "8  {'dv': 0.45, 'maximum': 0.335, 'heat': 0.267, ...   \n",
       "9  {'energy': 0.178, 'universe': 0.166, 'statisti...   \n",
       "\n",
       "                                                 pos  \n",
       "0    [engine, heat, change, substance, refrigerator]  \n",
       "1            [surroundings, proceeds, exchange, law]  \n",
       "2                  [dsreservoir, reservoir, dq, gas]  \n",
       "3  [ice, heat, definition, constitute, absolute, ...  \n",
       "4              [heat, observation, equivalent, cold]  \n",
       "5            [dq, heat, amount, friction, reservoir]  \n",
       "6             [state, dq, ssystem, meaning, defines]  \n",
       "7                             [t2, t1, heat, amount]  \n",
       "8                   [dv, heat, volume, pressure, dq]  \n",
       "9          [energy, statistical, quantity, property]  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "id": "7DwL-N4Fh-kw",
    "outputId": "0733ca2c-36ba-46f3-f975-a3c33ccfa80e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A process for which ΔS = 0 is reversible because an infinitesimal change would be sufficient to make the heat engine run backward as a refrigerator.The same reasoning can also determine the entropy change for the working substance in the heat engine, such as a gas in a cylinder with a movable piston.'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_f['doc'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting gensim\n",
      "  Downloading gensim-4.0.1-cp37-cp37m-win_amd64.whl (23.9 MB)\n",
      "Requirement already satisfied: scipy>=0.18.1 in c:\\users\\user\\anaconda3\\envs\\retinaface-tf2\\lib\\site-packages (from gensim) (1.6.0)\n",
      "Requirement already satisfied: Cython==0.29.21 in c:\\users\\user\\anaconda3\\envs\\retinaface-tf2\\lib\\site-packages (from gensim) (0.29.21)\n",
      "Collecting smart-open>=1.8.1\n",
      "  Downloading smart_open-5.0.0-py3-none-any.whl (56 kB)\n",
      "Requirement already satisfied: numpy>=1.11.3 in c:\\users\\user\\anaconda3\\envs\\retinaface-tf2\\lib\\site-packages (from gensim) (1.18.0)\n",
      "Installing collected packages: smart-open, gensim\n",
      "Successfully installed gensim-4.0.1 smart-open-5.0.0\n"
     ]
    }
   ],
   "source": [
    "!pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "id": "G-DxL012ikoR"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\USER\\anaconda3\\envs\\retinaface-tf2\\lib\\site-packages\\gensim\\similarities\\__init__.py:15: UserWarning: The gensim.similarities.levenshtein submodule is disabled, because the optional Levenshtein package <https://pypi.org/project/python-Levenshtein/> is unavailable. Install Levenhstein (e.g. `pip install python-Levenshtein`) to suppress this warning.\n",
      "  warnings.warn(msg)\n"
     ]
    }
   ],
   "source": [
    "import gensim\n",
    "from gensim.test.utils import datapath, get_tmpfile\n",
    "from gensim.models import KeyedVectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "id": "ZFKTeC_AsuTT"
   },
   "outputs": [],
   "source": [
    "glove_file = 'C:/Users/USER/Desktop/glove/glove.6B.300d.txt'\n",
    "tmp_file = 'word2vec-glove.6B.300d.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "id": "3qGPETOwtKhH"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\USER\\anaconda3\\envs\\retinaface-tf2\\lib\\site-packages\\ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `glove2word2vec` (KeyedVectors.load_word2vec_format(.., binary=False, no_header=True) loads GLoVE text vectors.).\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "from gensim.scripts.glove2word2vec import glove2word2vec\n",
    "glove2word2vec(glove_file, tmp_file)\n",
    "model = KeyedVectors.load_word2vec_format(tmp_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "id": "xpBrScqBvqwp"
   },
   "outputs": [],
   "source": [
    "def generate_distractors(answer, count):\n",
    "    answer = str.lower(answer)\n",
    "    \n",
    "    ##Extracting closest words for the answer. \n",
    "    try:\n",
    "        closestWords = model.most_similar(positive=[answer], topn=count)\n",
    "    except:\n",
    "        #In case the word is not in the vocabulary, or other problem not loading embeddings\n",
    "        return []\n",
    "\n",
    "    #Return count many distractors\n",
    "    distractors = list(map(lambda x: x[0], closestWords))[0:count]\n",
    "    \n",
    "    return distractors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GmIryWS9v0zJ",
    "outputId": "b2eeaef5-d400-4e7d-9ce7-e9f85768d836"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['extrovert', 'loner', 'egomaniac', 'introverted']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_distractors('introvert', 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "id": "D1tY0KATyNWd"
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import re\n",
    "\n",
    "# Create the blank in string\n",
    "def replaceIC(word, sentence):\n",
    "    rep = re.compile(re.escape(word), re.IGNORECASE)\n",
    "    return rep.sub('*********', sentence)\n",
    "\n",
    "# For a sentence create a blank space.\n",
    "# It first tries to randomly selection proper-noun \n",
    "# and if the proper noun is not found, it selects a noun randomly.\n",
    "def removeWord(sentence, poss):\n",
    "    \n",
    "    if len(poss) > 0:\n",
    "        word=lambda: random.choice(poss)\n",
    "        word=word()\n",
    "        #word= random.choice(poss)\n",
    "        #print(word)\n",
    "        replaced = replaceIC(word, sentence)\n",
    "        return (word, sentence, replaced)\n",
    "    else:\n",
    "        print(\"words are empty\")\n",
    "        return (None, sentence, None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KeJofXPhyj5U",
    "outputId": "99c30722-635c-46a9-e1de-663bd27cca68"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A process for which ΔS = 0 is reversible because an infinitesimal change would be sufficient to make the heat ********* run backward as a refrigerator.The same reasoning can also determine the entropy change for the working substance in the heat *********, such as a gas in a cylinder with a movable piston.\n",
      "----------\n",
      "Ans : engine\n",
      "============================================\n",
      "1. engine\n",
      "2. diesel\n",
      "3. cylinder\n",
      "4. engines\n",
      "============================================\n",
      "The Clausius form of the second law states that spontaneous change for an irreversible process in an isolated system (that is, one that does not ********* heat or work with its surroundings) always proceeds in the direction of increasing entropy.\n",
      "----------\n",
      "Ans : exchange\n",
      "============================================\n",
      "1. stock\n",
      "2. trading\n",
      "3. exchanges\n",
      "4. exchange\n",
      "============================================\n",
      "Because the net entropy change for the system plus reservoir is zero when maximum work is done and the entropy of the reservoir decreases by an amount dSreservoir = −dQ/T, this must be counterbalanced by an entropy increase of\n",
      "\n",
      " for the working ********* so that dSsystem + dSreservoir = 0.\n",
      "----------\n",
      "Ans : gas\n",
      "============================================\n",
      "1. gas\n",
      "2. gasoline\n",
      "3. natural\n",
      "4. oil\n",
      "============================================\n",
      "For example, the block of ********* and the stove constitute two parts of an isolated system for which total entropy increases as the ********* melts.By the Clausius definition, if an amount of heat Q  flows into a large heat reservoir at temperature T above absolute zero, then the entropy increase is ΔS = Q/T.\n",
      "----------\n",
      "Ans : ice\n",
      "============================================\n",
      "1. hockey\n",
      "2. skating\n",
      "3. ice\n",
      "4. cream\n",
      "============================================\n",
      "Thus, the observation that heat never flows spontaneously from cold to hot is ********* to requiring the net entropy change to be positive for a spontaneous flow of heat.\n",
      "----------\n",
      "Ans : equivalent\n",
      "============================================\n",
      "1. equivalent\n",
      "2. comparable\n",
      "3. roughly\n",
      "4. corresponding\n",
      "============================================\n",
      "For any real process, less than the maximum work would be done (because of friction, for example), and so the actual amount of ********* dQ′ absorbed from the ********* reservoir would be less than the maximum amount dQ.\n",
      "----------\n",
      "Ans : heat\n",
      "============================================\n",
      "1. humidity\n",
      "2. temperatures\n",
      "3. temperature\n",
      "4. heat\n",
      "============================================\n",
      "Therefore, it can be stated that\n",
      "\n",
      " with dQ′ = dQ in the case of maximum work corresponding to a reversible process.This equation defines Ssystem as a thermodynamic state variable, ********* that its value is completely determined by the current state of the system and not by how the system reached that state.\n",
      "----------\n",
      "Ans : meaning\n",
      "============================================\n",
      "1. word\n",
      "2. meaning\n",
      "3. means\n",
      "4. hence\n",
      "============================================\n",
      "If an ********* of heat Q flows from R1 to R2, then the net entropy change for the two reservoirs is \n",
      "\n",
      " which is positive provided that T1 > T2.\n",
      "----------\n",
      "Ans : amount\n",
      "============================================\n",
      "1. amount\n",
      "2. substantial\n",
      "3. quantity\n",
      "4. amounts\n",
      "============================================\n",
      "If the gas absorbs an incremental amount of heat dQ from a heat reservoir at temperature T and expands reversibly against the maximum possible restraining pressure P, then it does the maximum work dW = P dV, where dV is the change in *********.\n",
      "----------\n",
      "Ans : volume\n",
      "============================================\n",
      "1. volumes\n",
      "2. volume\n",
      "3. compared\n",
      "4. turnover\n",
      "============================================\n",
      "Entropy is an extensive property in that its magnitude depends on the amount of material in the system.In one ********* interpretation of entropy, it is found that for a very large system in thermodynamic equilibrium, entropy S is proportional to the natural logarithm of a quantity Ω representing the maximum number of microscopic ways in which the macroscopic state corresponding to S can be realized; that is, S = k ln Ω, in which k is the Boltzmann constant that is related to molecular energy.All spontaneous processes are irreversible; hence, it has been said that the entropy of the universe is increasing: that is, more and more energy becomes unavailable for conversion into work.\n",
      "----------\n",
      "Ans : statistical\n",
      "============================================\n",
      "1. statistical\n",
      "2. statistics\n",
      "3. mathematical\n",
      "4. analysis\n",
      "============================================\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(df_f)):\n",
    "  sentence=df_f['doc'][i]\n",
    "  poss=df_f['pos'][i]\n",
    " \n",
    "\n",
    "  (word,sentence,replaced)= removeWord(sentence,poss)\n",
    "\n",
    "  if replaced is None:\n",
    "    print(\"Founded none for \")\n",
    "    print(sentence)\n",
    "  else:\n",
    "    print(replaced)\n",
    "    print('----------')\n",
    "    print(\"Ans : {}\".format(word))\n",
    "    distractor= generate_distractors(word, 3)\n",
    "    print(\"============================================\")\n",
    "    distractor.append(word)\n",
    "    for i in range(len(distractor)):\n",
    "      random_index = lambda: random.randrange(len(distractor))\n",
    "      random_index=random_index()\n",
    "      print('{}. {}'.format(i+1,distractor.pop(random_index)))\n",
    "\n",
    "    print(\"============================================\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "MCQ_Q/A_gen.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "retinaface-tf2",
   "language": "python",
   "name": "retinaface-tf2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}