Wikipedia Article Summarizer with NLP.ipynb


{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "#               **Wikipedia Article Summarizer** _(Python project based on NLP techniques)_\n",
    "\n",
    "#                                      Text Summarization with NLTK\n",
    "*(NLTK, or Natural Language Toolkit, is a Python package that you can use for NLP.)*\n",
    "\n",
    " *Since I like F-Secure and wishing to attend their trainings, I search for them and with this simple Wikipedia Article summarizer, I also practise NLP and Python, meanwhile learning more about F-Secure, its history, culture, etc.*"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "source": [
    "# Install these as requirements if you need. You may also try \"pip3 install beautifulsoup4\" if \"pip\" encounters errors.\n",
    "# $ pip install beautifulsoup4\n",
    "# $ pip install lxml"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Scrapping Wikipedia Article"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "source": [
    "import bs4 as bs\n",
    "import urllib.request\n",
    "import re\n",
    "\n",
    "# Normally, in Jupyter Notebooks, you may prefer to give a fixed URL, change the URL when you need it\n",
    "# and not ask for user input.But I wanted to see which articles, \n",
    "# I can get a better summary and when the NLTK does \"so so\":)\n",
    "userLink = input(\"Which Wikipedia article would you want me to summarize: \")\n",
    "# Provide the Wikipedia URL like this: https://\n",
    "raw_data = urllib.request.urlopen(userLink) \n",
    "document = raw_data.read()\n",
    "\n",
    "parsed_document = bs.BeautifulSoup(document,'lxml')\n",
    "\n",
    "article_paras = parsed_document.find_all('p')\n",
    "\n",
    "scrapped_data = \"\"\n",
    "\n",
    "for para in article_paras:\n",
    "    scrapped_data += para.text"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "source": [
    "print(scrapped_data[:1500])"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Information Security, sometimes shortened to InfoSec, is the practice of protecting information by mitigating information risks. It is part of information risk management.[1] It typically involves preventing or reducing the probability of unauthorized/inappropriate access to data, or the unlawful use, disclosure, disruption, deletion, corruption, modification, inspection, recording, or devaluation of information.[2] It also involves actions intended to reduce the adverse impacts of such incidents. Protected information may take any form, e.g. electronic or physical, tangible (e.g. paperwork) or intangible (e.g. knowledge).[3][4] Information security's primary focus is the balanced protection of the confidentiality, integrity, and availability of data (also known as the CIA triad) while maintaining a focus on efficient policy implementation, all without hampering organization productivity.[5] This is largely achieved through a structured risk management process that involves: \n",
      "To standardize this discipline, academics and professionals collaborate to offer guidance, policies, and industry standards on password, antivirus software, firewall, encryption software, legal liability, security awareness and training, and so forth.[7] This standardization may be further driven by a wide variety of laws and regulations that affect how data is accessed, processed, stored, transferred and destroyed.[8] However, the implementation of any standards and guidance within an entity may have li\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Text Cleaning"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "source": [
    "scrapped_data = re.sub(r'\\[[0-9]*\\]', ' ',  scrapped_data)\n",
    "scrapped_data = re.sub(r'\\s+', ' ',  scrapped_data)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "source": [
    "formatted_text = re.sub('[^a-zA-Z]', ' ', scrapped_data)\n",
    "formatted_text = re.sub(r'\\s+', ' ', formatted_text)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Finding Word Frequencies"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "source": [
    "import nltk #if you don't have it, then>> python3 -m pip install nltk\n",
    "all_sentences = nltk.sent_tokenize(scrapped_data)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "source": [
    "# Stop Words are the words that you will most probably ignore, so we filter them out of the text.\n",
    "stopwords = nltk.corpus.stopwords.words('english')\n",
    "\n",
    "word_freq = {}\n",
    "for word in nltk.word_tokenize(formatted_text):\n",
    "    if word not in stopwords:\n",
    "        if word not in word_freq.keys():\n",
    "            word_freq[word] = 1\n",
    "        else:\n",
    "            word_freq[word] += 1"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "source": [
    "max_freq = max(word_freq.values())\n",
    "\n",
    "for word in word_freq.keys():\n",
    "    word_freq[word] = (word_freq[word]/max_freq)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Finding Sentence Scores"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "source": [
    "sentence_scores = {}\n",
    "for sentence in all_sentences:\n",
    "    for token in nltk.word_tokenize(sentence.lower()):\n",
    "        if token in word_freq.keys():\n",
    "            if len(sentence.split(' ')) <25:\n",
    "                if sentence not in sentence_scores.keys():\n",
    "                    sentence_scores[sentence] = word_freq[token]\n",
    "                else:\n",
    "                    sentence_scores[sentence] += word_freq[token]"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Printing Summaries"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "source": [
    "import heapq\n",
    "selected_sentences= heapq.nlargest(5, sentence_scores, key=sentence_scores.get)\n",
    "\n",
    "text_summary = ' '.join(selected_sentences)\n",
    "print(text_summary)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Information security must protect information throughout its lifespan, from the initial creation of the information on through to the final disposal of the information. During its lifetime, information may pass through many different information processing systems and through many different parts of information processing systems. Information Security, sometimes shortened to InfoSec, is the practice of protecting information by mitigating information risks. In the realm of information security, availability can often be viewed as one of the most important parts of a successful information security program. In information security, confidentiality \"is the property, that information is not made available or disclosed to unauthorized individuals, entities, or processes.\"\n"
     ]
    }
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.9.5 64-bit"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "interpreter": {
   "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}