article_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The web is full of data. You will find it in different shapes and formats; simple tabular sheets, excel files, large and unstructered NoSql databases. \n",
    "\n",
    "The variety of content is overwhelming: texts, logs, tweets, images, comments, likes, views, videos, news headlines. All of this is constantly produced by batch or on a real time fashion, all over the world, generating <a href=\"http://www.vcloudnews.com/every-day-big-data-statistics-2-5-quintillion-bytes-of-data-created-daily/\"> quintillions of bytes</a> everyday. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/bigdata_resized.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you think how you could do to extract value from this large amount of data, you could qualify many applications:\n",
    "\n",
    "- If you're a marketer, you could measure the impact of your newly released product by leveraging user's reviews posted online and applying sentiment analysis on them. You'll then catch the uninterested/unsatisfied users and understand what made them unhappy. \n",
    "- If you're into finance, you can collect stocks historical data and build statistical models to predict the future stock prices.\n",
    "- If you collect your country's open data i.e growth rate, crime rate, unemployment, etc, you could build applications that might solve social problems.\n",
    "\n",
    "In this tutorial, I'll show you how you can easily collect news feeds from 60 different data sources (Google News, The BBC, Business Insider, BuzzFeed, etc) and apply machine learning algorithms on them to automatically extract the hidden topics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/news_sources.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Topic mining is very helpful. It is integrated in many recommendation systems. It is present in social media monitoring tools, marketing intelligence softwares, etc.\n",
    "\n",
    "Once you have a large volume of text data that you can't analyze by hand, topic modeling algorithms can do the job for you.\n",
    "\n",
    "\n",
    "This post is structured as follow:\n",
    "\n",
    "- **Data extraction:** I'll collect the news by requesting an external REST API called newsapi. I'll connect to this service through a python script that may run on you PC.\n",
    "\n",
    "- **Modeling:** Once the data is collected and stored, I'll ingest it in a pandas dataframe. I'll first preprocess it using text preprocessing tokenization and the tfidf algorithm and then I'll cluster it using 2 different algorithms: K-means and Latent Dirichlet Allocation (LDA). Details below.\n",
    "\n",
    "- **Visualization:** Finally I'll visualize the news clusters using two interactive python visualization libraries. They're called **Bokeh** and **pyldavis**. They're awesome and you'll see why.\n",
    " \n",
    "Let's get started !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/comic.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Environment setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, I'll be using python 2.7\n",
    "\n",
    "One thing I recommend is downloading the Anaconda distribution for python 2.7 from this <a href=\"https://www.continuum.io/downloads\">link</a>. This distribution wraps python with the necessary packages used in data science like Numpy, Pandas, Scipy or Scikit-learn.\n",
    "\n",
    "For the purpose of this tutorial we'll also have to download external packages:\n",
    "\n",
    "- tqdm (a progress bar python utility): \n",
    "pip install tqdm \n",
    "- nltk (for natural language processing): \n",
    "conda install -c anaconda nltk=3.2.2 \n",
    "- bokeh (for interactive data viz): \n",
    "conda install bokeh\n",
    "- gensim: \n",
    "pip install --upgrade gensim\n",
    "- pyldavis (python package to visualize lda topics): \n",
    "pip install pyldavis \n",
    "\n",
    "\n",
    "To connect to the Newsapi service you'll have to create an account at https://newsapi.org/register to get a key. It's totally free. Then you'll have to put your key in the code and run the script on your own if you want to. \n",
    "\n",
    "All the data and scripts to run this tutorial can be found on my <a href=\"https://github.com/ahmedbesbes/How-to-mine-newsfeed-data-and-extract-interactive-insights-in-Python\"> github</a>. Don't hesitate to fork the project and bring your modifications to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - Data acquisition from Newsapi.org"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/newsapi.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, this looks like something very handy. This avoids the tedious data scrapping that you would perform on each site separately.\n",
    "\n",
    "Getting the latest news for a specific source like Techcrunch is as simple as sending a get request to this address:\n",
    "https://newsapi.org/v1/articles?source=techcrunch&apiKey={API_KEY}\n",
    "\n",
    "The JSON file resulting from this response is pretty straightforward:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "{\n",
    "\"status\": \"ok\",\n",
    "\"source\": \"techcrunch\",\n",
    "\"sortBy\": \"top\",\n",
    "-\"articles\": [\n",
    "-{\n",
    "\"author\": \"Khaled \\\"Tito\\\" Hamze\",\n",
    "\"title\": \"Crunch Report\",\n",
    "\"description\": \"Your daily roundup of the biggest TechCrunch stories and startup news.\",\n",
    "\"url\": \"https://techcrunch.com/video/crunchreport/\",\n",
    "\"urlToImage\": \"https://tctechcrunch2011.files.wordpress.com/2015/03/tccrshowogo.jpg?w=500&h=200&crop=1\",\n",
    "\"publishedAt\": \"2017-03-02T04:40:50Z\"\n",
    "},\n",
    "-{\n",
    "\"author\": \"Kate Conger, Devin Coldewey\",\n",
    "\"title\": \"Marissa Mayer forgoes bonus and equity in wake of Yahoo security incidents\",\n",
    "\"description\": \"Yahoo’s board has decided that CEO Marissa Mayer will not receive her annual bonus this year, a decision linked to Yahoo’s handling of the 2014 security..\",\n",
    "\"url\": \"https://techcrunch.com/2017/03/01/marissa-mayer-forgoes-bonus-and-equity-in-wake-of-yahoo-security-incidents/\",\n",
    "\"urlToImage\": \"https://tctechcrunch2011.files.wordpress.com/2014/05/marissa-mayer3.jpg?w=764&h=400&crop=1\",\n",
    "\"publishedAt\": \"2017-03-01T22:20:38Z\"\n",
    "},\n",
    "-{\n",
    "\"author\": \"Matthew Lynley\",\n",
    "\"title\": \"Snap values itself at nearly $24B with its IPO pricing\",\n",
    "\"description\": \"Snap has given a final price for its IPO, setting the company's valuation at nearly $24 billion with a price of $17 per share, according to a report by The..\",\n",
    "\"url\": \"https://techcrunch.com/2017/03/01/snap-values-itself-at-nearly-24b-with-its-ipo-pricing/\",\n",
    "\"urlToImage\": \"https://tctechcrunch2011.files.wordpress.com/2016/12/8a82586a123a7429edd0ca2f65ddbeda.jpg?w=764&h=400&crop=1\",\n",
    "\"publishedAt\": \"2017-03-01T19:49:15Z\"\n",
    "},\n",
    "-{\n",
    "\"author\": \"Lucas Matney\",\n",
    "\"title\": \"Oculus co-founder talks new Rift pricing, Touch adoption and possible Mac support\",\n",
    "\"description\": \"While so many virtual reality hardware companies have been tasked only with selling their own product, Oculus has had the intense challenge of building the..\",\n",
    "\"url\": \"https://techcrunch.com/2017/03/01/oculus-co-founder-talks-new-rift-pricing-touch-adoption-and-possible-mac-support/\",\n",
    "\"urlToImage\": \"https://tctechcrunch2011.files.wordpress.com/2017/03/2016-09-14_oculus_134750-1-0022.jpg?w=764&h=400&crop=1\",\n",
    "\"publishedAt\": \"2017-03-01T18:56:02Z\"\n",
    "},\n",
    "-{\n",
    "\"author\": \"Matthew Lynley\",\n",
    "\"title\": \"Five burning questions that Snap’s IPO is about to answer\",\n",
    "\"description\": \"Snap will begin publicly trading tomorrow, which means that it will officially give a price for its shares in its initial public offering this evening...\",\n",
    "\"url\": \"https://techcrunch.com/2017/03/01/five-burning-questions-that-snaps-ipo-is-about-to-answer/\",\n",
    "\"urlToImage\": \"https://tctechcrunch2011.files.wordpress.com/2017/02/gettyimages-616058338.jpg?w=764&h=400&crop=1\",\n",
    "\"publishedAt\": \"2017-03-01T17:55:28Z\"\n",
    "}\n",
    "]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **articles** object is a list of JSON files corresponding to the latest published articles. As you can see, we can not go far into the historical data to extract a large dump of articles.\n",
    "\n",
    "One solution I came up with to get a large set of news articles was to request the address above for every source at every 5 minutes for a long period of time. As for now, the script has been running for more than two weeks.\n",
    "\n",
    "Let's get into the code to see how to manage this data acquisition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import packages\n",
    "from __future__ import print_function\n",
    "import requests\n",
    "import pandas as pd\n",
    "from datetime import datetime\n",
    "from tqdm import tqdm\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this post, we'll be analyzing english news sources only. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def getSources():\n",
    "    source_url = 'https://newsapi.org/v1/sources?language=en'\n",
    "    response = requests.get(source_url).json()\n",
    "    sources = []\n",
    "    for source in response['sources']:\n",
    "        sources.append(source['id'])\n",
    "    return sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sources = getSources()\n",
    "print('number of sources :', len(sources))\n",
    "print('sources :', ', '.join(sources))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Newsapi allows you to map each data source to its category. Let's use this information as an additional feature in our dataset. This may be useful later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def mapping():\n",
    "    d = {}\n",
    "    response = requests.get('https://newsapi.org/v1/sources?language=en')\n",
    "    response = response.json()\n",
    "    for s in response['sources']:\n",
    "        d[s['id']] = s['category']\n",
    "    return d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the category of reuters and techcrunch for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = mapping()\n",
    "print('category of reuters:', m['reuters'])\n",
    "print('category of techcrunch:', m['techcrunch'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "let's see what categories we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('categories:', list(set(m.values())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main function is getDailyNews. It will loop on each news source, request the api, extract the data and dump it to a pandas DataFrame and then export the result into csv file.\n",
    "\n",
    "On each iteration of the loop, the csv file is updated and cleaned. Redundant lines are removed. This is handled by cleanData function.\n",
    "\n",
    "For each article we'll collect these fields:\n",
    "\n",
    "- author\n",
    "- title\n",
    "- description\n",
    "- url\n",
    "- urlToImage\n",
    "- publishedAt\n",
    "\n",
    "And add two other features:\n",
    "- category\n",
    "- scraping_date : the time at which the script runs. This will help us track the data.\n",
    "\n",
    "Here is the complete script:\n",
    "\n",
    "PS: since the newsapi.org has been modified, you may encounter rate limit errors. This happens when you exceed 250 requests within 6 hours. What you could do is either wait for 6 hours or sign in different accounts and launch many instances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Full script:\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "from datetime import datetime\n",
    "from tqdm import tqdm, tqdm_notebook\n",
    "\n",
    "def getSources():\n",
    "    source_url = 'https://newsapi.org/v1/sources?language=en'\n",
    "    response = requests.get(source_url).json()\n",
    "    sources = []\n",
    "    for source in response['sources']:\n",
    "        sources.append(source['id'])\n",
    "    return sources\n",
    "\n",
    "def mapping():\n",
    "    d = {}\n",
    "    response = requests.get('https://newsapi.org/v1/sources?language=en')\n",
    "    response = response.json()\n",
    "    for s in response['sources']:\n",
    "        d[s['id']] = s['category']\n",
    "    return d\n",
    "\n",
    "def category(source, m):\n",
    "    try:\n",
    "        return m[source]\n",
    "    except:\n",
    "        return 'NC'\n",
    "\n",
    "def getDailyNews():\n",
    "    sources = getSources()\n",
    "    key = 'c12a5c07a7bd42edbf54d59aca007a54'\n",
    "    url = 'https://newsapi.org/v1/articles?source={0}&sortBy={1}&apiKey={2}'\n",
    "    responses = []\n",
    "    for i, source in tqdm_notebook(enumerate(sources), total=len(sources)):\n",
    "        \n",
    "        try:\n",
    "            u = url.format(source, 'top', key)\n",
    "        except:\n",
    "            u = url.format(source, 'latest', key)\n",
    "        \n",
    "        response = requests.get(u)\n",
    "        r = response.json()\n",
    "        try:\n",
    "            for article in r['articles']:\n",
    "                article['source'] = source\n",
    "            responses.append(r)\n",
    "        except:\n",
    "            print('Rate limit exceeded ... please wait and retry in 6 hours')\n",
    "            return None\n",
    "                \n",
    "    articles = list(map(lambda r: r['articles'], responses))\n",
    "    articles = list(reduce(lambda x,y: x+y, articles))\n",
    "    \n",
    "    news = pd.DataFrame(articles)\n",
    "    news = news.dropna()\n",
    "    news = news.drop_duplicates()\n",
    "    news.reset_index(inplace=True, drop=True)\n",
    "    d = mapping()\n",
    "    news['category'] = news['source'].map(lambda s: category(s, d))\n",
    "    news['scraping_date'] = datetime.now()\n",
    "\n",
    "    try:\n",
    "        aux = pd.read_csv('./data/news.csv')\n",
    "        aux = aux.append(news)\n",
    "        aux = aux.drop_duplicates('url')\n",
    "        aux.reset_index(inplace=True, drop=True)\n",
    "        aux.to_csv('./data/news.csv', encoding='utf-8', index=False)\n",
    "    except:\n",
    "        news.to_csv('./data/news.csv', index=False, encoding='utf-8')\n",
    "        \n",
    "    print('Done')\n",
    "    \n",
    "if __name__=='__main__':\n",
    "    getDailyNews()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now this script needs to run repetitively to collect the data. \n",
    "\n",
    "To do this:\n",
    "\n",
    "I uploaded the script to my linux server at this path /home/news/news.py. \n",
    "\n",
    "Then I created a crontab schedule to tell my server to run news.py every 5 minutes (back when there were no rate limits, but you could tweak it to make it run every 6 hours). To do this: \n",
    " - from the terminal, type crontab -e to edit the crontab file\n",
    " - add this line to the end of the file using nano or vim (put absolute paths for your executables): \n",
    "\n",
    "\\*/5 \\* \\* \\* \\* /root/anaconda2/bin/python /home/article_2/news.py\n",
    "\n",
    "    \n",
    "  What this command tells the server is: \"for every 5 minutes (\\*/5) of every hour (\\*) of every day of the month (\\*) of every month (\\*) and whatever the day of the week (\\*), run the news.py script.\n",
    "    \n",
    " - give your script the execution permission. Otherwise, this won't work. To do this, run:\n",
    "\n",
    "chmod +x news.py "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the data has been collected, we will start anlayzing it :\n",
    "\n",
    "- We'll have a look at the dataset and inspect it\n",
    "- We'll apply some preoprocessings on the texts: tokenization, tf-idf\n",
    "- We'll cluster the articles using two different algorithms (Kmeans and LDA)\n",
    "- We'll visualize the clusters using Bokeh and pyldavis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Data analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 - 1 - Data discovery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.options.display.max_columns = 200\n",
    "pd.options.mode.chained_assignment = None\n",
    "\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "stop = set(stopwords.words('english'))\n",
    "from string import punctuation\n",
    "\n",
    "from collections import Counter\n",
    "import re\n",
    "import numpy as np\n",
    "\n",
    "from tqdm import tqdm_notebook\n",
    "tqdm_notebook().pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('./data/news.csv')\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is now ingested in a Pandas DataFrame.\n",
    "\n",
    "Let's see what it looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's cool to have all these features. In this article, we will be mainly focusing on the description column.\n",
    "\n",
    "Let's check the distribution of the different categories across the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "data.category.value_counts(normalize=True).plot(kind='bar', grid=True, figsize=(16, 9))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/categories.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many mixed topics are included in the \"general\" category.\n",
    "\n",
    "This gives us a very superficial classificaion of the news. It doesn't tell us the underlying topics, nor the keywords and and the most relevant news per each category. All we have is a high level idea.\n",
    "\n",
    "To get that sort of information, we'll have to process the descriptions of each article since these variables naturally carry more meanings.\n",
    "\n",
    "Before doing that, let's focus on the news articles whose descriptions' length is higher than 140 characters (a tweet length). Shorter descriptions happen to introduce lots of noise.\n",
    "\n",
    "Remove duplicate description columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = data.drop_duplicates('description')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove rows with empty descriptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data[~data['description'].isnull()]\n",
    "\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select descriptions of length between 140 and 300 characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data[(data.description.map(len) > 140) & (data.description.map(len) <= 300)]\n",
    "data.reset_index(inplace=True, drop=True)\n",
    "\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now check the disctribution of the descriptions' lengths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data.description.map(len).hist(figsize=(15, 5), bins=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/lengths.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Due to memory constraints on some packages used in this tutorial, we will limit the analysis to 10000 news.\n",
    "\n",
    "Just get some random news:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data.sample(10000, random_state=42)\n",
    "data.reset_index(inplace=True, drop=True)\n",
    "\n",
    "data.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 - 2 - Text processing : tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we start by building a tokenizer. This will, for every description:\n",
    "\n",
    "- lowercase the text and clean it (separate function)\n",
    "- break the descriptions into sentences and then break the sentences into tokens\n",
    "- remove punctuation, stop words\n",
    "\n",
    "I was used to nltk's stopwords list, then I made my own, which is richer. It's in txt file and you can find on my github account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = []\n",
    "\n",
    "f = open('./data/stopwords.txt', 'r')\n",
    "for l in f.readlines():\n",
    "    stop_words.append(l.replace('\\n', ''))\n",
    "    \n",
    "additional_stop_words = ['t', 'will']\n",
    "stop_words += additional_stop_words\n",
    "\n",
    "print(len(stop_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two functions below are borrowed from a <a href=\"https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge\"> Kaggle competition </a> I participated to.\n",
    "\n",
    "They remove non ascii characters and standardize the text (can't -> cannot, i'm -> i am). This will make the tokenization process more efficient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def _removeNonAscii(s): \n",
    "    return \"\".join(i for i in s if ord(i)<128)\n",
    "\n",
    "def clean_text(text):\n",
    "    text = text.lower()\n",
    "    text = re.sub(r\"what's\", \"what is \", text)\n",
    "    text = text.replace('(ap)', '')\n",
    "    text = re.sub(r\"\\'s\", \" is \", text)\n",
    "    text = re.sub(r\"\\'ve\", \" have \", text)\n",
    "    text = re.sub(r\"can't\", \"cannot \", text)\n",
    "    text = re.sub(r\"n't\", \" not \", text)\n",
    "    text = re.sub(r\"i'm\", \"i am \", text)\n",
    "    text = re.sub(r\"\\'re\", \" are \", text)\n",
    "    text = re.sub(r\"\\'d\", \" would \", text)\n",
    "    text = re.sub(r\"\\'ll\", \" will \", text)\n",
    "    text = re.sub(r'\\W+', ' ', text)\n",
    "    text = re.sub(r'\\s+', ' ', text)\n",
    "    text = re.sub(r\"\\\\\", \"\", text)\n",
    "    text = re.sub(r\"\\'\", \"\", text)    \n",
    "    text = re.sub(r\"\\\"\", \"\", text)\n",
    "    text = re.sub('[^a-zA-Z ?!]+', '', text)\n",
    "    text = _removeNonAscii(text)\n",
    "    text = text.strip()\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Integrate these two functions into a tokenizing function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tokenizer(text):\n",
    "    text = clean_text(text)    \n",
    "    tokens = [word_tokenize(sent) for sent in sent_tokenize(text)]\n",
    "    tokens = list(reduce(lambda x,y: x+y, tokens))\n",
    "    tokens = list(filter(lambda token: token not in (stop_words + list(punctuation)) , tokens))\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A new column 'tokens' can be easily created using the map method applied to the 'description' column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['description'] = data['description'].map(lambda d: unicode(d.decode('utf-8')))\n",
    "data['tokens'] = data['description'].progress_map(lambda d: tokenizer(d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tokenizer has been applied to each description through all rows. Each resulting value is then put into the 'tokens' column that is created after the assignment. Let's check what the tokenization looks like for the first 5 descriptions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for descripition, tokens in zip(data['description'].head(5), data['tokens'].head(5)):\n",
    "    print('description:', descripition)\n",
    "    print('tokens:', tokens)\n",
    "    print() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's group the tokens by category, apply a word count and display the top 10 most frequent tokens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def keywords(category):\n",
    "    tokens = data[data['category'] == category]['tokens']\n",
    "    alltokens = []\n",
    "    for token_list in tokens:\n",
    "        alltokens += token_list\n",
    "    counter = Counter(alltokens)\n",
    "    return counter.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for category in set(data['category']):\n",
    "    print('category :', category)\n",
    "    print('top 10 keywords:', keywords(category))\n",
    "    print('---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at these lists, we can formulate some hypotheses:\n",
    "\n",
    "- the sport category deals with NFL i.e *National Football League*. we can also see some sport related terms such as \"win\", \"time\", \"team\" and players.\n",
    "- tech articles refer to Google, Uber, and app\n",
    "- the business news seem to be highly linked to US politics and Donald Trump (this mainly originates from us press)\n",
    "\n",
    "Extracting the top 10 most frequent words per each category is straightforward and can point to important keywords. \n",
    "\n",
    "However, although we did preprocess the descriptions and remove the stop words before, we still end up with words that are very generic (e.g: today, year, people) that don't carry a specific meaning that may describe a topic.\n",
    "\n",
    "As a first approach to prevent this, we will use tf-idf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 - 3 - Text processing : tf-idf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tf-idf stands for term frequencey-inverse document frequency. It's a numerical statistic intended to reflect how important a word is to a document or a corpus (i.e a collection of documents). \n",
    "\n",
    "To relate to this post, words correpond to tokens and documents correpond to descriptions. A corpus is therefore a collection of descriptions.\n",
    "\n",
    "The tf-idf a of a term t in a document d is proportional to the number of times the word t appears in the document d but is also offset by the frequency of the term t in the collection of the documents of the corpus. This helps adjusting the fact that some words appear more frequently in general and don't especially carry a meaning.\n",
    "\n",
    "tf-idf acts therefore as a weighting scheme to extract relevant words in a document.\n",
    "\n",
    "$$tfidf(t,d) = tf(t,d) . idf(t) $$\n",
    "\n",
    "tf(t,d) is the term frequency of t in the document d (i.e. how many times the token t appears in the description d)\n",
    "\n",
    "idf(t) is the inverse document frequency of the term t. it's computed by this formula:\n",
    "\n",
    "$$idf(t) = log(1 + \\frac{1 + n_d}{1 + df(d,t)}) $$\n",
    "\n",
    "- n_d : the number of documents\n",
    "- df(d,t): the number of documents (or descriptions) containing the term t \n",
    "\n",
    "Computing the tfidf matrix is done using the TfidfVectorizer method from scikit-learn. Let's see how to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "vectorizer = TfidfVectorizer(min_df=5, analyzer='word', ngram_range=(1, 2), stop_words='english')\n",
    "vz = vectorizer.fit_transform(list(data['tokens'].map(lambda tokens: ' '.join(tokens))))\n",
    "\n",
    "vz.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "vz is a tfidf matrix. \n",
    "\n",
    "- its number of rows is the total number of documents (descriptions) \n",
    "- its number of columns is the total number of unique terms (tokens) across the documents (descriptions)\n",
    "\n",
    "x_dt  = tfidf(t,d) where x_dt is the element at the index (d,t) in the matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a dictionary mapping the tokens to their tfidf values "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tfidf = dict(zip(vectorizer.get_feature_names(), vectorizer.idf_))\n",
    "tfidf = pd.DataFrame(columns=['tfidf']).from_dict(dict(tfidf), orient='index')\n",
    "tfidf.columns = ['tfidf']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the distribution of the tfidf scores through an histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tfidf.tfidf.hist(bins=25, figsize=(15,7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/tfidf.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's display the 30 tokens that have the lowest tfidf scores "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "\n",
    "def plot_word_cloud(terms):\n",
    "    text = terms.index\n",
    "    text = ' '.join(list(text))\n",
    "    # lower max_font_size\n",
    "    wordcloud = WordCloud(max_font_size=40).generate(text)\n",
    "    plt.figure(figsize=(25, 25))\n",
    "    plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "    plt.axis(\"off\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "plot_word_cloud(tfidf.sort_values(by=['tfidf'], ascending=True).head(40))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/word_cloud_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not surprisingly, we end up with a list of very generic words. These are very common across many descriptions. tfidf attributes a low score to them as a penalty for not being relevant. Words likes tuesday, friday, day, time, etc...\n",
    "\n",
    "You may also notice that Trump, Donald, and president are part of this list for being mentioned in many articles.\n",
    "\n",
    "Now let's check out the 30 words with the highest tfidf scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plot_word_cloud(tfidf.sort_values(by=['tfidf'], ascending=False).head(40))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/word_cloud_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We end up with less common words. These words naturally carry more meaning for the given description and may outline the underlying topic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you've noticed, the documents have more than 7000 features (see the vz shape). put differently, each document has more than 7000 dimensions.\n",
    "\n",
    "If we want to plot them like we usually do with geometric objects, we need to reduce their dimension to 2 or 3 depending on whether we want to display on a 2D plane or on a 3D space. This is what we call dimensionality reduction.\n",
    "\n",
    "To perform this task, we'll be using a combination of two popular techniques: Singular Value Decomposition (SVD) to reduce the dimension to 50 and then t-SNE to reduce the dimension from 50 to 2. t-SNE is more suitable for dimensionality reduction to 2 or 3.\n",
    "\n",
    "Let's start reducing the dimension of each vector to 50 by SVD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "svd = TruncatedSVD(n_components=50, random_state=0)\n",
    "svd_tfidf = svd.fit_transform(vz)\n",
    "\n",
    "svd_tfidf.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bingo. Now let's do better. From 50 to 2!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "run = False\n",
    "if run:\n",
    "# run this (takes times)\n",
    "    from sklearn.manifold import TSNE\n",
    "    tsne_model = TSNE(n_components=2, verbose=1, random_state=0, n_iter=500)\n",
    "    tsne_tfidf = tsne_model.fit_transform(svd_tfidf)\n",
    "    print(tsne_tfidf.shape)\n",
    "    tsne_tfidf_df = pd.DataFrame(tsne_tfidf)\n",
    "    tsne_tfidf_df.columns = ['x', 'y']\n",
    "    tsne_tfidf_df['category'] = data['category']\n",
    "    tsne_tfidf_df['description'] = data['description']\n",
    "    tsne_tfidf_df.to_csv('./data/tsne_tfidf.csv', encoding='utf-8', index=False)\n",
    "else:\n",
    "# or import the dataset directly\n",
    "    tsne_tfidf_df = pd.read_csv('./data/tsne_tfidf.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each description is now modeled by a two dimensional vector. \n",
    "\n",
    "Let's see what tsne_idf looks like on a 2D plane."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "groups = tsne_tfidf_df.groupby('category')\n",
    "fig, ax = plt.subplots(figsize=(15, 10))\n",
    "ax.margins(0.05) # Optional, just adds 5% padding to the autoscaling\n",
    "for name, group in groups:\n",
    "    ax.plot(group.x, group.y, marker='o', linestyle='', label=name)\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/tfidf_tsne.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're having two float numbers per discription. This is not interpretable at first sight. \n",
    "\n",
    "What we need to do is find a way to display these points on a plot and also attribute the corresponding description to each point.\n",
    "\n",
    "matplotlib is a very good python visualization libaray. However, we cannot easily use it to display our data since we need interactivity on the objects. One other solution could be d3.js that provides huge capabilities in this field. \n",
    "\n",
    "Right now I'm choosing to stick to python so I found a tradeoff : it's called **Bokeh**.\n",
    "\n",
    ">\"Bokeh is a Python interactive visualization library that targets modern web browsers for presentation. Its goal is to provide elegant, concise construction of novel graphics in the style of D3.js, and to extend this capability with high-performance interactivity over very large or streaming datasets. Bokeh can help anyone who would like to quickly and easily create interactive plots, dashboards, and data applications.\" To know more, please refer to this <a href=\"http://bokeh.pydata.org/en/latest/\"> link </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by importing bokeh packages and initializing the plot figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import bokeh.plotting as bp\n",
    "from bokeh.models import HoverTool, BoxSelectTool\n",
    "from bokeh.plotting import figure, show, output_notebook, reset_output\n",
    "from bokeh.palettes import d3\n",
    "import bokeh.models as bmo\n",
    "from bokeh.io import save, output_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bokeh need a pandas dataframe to be passed as a source data. this is a very elegant way to read data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_notebook()\n",
    "plot_tfidf = bp.figure(plot_width=700, plot_height=600, title=\"tf-idf clustering of the news\",\n",
    "    tools=\"pan,wheel_zoom,box_zoom,reset,hover,previewsave\",\n",
    "    x_axis_type=None, y_axis_type=None, min_border=1)\n",
    "\n",
    "palette = d3['Category10'][len(tsne_tfidf_df['category'].unique())]\n",
    "color_map = bmo.CategoricalColorMapper(factors=tsne_tfidf_df['category'].map(str).unique(), palette=palette)\n",
    "\n",
    "plot_tfidf.scatter(x='x', y='y', color={'field': 'category', 'transform': color_map}, \n",
    "                   legend='category', source=tsne_tfidf_df)\n",
    "hover = plot_tfidf.select(dict(type=HoverTool))\n",
    "hover.tooltips={\"description\": \"@description\", \"category\":\"@category\"}\n",
    "\n",
    "show(plot_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, what do we see here? \n",
    "\n",
    "- First of all, a 2D projection of a high dimensional data. Pretty hein? Actually T-SNE does a  good job when it comes in dimensionality reduction and aesthetic plots. \n",
    "\n",
    "- Each dot is a document and the color encodes its category: notice that documents belonging to the general category (orange) are scattered all over the plot while those belonging to sport (red) are close to each other.\n",
    "\n",
    "Bokeh charts offer many functionalities:\n",
    "\n",
    "- navigating in the data\n",
    "- zooming\n",
    "- hovering on each data point and displaying the corresponding description\n",
    "- saving the chart\n",
    "\n",
    "When the description popup doesn't show properly you have to move the data point slightly on the left.\n",
    "\n",
    "By hovering on each news cluster, we can see groups of descriptions of similar keywords and thus referring to the same topic.\n",
    "\n",
    "Now we're going to use clustering algorithms on the tf-idf matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 - Clustering \n",
    "### 4 - 1 - KMeans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our starting point is the tf-idf matrix vz. Let's check its size again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This matrix can be seen as a collection of (x) high-dimensional vectors (y). Some algorithms like K-means can crunch this data structure and produce blocks of similar or \"close\" data points based on some similarity measure like the euclidean distance.\n",
    "\n",
    "Kmeans needs the number of cluster as parameter. This number is usually determined by trying out different values until the result satisfies a given metrics (silhouette score or distorsion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "from sklearn.cluster import MiniBatchKMeans, KMeans\n",
    "from sklearn.metrics import silhouette_score\n",
    "\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import Normalizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can pick the number of clusters by varying it in a range of possible values and compute the silhouette score and the distorsion at every iteration. The optimal number is the one that maximizes the first and minimizes the second."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "distorsions = []\n",
    "sil_scores = []\n",
    "k_max = 80\n",
    "for k in tqdm_notebook(range(2, k_max)):\n",
    "    kmeans_model = MiniBatchKMeans(n_clusters=k, init='k-means++', n_init=1, random_state=42,  \n",
    "                         init_size=1000, verbose=False, max_iter=1000)\n",
    "    kmeans_model.fit(vz)\n",
    "    sil_score = silhouette_score(vz, kmeans_model.labels_)\n",
    "    sil_scores.append(sil_score)\n",
    "    distorsions.append(kmeans_model.inertia_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "f, (ax1, ax2) = plt.subplots(2, 1, sharex=True, figsize=(15, 10))\n",
    "\n",
    "ax1.plot(range(2, k_max), distorsions)\n",
    "ax1.set_title('Distorsion vs num of clusters')\n",
    "ax1.grid(True)\n",
    "\n",
    "ax2.plot(range(2, k_max), sil_scores)\n",
    "ax2.set_title('Silhouette score vs num of clusters')\n",
    "ax2.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/silhouette_score_and_distorsion_vs_num_clusters.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So how to select the optimal number of clusters from these two plots? this is tricky. \n",
    "A small number of cluster may not capture enough topics while a large number (that can although maximize the silhouette score) may induce noise. \n",
    "\n",
    "I chose 40 for two reasons:\n",
    "\n",
    "- when you smooth the silhouette score and ignore the values before k = 10 you notice a peak around this value \n",
    "- the elbow curve (distorsion) starts to plateau at 40 clusters\n",
    "\n",
    "40 clusters. This basically means 40 main topics in a two-week period. Given that we have 8 categories of news, this results in an average number of 5 topics per category. \n",
    "\n",
    "Of course you'll have much more topics in reality. But these topics we're detecting are the most important. The recurrent ones, the ones that have been relayed many times by the media."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_clusters = 40\n",
    "kmeans_model = MiniBatchKMeans(n_clusters=num_clusters, init='k-means++', n_init=1, random_state=42,                       \n",
    "                         init_size=1000, batch_size=1000, verbose=False, max_iter=1000, )\n",
    "kmeans = kmeans_model.fit(vz)\n",
    "kmeans_clusters = kmeans.predict(vz)\n",
    "kmeans_distances = kmeans.transform(vz)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the five first description and the associated cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for (i, desc),category in zip(enumerate(data.description),data['category']):\n",
    "    if(i < 5):\n",
    "        print(\"Cluster \" + str(kmeans_clusters[i]) + \": \" + desc + \n",
    "              \"(distance: \" + str(kmeans_distances[i][kmeans_clusters[i]]) + \")\")\n",
    "        print('category: ',category)\n",
    "        print('---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This doesn't tell us much. What we need to look up are the \"hot\" keywords that describe each clusters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_centroids = kmeans.cluster_centers_.argsort()[:, ::-1]\n",
    "terms = vectorizer.get_feature_names()\n",
    "all_keywords = []\n",
    "for i in range(num_clusters):\n",
    "    topic_keywords = []\n",
    "    for j in sorted_centroids[i, :10]:\n",
    "        topic_keywords.append(terms[j])\n",
    "    all_keywords.append(topic_keywords)\n",
    "\n",
    "keywords_df = pd.DataFrame(index=['topic_{0}'.format(i) for i in range(num_clusters)], \n",
    "                           columns=['keyword_{0}'.format(i) for i in range(10)],\n",
    "                           data=all_keywords)\n",
    "keywords_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at some topics, and *try* to summarize them in one sentence based on their keywords:\n",
    "\n",
    "- topic 0: A spoiler alert about movies (ex.  the walking dead)\n",
    "- topic 1: The super bowl event\n",
    "- topic 5: Obamacare plan to provide affordable healthcare\n",
    "- topic 7: British Football (Manchester United) and Zlatan\n",
    "- topic 18: La La Land movie release and the Oscar ceremony\n",
    "- topic 30: French political news about François Fillion (ex. french minister) and the presidential elections\n",
    "- topic 33: Oscar nominations\n",
    "- topic 36: News about the Brexit\n",
    "\n",
    "Looking at these clusters you can roughly have an idea of what's going on.\n",
    "\n",
    "Let's plot them, to have an interactive view."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "run = False\n",
    "if run:\n",
    "    tsne_model = TSNE(n_components=2, verbose=1, random_state=0, n_iter=500)\n",
    "    tsne_kmeans = tsne_model.fit_transform(svd_kmeans)\n",
    "    kmeans_df = pd.DataFrame(tsne_kmeans, columns=['x', 'y'])\n",
    "    kmeans_df['cluster'] = kmeans_clusters\n",
    "    kmeans_df['cluster'] = kmeans_df['cluster'].map(str)\n",
    "    kmeans_df['description'] = data['description']\n",
    "    kmeans_df['category'] = data['category']\n",
    "    kmeans_df.to_csv('./data/tsne_kmeans.csv', index=False, encoding='utf-8')\n",
    "else:\n",
    "    kmeans_df = pd.read_csv('./data/tsne_kmeans.csv')\n",
    "    kmeans_df['cluster'] = kmeans_df['cluster'].map(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_output()\n",
    "output_notebook()\n",
    "plot_kmeans = bp.figure(plot_width=700, plot_height=600, title=\"KMeans clustering of the news\",\n",
    "    tools=\"pan,wheel_zoom,box_zoom,reset,hover,previewsave\",\n",
    "    x_axis_type=None, y_axis_type=None, min_border=1)\n",
    "\n",
    "palette = d3['Category20'][20] + d3['Category20b'][20] \n",
    "color_map = bmo.CategoricalColorMapper(factors=kmeans_df['cluster'].unique(), palette=palette)\n",
    "\n",
    "plot_kmeans.scatter('x', 'y', source=kmeans_df, \n",
    "                    color={'field': 'cluster', 'transform': color_map}, \n",
    "                    legend='cluster')\n",
    "hover = plot_kmeans.select(dict(type=HoverTool))\n",
    "hover.tooltips={\"description\": \"@description\", \"cluster\": \"@cluster\", \"category\": \"@category\"}\n",
    "\n",
    "show(plot_kmeans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks that there is a dominant cluster scattered all over the space: this is mainly due to the general category of news. \n",
    "\n",
    "By hovering on each point you can see the corresponding description. At first sight you could notice that they deal approximately with the same topic. This is coherent since we build our clusters using similarities between relevant keywords.\n",
    "\n",
    "You can look up a topic in the dataframe above (based on its keywords), and switch to this plot and see the corresponding articles.\n",
    "\n",
    "Kmeans separates the documents into disjoint clusters. the assumption is that each cluster is attributed a single topic.\n",
    "\n",
    "However, descriptions may in reality be characterized by a \"mixture\" of topics. For example, let's take an article that deals with the hearing that Zuckerberg had in front of the congress: you'll obviously have different topics rising based on the keywords: Privacy, Technology, Facebook app, data, etc.\n",
    "\n",
    "We'll cover how to deal with this problem with the LDA algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 - 2 - 1 - Latent Dirichlet Allocation (with Bokeh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part, we'll address the topic modeling with another method.\n",
    "\n",
    "We'll use a well-known algorithm called **LDA**, which stands for Latent Dirichlet Allocation.\n",
    "\n",
    "LDA considers two things:\n",
    "\n",
    "- Each document in a corpus is a weighted combination of several topics (doc1-> 0.3 * politics + 0.2 * science + 0.05 * religion, etc.)\n",
    "- Each topic has its collection of representative keywords (politics -> ['obama', 'white_house', 'war', ...])\n",
    "\n",
    "Ultimately, these are two probability distributions that the algorithm tries to approximate, starting from a random initialization until convergence:\n",
    "\n",
    "1. For a given document, what is the disctribution of topics that describe it?\n",
    "2. For a given topic, what is the distribution of its words. put differently, what is the importance (probability) of each word in defining the topic nature? \n",
    "\n",
    "The main hyperparameter that we have to correctly in LDA models set is the number of topics. Just like we did with Kmeans, we'll show how to select this optimal number using some metrics.\n",
    "\n",
    "Given the number of topics, LDA starts shuffling the topic distribution in each document and the word distribution in each topic until the final results shows a high segregation of topics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/lda.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform LDA, we'll use gensim."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "warnings.filterwarnings(action='ignore', category=UserWarning, module='gensim')\n",
    "\n",
    "import gensim\n",
    "import gensim.corpora as corpora\n",
    "from gensim import matutils\n",
    "from gensim.models import CoherenceModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a corpus and a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aux = data.copy()\n",
    "\n",
    "bigram = gensim.models.Phrases(aux['tokens'], min_count=5, threshold=100)\n",
    "bigram_mod = gensim.models.phrases.Phraser(bigram)\n",
    "aux['tokens_bigram'] = aux['tokens'].progress_map(lambda tokens: bigram_mod[tokens])\n",
    "\n",
    "id2word = corpora.Dictionary(aux['tokens_bigram'])\n",
    "texts = aux['tokens_bigram'].values\n",
    "corpus = [id2word.doc2bow(text) for text in texts]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the LDA model in function that takes the number of topics as a parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def LDA_model(num_topics, passes=1):\n",
    "    return gensim.models.ldamodel.LdaModel(corpus=tqdm_notebook(corpus, leave=False),\n",
    "                                               id2word=id2word,\n",
    "                                               num_topics=num_topics, \n",
    "                                               random_state=100,\n",
    "                                               eval_every=10,\n",
    "                                               chunksize=2000,\n",
    "                                               passes=passes,\n",
    "                                               per_word_topics=True\n",
    "                                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a metric to assess a good topic model: the coherence score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_coherence(model):\n",
    "    coherence = CoherenceModel(model=model, \n",
    "                           texts=aux['tokens_bigram'].values,\n",
    "                           dictionary=id2word, coherence='c_v')\n",
    "    return coherence.get_coherence()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function to display topics and corresponding keywords:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def display_topics(model):\n",
    "    topics = model.show_topics(num_topics=model.num_topics, formatted=False, num_words=10)\n",
    "    topics = map(lambda c: map(lambda cc: cc[0], c[1]), topics)\n",
    "    df = pd.DataFrame(topics)\n",
    "    df.index = ['topic_{0}'.format(i) for i in range(model.num_topics)]\n",
    "    df.columns = ['keyword_{0}'.format(i) for i in range(1, 10+1)]\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like Kmeans, we vary the number of topics in an interval and pick the number that optimizes the coherence score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def explore_models(df, rg=range(5, 25)):\n",
    "    id2word = corpora.Dictionary(df['tokens_bigram'])\n",
    "    texts = df['tokens_bigram'].values\n",
    "    corpus = [id2word.doc2bow(text) for text in texts]\n",
    "\n",
    "    models = []\n",
    "    coherences = []\n",
    "    \n",
    "    for num_topics in tqdm_notebook(rg, leave=False):\n",
    "        lda_model = LDA_model(num_topics, passes=5)\n",
    "        models.append(lda_model)\n",
    "        coherence = compute_coherence(lda_model)\n",
    "        coherences.append(coherence)\n",
    "      \n",
    "\n",
    "    fig = plt.figure(figsize=(15, 5))\n",
    "    plt.title('Choosing the optimal number of topics')\n",
    "    plt.xlabel('Number of topics')\n",
    "    plt.ylabel('Coherence')\n",
    "    plt.grid(True)\n",
    "    plt.plot(rg, coherences)\n",
    "    \n",
    "    return coherences, models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "coherences, models = explore_models(aux, rg=range(5, 85, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/coherence_vs_number_of_topics.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are quite consistent with the previous approach. 40 seems to be a good number of topics, since it's a value after which the coherence stops increasing rapidly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_model = LDA_model(num_topics=40, passes=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_topics(model=best_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now let's build a document/topic matrix, that we'll use for our Bokeh plot after a TSNE.\n",
    "\n",
    "A cell i,j is the probabily of topic j in the document i."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_document_topic_matrix(corpus, num_topics=best_model.num_topics):\n",
    "    matrix = []\n",
    "    for row in tqdm_notebook(corpus):\n",
    "        output = np.zeros(num_topics)\n",
    "        doc_proba = best_model[row][0]\n",
    "        for doc, proba in doc_proba:\n",
    "            output[doc] = proba\n",
    "        matrix.append(output)\n",
    "    matrix = np.array(matrix)\n",
    "    return matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = get_document_topic_matrix(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LDA outputs a distribution of topic for each document. We'll assume that a document's topic is the one with the highest probability. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doc_topic = best_model.get_document_topics(corpus)\n",
    "lda_keys = []\n",
    "for i, desc in enumerate(data['description']):\n",
    "    lda_keys.append(np.argmax(matrix[i, :]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "run = False\n",
    "if run: \n",
    "    tsne_model = TSNE(n_components=2, verbose=1, random_state=0, n_iter=500)\n",
    "    tsne_lda = tsne_model.fit_transform(matrix)\n",
    "    lda_df = pd.DataFrame(tsne_lda, columns=['x', 'y'])\n",
    "    lda_df['topic'] = lda_keys\n",
    "    lda_df['topic'] = lda_df['topic'].map(str)\n",
    "    lda_df['description'] = data['description']\n",
    "    lda_df['category'] = data['category']\n",
    "    lda_df.to_csv('./data/tsne_lda.csv', index=False, encoding='utf-8')\n",
    "else:\n",
    "    lda_df = pd.read_csv('./data/tsne_lda.csv')\n",
    "    lda_df['topic'] = lda_df['topic'].map(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_output()\n",
    "output_notebook()\n",
    "plot_lda = bp.figure(plot_width=700, plot_height=600, title=\"KMeans clustering of the news\",\n",
    "    tools=\"pan,wheel_zoom,box_zoom,reset,hover,previewsave\",\n",
    "    x_axis_type=None, y_axis_type=None, min_border=1)\n",
    "\n",
    "palette = d3['Category20'][20] + d3['Category20b'][20] \n",
    "color_map = bmo.CategoricalColorMapper(factors=lda_df['topic'].unique(), palette=palette)\n",
    "\n",
    "plot_lda.scatter('x', 'y', source=lda_df, \n",
    "                    color={'field': 'topic', 'transform': color_map}, \n",
    "                    legend='topic')\n",
    "hover = plot_lda.select(dict(type=HoverTool))\n",
    "hover.tooltips={\"description\": \"@description\", \"topic\": \"@topic\", \"category\": \"@category\"}\n",
    "\n",
    "show(plot_lda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 - 2 - 2 Visualization of the topics using pyLDAvis\n",
    "\n",
    "\n",
    "We plotted LDA just like we did with Kmeans. But it's absurd, if you think about it. Why? Because we lost information. \n",
    "\n",
    "We assumed that a document's topic is the one that has the highest probability. We don't talk about the topic mixture (or distribution) anymore.\n",
    "\n",
    "Hopefully pyLDAvis is a visualization package that'll help us solve this problem!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](images/article_2/pyldavis.PNG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pyLDAvis\n",
    "import pyLDAvis.gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pyLDAvis.enable_notebook()\n",
    "panel = pyLDAvis.gensim.prepare(best_model, corpus, id2word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pyLDAvis.save_html(panel, './plots/pyLDAvis.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " **Demo ** : <a href=\"https://ahmedbesbes.com/pyldavis.html\">  here </a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [BONUS] NMF: Non-negative Matrix Factorization\n",
    "\n",
    "This is a small section where I show that a popular technique based on linear algebra can work as well for topic modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import NMF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vectorizer = TfidfVectorizer(min_df=5, analyzer='word', ngram_range=(1, 2), stop_words='english')\n",
    "vz = vectorizer.fit_transform(list(data['tokens'].map(lambda tokens: ' '.join(tokens))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nmf = NMF(n_components=40, random_state=1, alpha=.1, l1_ratio=.5, init='nndsvd').fit(vz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = vectorizer.get_feature_names()\n",
    "no_top_words = 10\n",
    "\n",
    "for topic_idx, topic in enumerate(nmf.components_[:10]):\n",
    "    print(\"Topic %d:\"% (topic_idx))\n",
    "    print(\" | \".join([feature_names[i]\n",
    "                    for i in topic.argsort()[:-no_top_words - 1:-1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 - Conclusion\n",
    "\n",
    "In this post we explored many topics. \n",
    "\n",
    "- We set up a script to automatically extract newsfeed data from a REST API called newsapi.\n",
    "- We processed the raw text by using different tools (pandas, nltk, scikit-learn)\n",
    "- We applied tf-idf statistics as a natural language preprocessing technique\n",
    "- We created clusters on top of the tf-idf matrix using the KMeans algorithm and visualized them using Bokeh\n",
    "- We extracted topics using the Latent Dirichlet Allocation algorithm and visualized them using Bokeh and pyldavis\n",
    "\n",
    "Different techniques have been used but I'm pretty sure there's plenty of better methods. In fact, one way to extend this tutorial could be to dive in: \n",
    "\n",
    "- word2vec and doc2vec to model the topics\n",
    "- setting up a robust way to select the number of clusters/topics up front\n",
    "\n",
    "Thanks for reading ! Don't hesitate to comment if you have a suggestion or an improvement. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6 - References\n",
    "\n",
    "- https://newsapi.org/\n",
    "- http://pythonhosted.org/lda/\n",
    "- <a href=\"http://nbviewer.jupyter.org/github/skipgram/modern-nlp-in-python/blob/master/executable/Modern_NLP_in_Python.ipynb#topic=3&lambda=0.87&term=\">NLP post </a>\n",
    "- <a href=\"https://www.machinelearningplus.com/nlp/topic-modeling-python-sklearn-examples/\"> Topic Modeling python - sklearn</a>\n",
    "- <a href=\"https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/\"> Topic Modeling python - gensim</a>"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:Anaconda2]",
   "language": "python",
   "name": "conda-env-Anaconda2-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  },
  "widgets": {
   "state": {
    "096725bddc2640a387bc975f8170c8a5": {
     "views": [
      {
       "cell_index": 134
      }
     ]
    },
    "37e6f7c391c24e5788bf81156c9e6a59": {
     "views": [
      {
       "cell_index": 118
      }
     ]
    },
    "3b74615e3a81456aa3b1338402509e17": {
     "views": [
      {
       "cell_index": 30
      }
     ]
    },
    "473a9130f2be44548a21c6eb9ede8fc2": {
     "views": [
      {
       "cell_index": 56
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}