Scholarly_Literature_Review.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Literature Review with Scholarly for Python"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scholarly is a python module providing programmatic access to metadata from Google Scholar.\n",
    "\n",
    "#### Installation\n",
    "\n",
    "To use this notebook you need to install the Scholarly module from the command line using `pip install scholarly`.\n",
    "\n",
    "#### Documentation\n",
    "\n",
    "- Scholarly github repository: https://github.com/scholarly-python-package/scholarly\n",
    "- Scholarly documentation: https://scholarly.readthedocs.io/en/stable/\n",
    "\n",
    "#### Notes:\n",
    "\n",
    "1. Scholarly doesn't require an API key because it doesn't use an API.\n",
    "2. The module searches Google Scholar by using a query string and returning a generator-object which behaves like an iterator: https://www.geeksforgeeks.org/generators-in-python/\n",
    "3. Generator objects interact with Google Scholar dynamically and can be used either by calling the `next()` method on the generator object, or by using the generator object in a `for in` loop.\n",
    "\n",
    "**Warning:** Google Scholar can block your IP address when using Scholarly. These temporary bans can last for between 1 and 48 hours. To avoiding issues of this kind it is advised that you use a proxy server. Refer to the Scholarly GitHub page and documentation for further information."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scholarly import scholarly\n",
    "import pandas as pd\n",
    "import json\n",
    "from flatten_json import flatten\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search for Author"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# User inputs name of author\n",
    "author_name = input('Enter author name: ')\n",
    "\n",
    "# Request author from Google Scholar\n",
    "author = next(scholarly.search_author(author_name))\n",
    "\n",
    "# Dump author object as string and load into json object\n",
    "j = json.loads(json.dumps(author))\n",
    "\n",
    "# Display json with formatting\n",
    "print(json.dumps(j, indent=3))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Search for Author and Fill Respone with Additional Details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# User inputs name of author\n",
    "author_name = input('Enter author name: ')\n",
    "\n",
    "# Retrieve the author's data, fill-in, and print\n",
    "search_query = scholarly.search_author(author_name)\n",
    "\n",
    "# Obtain additional author details with Fill\n",
    "author = scholarly.fill(next(search_query))\n",
    "\n",
    "# Dump author object as string and load into json object\n",
    "j = json.loads(json.dumps(author))\n",
    "\n",
    "# Display json with formatting\n",
    "print(json.dumps(j, indent=3))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search for Publications\n",
    "\n",
    "When performing a search be aware that the choice and ordering of search terms can impact the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# User provides search parameters\n",
    "search_term = input('Enter search term(s) separated by commas: ')\n",
    "number_articles = int(input('Number of publications to return? '))\n",
    "print(f\"Search terms: {search_term}\")\n",
    "print(f\"Publications to retrieve: {number_articles}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Request Publications from Google Scholar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_query = scholarly.search_pubs(search_term)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Intialise array of dataframes and count of articles for data collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a list of to store data\n",
    "data = []\n",
    "#Count articles\n",
    "count = 0"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Call the generator object in a loop and output to a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Convert each publication to a dataframe and add it to the list\n",
    "for i in range(number_articles):\n",
    "    # Check count of articles does not exceed requested articles\n",
    "    if(count < number_articles):\n",
    "        try:\n",
    "            # Request next publication\n",
    "            pub = scholarly.fill(next(search_query))\n",
    "\n",
    "            # Flatten publication details and removed unnecessary fields\n",
    "            flat_dict = flatten(pub, root_keys_to_ignore={'source', 'author_id'})\n",
    "\n",
    "            # Add dictionary to list of data\n",
    "            data.append(flat_dict)\n",
    "\n",
    "            #Increment count of articles\n",
    "            count = count + 1\n",
    "\n",
    "            #Debug logging\n",
    "            print('Article ' + str(count) + ': ' + flat_dict['bib_title'])\n",
    "\n",
    "        except StopIteration:\n",
    "            print('Ending as generator exhausted')\n",
    "            break\n",
    "\n",
    "        except Exception as e:\n",
    "            print('Error: ' + str(e))\n",
    "\n",
    "        #Wait time in secs before calling for next article\n",
    "        time.sleep(5)\n",
    "    else:\n",
    "        break\n",
    "    \n",
    "print('Search Complete - Reset count if more articles required.')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prepare dataframe by concatenating search results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(data)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Dataframe head\n",
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tidy Column Headings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns = df.columns.str.replace('bib_', '')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Remove Duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop_duplicates(ignore_index=True).reset_index(drop=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Summary Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Summary information\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Unique publications\n",
    "len(df['id'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Publication provenance\n",
    "provenance_table = df.groupby(['id', 'journal']).size().reset_index()\n",
    "provenance_table = provenance_table.groupby('journal').size()\n",
    "provenance_table.replace('&', '\\&')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Unique Publications per Year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles_per_year = df.groupby(['id', 'pub_year']).size().reset_index().groupby('pub_year').size()\n",
    "\n",
    "fig, ax = plt.subplots(figsize = (10, 5))\n",
    "\n",
    "plt.title('Total Articles per Year', fontsize = 20)\n",
    "plt.ylabel('Number of Records', fontsize = 10); \n",
    "\n",
    "ax.plot(articles_per_year, zorder = 1, linewidth = 3)\n",
    "\n",
    "plt.yticks(fontsize = 10)\n",
    "plt.xticks(fontsize = 10, rotation = 90); \n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export Data\n",
    "\n",
    "A folder called 'data' needs to be added to the folder containing this Jupyter notebook before writing the file."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Format Filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "replacement_name = search_term.replace(' ', '_')\n",
    "replacement_name = replacement_name.replace(',', '')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save to CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('data/' + replacement_name + '_Search_Scholarly' + '.csv', index=False, encoding='utf-8-sig')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save to JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_json('data/' + replacement_name + '_Search_Scholarly' + '.json', orient='records')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}