cord19kg/examples/notebooks/Co-occurrence analysis tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites and installation instructions\n",
    "\n",
    "\n",
    "In order to run this notebook `graph-tool` must be installed manually (it cannot be installed as a part of `pip install bluegraph`, as it is not an ordinary Python library, but a wrapper around a C++ library). Please, see [graph-tool installation instructions](https://git.skewed.de/count0/graph-tool/-/wikis/installation-instructions#native-installation) (currently, BlueGraph supports :code:`graph-tool<=2.37`.)\n",
    "\n",
    "We recommend using `conda` for installing `graph-tool`. For example:\n",
    "\n",
    "```\n",
    "conda install -c conda-forge graph-tool==2.37\n",
    "```\n",
    "\n",
    "or as a part of a new `conda` environment:\n",
    "\n",
    "```\n",
    "conda create --name <your_environment> -c conda-forge graph-tool==2.37\n",
    "conda activate <your_environment>\n",
    "```\n",
    "\n",
    "\n",
    "BlueGraph and the set of dependecies supporting custom tools for CORD-19 analysis can be installed using:\n",
    "\n",
    " ```\n",
    " pip install bluegraph[cord19kg]\n",
    " ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Co-occurrence network analysis tutorial\n",
    "\n",
    "In this notebook we will illustrate how interactive exploration and analysis of the [CORD-19](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge) dataset can be performed using the `cord19kg` package. The exploration and analysis techniques presented here focus on named entities and their co-occurrence in the scientific articles constituting the dataset.\n",
    "\n",
    "The input data for this notebook contains the named entities extracted from a small selection of 20 articles representing 10 different entity types (i.e. proteins, chemicals, drugs, diseases, condtions, organs, organisms, pathways, cell types, cell compartments). The article selection corresponds to 20 most relevant articles to the query _\"Glucose is a risk factor of COVID-19\"_ obtained using the article search model provided by [BlueSearch](https://github.com/BlueBrain/Search). The entity extraction was performed using the Named Entity Recognition (NER) techniques also included as a part of [BlueSearch](https://github.com/BlueBrain/Search).\n",
    "\n",
    "The `cord19kg` package provides a set of tools for interactive literature exploration through the named entity co-occurrence analysis consisting of the following steps:\n",
    "\n",
    "1. __Data preparation__ step converts raw mentions into aggregated entity occurrence statistics.\n",
    "2. __Data curation__ step allows the user to manage extracted entities: modify, filter them and link to the ontology.\n",
    "3. __Network generation__ step allows creating entity co-occurrence networks based on paper-, section- and paragraph-level co-occurrence relations between entities. These entity relations are quantified using mutual-information-based scores (pointwise mutual information and its normalized version).\n",
    "4. __Network visualization and analysis__ step allows the user to perform interactive network visualization, edit network elements and perform its analysis (spanning tree, mutual-information based shortest paths between entities, etc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import zipfile\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import dash_cytoscape as cyto\n",
    "\n",
    "from cord19kg.utils import (generate_curation_table,\n",
    "                            link_ontology,\n",
    "                            generate_cooccurrence_analysis)\n",
    "from cord19kg.apps.curation_app import curation_app\n",
    "from cord19kg.apps.visualization_app import visualization_app"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell below loads additional graph layouts used in the graph visualization app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cyto.load_extra_layouts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data preparation\n",
    "\n",
    "The input dataset contains occurrences of different terms in paragraphs of scientific articles from the CORD-19 dataset previously extracted by means of a NER model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"../data/Glucose_risk_20_papers.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the first preparation step, we group and aggregate the input data by unique entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "print(\"Prepating curatation data...\")\n",
    "curation_input_table, factor_counts = generate_curation_table(data)\n",
    "print(\"Done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting dataframe contains a row per unique named entity together with the following occurrence data: \n",
    "- sets of unique paragraphs, papers, sections, where the corresponding entity is mentioned (`paper`, `section`, `paragraph` columns);\n",
    "- number of total entity occurrences (the `raw_frequency` column);\n",
    "- number of unique papers where it occurs (the `paper_frequency` column);\n",
    "- unique entity types assigned by the NER model (the `entity_type` column, multiple types are possible).\n",
    "- raw entity types assigned by the NER model with the multiplicity of thier occurrence (the `raw_entity_types` column).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curation_input_table.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second output of the data preparation step outputs the counts of different instances of occurrence factors: number of distinct papers/sections/paragraphs in the input corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "factor_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Data curation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the NCIT ontology linking data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To group synonymical entities in the previously extracted table (e.g. `ace2`, `ace-2`, `angiotensin-converting enzyme 2`), as well as assign additional semantics to these entities (e.g. human-readable definition, taxonomy, etc), we peform further _linking_ of the entities to the terms from the [NCIT ontology](https://ncithesaurus.nci.nih.gov/ncitbrowser/).\n",
    "\n",
    "To be able to perform such ontology linking, we load some additional (pre-computed using ML-based linking models) data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "print(\"Loading the ontology linking data...\")\n",
    "    \n",
    "print(\"\\tDecompressing the input data file...\")\n",
    "with zipfile.ZipFile(\"../data/NCIT_ontology_linking_3000_papers.csv.zip\", 'r') as zip_ref:\n",
    "    zip_ref.extractall(\"../data/\")\n",
    "\n",
    "print(\"\\tLoading the linking dataframe in memory...\")\n",
    "ontology_linking = pd.read_csv(\"../data/NCIT_ontology_linking_3000_papers.csv\")\n",
    "\n",
    "print(\"\\tLoading ontology type mapping...\")\n",
    "with open(\"../data/NCIT_type_mapping.json\", \"rb\") as f:\n",
    "    type_mapping = json.load(f)\n",
    "print(\"Done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ontology linking table contains the following columns:\n",
    "- `mention` entity mentioned in the text\n",
    "- `concept` ontology concept linked to the entity mention\n",
    "- `uid` unique identifier of the ontology concept\n",
    "- `definition` definition of the concept\n",
    "- `taxonomy` taxonomy of semantic types associated with the concept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ontology_linking.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive curation of  entity occurrence data\n",
    "\n",
    "The package provides an interactive entity curation app that allows the user to visualize the entity occurrence data, modify it, perform ontology linking (see `Link to NCIT ontology` button), filter short or unfrequent entities.\n",
    "\n",
    "The field `Keep` allows specifying a set of entities that must be kept in the dataset at all times (even if they don't satisfy the selected filtering criteria).\n",
    "\n",
    "Finally the value specified in the `Generate Graphs from top 500 frequent entities` field corresponds to the number of top entities (by the frequency of their occurrence in papers) to be included in the co-occurrence network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the prepared data table into the curation app as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curation_app.set_table(curation_input_table.copy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can specify the default entities to keep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_entities_to_keep = [\"glucose\", \"covid-19\"]\n",
    "curation_app.set_default_terms_to_include(default_entities_to_keep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we set the ontology linking callback to be fired upon a click on the `Link to NCIT ontology` button."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "curation_app.set_ontology_linking_callback(lambda x: link_ontology(ontology_linking, type_mapping, x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launch the curation app"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The application can be launched either inline (inside the current notebook) as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "curation_app.run(port=8072, mode=\"inline\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or it can be opened externally (by the URL that you can open in a separate tab of your browser, try uncommenting, executing and doing Ctrl+Click on the displayed URL)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# curation_app.run(port=8070, mode=\"external\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Co-occurrence network generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Current curation table displayed in the curation app can be extracted using the `get_curated_table` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curated_occurrence_data = curation_app.get_curated_table()\n",
    "curated_occurrence_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also retreive current values of the `Keep` field (these entities will be also included in the resulting co-occurrence network)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curation_app.get_terms_to_include()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the number of top frequent entities to use for network generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curation_app.n_most_frequent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating co-occurrence networks\n",
    "\n",
    "In the cell below we generate two co-occurrence graphs: for paper- and paragraph-based entity co-occurrences. Along with the graph generation the `generate_cooccurrence_analysis` function \n",
    "\n",
    "- computes node centrality metrics (such as degree, RageRank)\n",
    "- computes co-occurrence statistics (such as frequency, pointwise mutual information and normalized pointwise mutual information) and assignes them as weights to the corresponding edges\n",
    "- performs entity community detection based on different co-occurrence statistics\n",
    "- computes mutual-information-based minimum spanning trees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we run the co-occurrence analysis, we will create a dictionary with backend configurations for the analytics: we set metrics (centalities) computation to use `graph_tool`, community detection to use `networkx` and, finally, path search to use `graph_tool` as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "backend_configs = {\n",
    "    \"metrics\": \"graph_tool\",\n",
    "    \"communities\": \"networkx\",\n",
    "    \"paths\": \"graph_tool\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "type_data = curated_occurrence_data[[\"entity_type\"]].rename(columns={\"entity_type\": \"type\"})\n",
    "\n",
    "graphs, trees = generate_cooccurrence_analysis(\n",
    "    curated_occurrence_data,  factor_counts,\n",
    "    n_most_frequent=curation_app.n_most_frequent,\n",
    "    type_data=type_data, \n",
    "    factors=[\"paper\", \"paragraph\"],\n",
    "    keep=curation_app.get_terms_to_include(),\n",
    "    cores=8, \n",
    "    backend_configs=backend_configs)\n",
    "\n",
    "print(\"Done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting graph objects are instances of `PandasPGFrame` - frame-based property graphs provided by `BlueGraph`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graphs[\"paper\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graphs[\"paper\"].nodes(raw_frame=True).sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graphs[\"paper\"].edges(raw_frame=True).sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Network visualization and analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we set a backend for the visualization app (currently two backends are supported: based on `NetworkX` and `graph-tool`, in this example we use the latter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualization_app.set_backend(\"graph_tool\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Run the following use NetworkX as the backend for the visualization app\n",
    "# visualization_app.set_backend(\"networkx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the generated graphs into the visualization app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualization_app.add_graph(\n",
    "    \"Paper-based graph\", graphs[\"paper\"],\n",
    "    tree=trees[\"paper\"], default_top_n=100)\n",
    "\n",
    "visualization_app.add_graph(\n",
    "    \"Paragraph-based graph\", graphs[\"paragraph\"],\n",
    "    tree=trees[\"paragraph\"], default_top_n=100)\n",
    "\n",
    "visualization_app.set_current_graph(\"Paragraph-based graph\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading papers' meta-data into the app\n",
    "\n",
    "We now load an additional dataset containing some meta-data on the papers where the entities analyzed in this notebook occur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "paper_data = pd.read_csv(\"../data/Glucose_risk_3000_paper_meta_data.csv\")\n",
    "paper_data = paper_data.set_index(\"id\")\n",
    "paper_data.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We pass a callback for the lookup of paper meta-data to the visualization app using the `set_list_papers_callback` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def list_papers(paper_data, selected_papers, limit=200):\n",
    "    selected_paper_data = paper_data.loc[[int(p) for p in selected_papers]].head(200)\n",
    "    return selected_paper_data.to_dict(\"records\")\n",
    "\n",
    "visualization_app.set_list_papers_callback(lambda x: list_papers(paper_data, x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ontology linking process described above is noisy, therefore, we would like to keep a possibility of accessing, the raw entities that were linked to particular ontology concepts. For this we define the function `get_aggregated_entities` that retreives such raw entities and we pass it to the visualization app using the `set_aggregated_entities_callback` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top_n(data_dict, n, smallest=False):\n",
    "    \"\"\"Return top `n` keys of the input dictionary by their value.\"\"\"\n",
    "    df = pd.DataFrame(dict(data_dict).items(), columns=[\"id\", \"value\"])\n",
    "    if smallest:\n",
    "        df = df.nsmallest(n, columns=[\"value\"])\n",
    "    else:\n",
    "        df = df.nlargest(n, columns=[\"value\"])\n",
    "    return(list(df[\"id\"]))\n",
    "\n",
    "\n",
    "def get_aggregated_entities(entity, n):\n",
    "    if \"aggregated_entities\" in curated_occurrence_data.columns:\n",
    "        if entity not in curated_occurrence_data.index:\n",
    "            return None\n",
    "        aggregated = curated_occurrence_data.loc[entity][\"aggregated_entities\"]\n",
    "    else:\n",
    "        aggregated = [entity]\n",
    "    if curation_input_table is not None:\n",
    "        df = curation_input_table.set_index(\"entity\")\n",
    "        if entity in curated_occurrence_data.index:\n",
    "            freqs = df.loc[aggregated][\"paper_frequency\"].to_dict()\n",
    "        else:\n",
    "            return {}\n",
    "    else:\n",
    "        df = data.copy()\n",
    "        df[\"entity\"] = data[\"entity\"].apply(lambda x: x.lower())\n",
    "        freqs = df[df[\"entity\"].apply(lambda x: x.lower() in aggregated)].groupby(\"entity\").aggregate(\n",
    "            lambda x: len(x))[\"entity_type\"].to_dict()\n",
    "    if len(freqs) == 0:\n",
    "        return {}\n",
    "    return {e: freqs[e] for e in top_n(freqs, n)}\n",
    "\n",
    "visualization_app.set_aggregated_entities_callback(\n",
    "    lambda x: get_aggregated_entities(x, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we create a dictionary `definitions` that will serve the visualization app as the lookup table for accessing the definitions of different ontology concepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "definitions = ontology_linking[[\"concept\", \"definition\"]].groupby(\n",
    "    \"concept\").aggregate(lambda x: list(x)[0]).to_dict()[\"definition\"]\n",
    "visualization_app.set_entity_definitons(definitions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launching the visualization app"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, the interactive graph visualization app can be launched in two modes: inline and external. Here we recommend the external mode for better user experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "visualization_app.run(port=8082, mode=\"external\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "clean_bg",
   "language": "python",
   "name": "clean_bg"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}