AfricArxiv_stats_on_Zenodo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "AfricArxiv_stats_on_Zenodo",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AfricArxiv/decolonise-science/blob/master/AfricArxiv_stats_on_Zenodo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## [Decolonise Science's Metadata extraction](https://github.com/AfricArxiv/decolonise-science/wiki/Metadata-extraction)\n",
        "\n",
        "This was adapted from `zenodo-community-stats.sh` and configured to work on colab"
      ],
      "metadata": {
        "id": "j3bn4xIgivXi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run the cells below to the end to get your extracted CSV file. You can also do `Runtime` -> `Run all` to run all automatically. "
      ],
      "metadata": {
        "id": "UJFg3X2hjz_9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Colab allows you to work with scripts adapted for other shells e.g jq used to parse json files is used in the ubuntu shell"
      ],
      "metadata": {
        "id": "PyfF-PC_08fN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "sudo apt is used here to install the packages curl and jq. # To use this script you need to have \"curl\" and \"jq\" installed."
      ],
      "metadata": {
        "id": "asR7ABei2Oy8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! sudo apt-get install curl jq"
      ],
      "metadata": {
        "id": "HVFOupqVc6ZL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We are using Python to set the variables because they do not get recognized just using `!`.\n",
        "\n",
        "If you want to change the name of your extracted file, change the vairable `FILENAME`."
      ],
      "metadata": {
        "id": "4ZsI6v-lkn62"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from datetime import datetime\n",
        "\n",
        "# current date and time\n",
        "curDT = datetime.now()\n",
        "# current date\n",
        "date = curDT.strftime(\"%Y-%m-%d\")\n",
        "# creating the date object of today's date\n",
        "os.environ[\"date\"] =  date\n",
        "# Change below for your filename\n",
        "FILENAME = f\"decolsci_zenodo-extract_{date}\"\n",
        "os.environ[\"filename\"] = FILENAME"
      ],
      "metadata": {
        "id": "Lro4qUolg5ZD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Download all records (including multiple versions) from the community (max 10k records)\n",
        "!curl \"https://zenodo.org/api/records/?page=1&size=10000&q=&all_versions=true&communities=africarxiv\" > zenodo-resp.json"
      ],
      "metadata": {
        "id": "bBmed65P2owr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create CSV file header\n",
        "!echo \"URL,DOI,Title,Abstract,Views,Downloads,Author,Affiliation,ORCID,Created,Updated,License,Keywords\" > \"${filename}.csv\"\n",
        "# Process response and append to CSV file"
      ],
      "metadata": {
        "id": "oYKJf5Xz2s2P"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ntiiwu88ckUr",
        "outputId": "97359765-2a6d-4f10-c4cd-38dc0477e705"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100  864k  100  864k    0     0   187k      0  0:00:04  0:00:04 --:--:--  211k\n"
          ]
        }
      ],
      "source": [
        "# Take first author information, parse timestamps, join keywords into a single string if array\n",
        "!jq -r '.hits.hits[] | [.links.self, .metadata.doi, .metadata.title, .metadata.description, .stats.views, .stats.downloads, .metadata.creators[0].name, .metadata.creators[0].affiliation, .metadata.creators[0].orcid, (.created | split(\".\")[0] + \"Z\" | fromdate | strftime(\"%Y-%m-%d\")), (.updated | split(\".\")[0] + \"Z\" | fromdate | strftime(\"%Y-%m-%d\")), .metadata.license.id, (.metadata.keywords | if type == \"array\" then join(\", \") else . end)] | @csv' zenodo-resp.json >> \"${filename}.csv\"\n",
        "\n",
        "# Add Excel UTF-8 BOM to CSV file\n",
        "!echo -ne \"\\xEF\\xBB\\xBF\" | cat - \"${filename}.csv\" > temp.csv && mv temp.csv \"${filename}.csv\"\n"
      ]
    }
  ]
}