flair_nikki_tagger.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "flair-nikki-tagger.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nikolito/ogiNikki_flair/blob/main/flair_nikki_tagger.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oLPvhxN4jW9s"
      },
      "source": [
        "# **Named Entity Recognition (NER) for Ogihan Nikki Mokuroku titles**\n",
        "## Playground\n",
        "\n",
        "  This script is based on Flair modeling scripts (https://colab.research.google.com/drive/1Mi8GN_x_jQUgOCDjkkjtlN_OXXO3dCF7) created by Hironsan. We try to extract NER class such as persons, places, roles and so on from titles on Ogihan Nikki Database (OND). Please refer to https://crch.dl.saga-u.ac.jp/nikki/ for further information.\n",
        "  OND has a corpus of ca. 74000 sentences that freely available under the Creative Commons CC BY-NC-SA 4.0. We have prepared OND corpus approximately 40000 sentences with NER annotation labels. Thanks to Google Colaboratory, we can create models with up to 30000 sentences here. (Please reduce the mini batch size from 32 to 16 preventing OOM in case of 30000 sentences. )\n",
        "  Once creating a model, you can deploy NER with titles on OND. We recommend that you try \"uid\" 40000 or later for real prediction. \n",
        "\n",
        "## Model Creation\n",
        "Refer to [our script](https://colab.research.google.com/drive/1vrxCIx2o-4GiP8zewZO18UngEAQ6Lfnv?usp=sharing). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tWuXaOQlQMex",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "444cc65b-8832-4d51-9388-b37052f0ce0c"
      },
      "source": [
        "!python -V"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Python 3.7.12\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A0AngDOrZWQr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e88bc68a-ba3d-4503-a3b2-96f9bf2d6875"
      },
      "source": [
        "!pip install flair"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting flair\n",
            "  Downloading flair-0.10-py3-none-any.whl (322 kB)\n",
            "\u001b[K     |████████████████████████████████| 322 kB 5.3 MB/s \n",
            "\u001b[?25hCollecting huggingface-hub\n",
            "  Downloading huggingface_hub-0.2.1-py3-none-any.whl (61 kB)\n",
            "\u001b[K     |████████████████████████████████| 61 kB 476 kB/s \n",
            "\u001b[?25hCollecting janome\n",
            "  Downloading Janome-0.4.1-py2.py3-none-any.whl (19.7 MB)\n",
            "\u001b[K     |████████████████████████████████| 19.7 MB 1.3 MB/s \n",
            "\u001b[?25hCollecting deprecated>=1.2.4\n",
            "  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)\n",
            "Collecting konoha<5.0.0,>=4.0.0\n",
            "  Downloading konoha-4.6.5-py3-none-any.whl (20 kB)\n",
            "Requirement already satisfied: gensim>=3.4.0 in /usr/local/lib/python3.7/dist-packages (from flair) (3.6.0)\n",
            "Collecting sentencepiece==0.1.95\n",
            "  Downloading sentencepiece-0.1.95-cp37-cp37m-manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.2 MB 35.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from flair) (0.8.9)\n",
            "Collecting bpemb>=0.3.2\n",
            "  Downloading bpemb-0.3.3-py3-none-any.whl (19 kB)\n",
            "Collecting sqlitedict>=1.6.0\n",
            "  Downloading sqlitedict-1.7.0.tar.gz (28 kB)\n",
            "Requirement already satisfied: matplotlib>=2.2.3 in /usr/local/lib/python3.7/dist-packages (from flair) (3.2.2)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from flair) (2019.12.20)\n",
            "Collecting transformers>=4.0.0\n",
            "  Downloading transformers-4.15.0-py3-none-any.whl (3.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.4 MB 30.2 MB/s \n",
            "\u001b[?25hCollecting conllu>=4.0\n",
            "  Downloading conllu-4.4.1-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: tqdm>=4.26.0 in /usr/local/lib/python3.7/dist-packages (from flair) (4.62.3)\n",
            "Collecting mpld3==0.3\n",
            "  Downloading mpld3-0.3.tar.gz (788 kB)\n",
            "\u001b[K     |████████████████████████████████| 788 kB 49.1 MB/s \n",
            "\u001b[?25hCollecting gdown==3.12.2\n",
            "  Downloading gdown-3.12.2.tar.gz (8.2 kB)\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "    Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: torch!=1.8,>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from flair) (1.10.0+cu111)\n",
            "Collecting langdetect\n",
            "  Downloading langdetect-1.0.9.tar.gz (981 kB)\n",
            "\u001b[K     |████████████████████████████████| 981 kB 35.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.7/dist-packages (from flair) (1.0.1)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.7/dist-packages (from flair) (2.8.2)\n",
            "Collecting segtok>=1.5.7\n",
            "  Downloading segtok-1.5.11-py3-none-any.whl (24 kB)\n",
            "Collecting ftfy\n",
            "  Downloading ftfy-6.0.3.tar.gz (64 kB)\n",
            "\u001b[K     |████████████████████████████████| 64 kB 2.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from flair) (4.2.6)\n",
            "Collecting more-itertools~=8.8.0\n",
            "  Downloading more_itertools-8.8.0-py3-none-any.whl (48 kB)\n",
            "\u001b[K     |████████████████████████████████| 48 kB 4.7 MB/s \n",
            "\u001b[?25hCollecting wikipedia-api\n",
            "  Downloading Wikipedia-API-0.5.4.tar.gz (18 kB)\n",
            "Requirement already satisfied: requests[socks] in /usr/local/lib/python3.7/dist-packages (from gdown==3.12.2->flair) (2.23.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from gdown==3.12.2->flair) (3.4.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from gdown==3.12.2->flair) (1.15.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from bpemb>=0.3.2->flair) (1.19.5)\n",
            "Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.7/dist-packages (from deprecated>=1.2.4->flair) (1.13.3)\n",
            "Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.7/dist-packages (from gensim>=3.4.0->flair) (5.2.1)\n",
            "Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.7/dist-packages (from gensim>=3.4.0->flair) (1.4.1)\n",
            "Collecting requests\n",
            "  Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)\n",
            "\u001b[K     |████████████████████████████████| 62 kB 756 kB/s \n",
            "\u001b[?25hCollecting importlib-metadata<4.0.0,>=3.7.0\n",
            "  Downloading importlib_metadata-3.10.1-py3-none-any.whl (14 kB)\n",
            "Collecting overrides<4.0.0,>=3.0.0\n",
            "  Downloading overrides-3.1.0.tar.gz (11 kB)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata<4.0.0,>=3.7.0->konoha<5.0.0,>=4.0.0->flair) (3.6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata<4.0.0,>=3.7.0->konoha<5.0.0,>=4.0.0->flair) (3.10.0.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.2.3->flair) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.2.3->flair) (1.3.2)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.2.3->flair) (3.0.6)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->bpemb>=0.3.2->flair) (2.10)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.7/dist-packages (from requests->bpemb>=0.3.2->flair) (2.0.8)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->bpemb>=0.3.2->flair) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->bpemb>=0.3.2->flair) (2021.10.8)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->flair) (1.1.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->flair) (3.0.0)\n",
            "Collecting sacremoses\n",
            "  Downloading sacremoses-0.0.46-py3-none-any.whl (895 kB)\n",
            "\u001b[K     |████████████████████████████████| 895 kB 41.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.0.0->flair) (21.3)\n",
            "Collecting tokenizers<0.11,>=0.10.1\n",
            "  Downloading tokenizers-0.10.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.3 MB 36.5 MB/s \n",
            "\u001b[?25hCollecting pyyaml>=5.1\n",
            "  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)\n",
            "\u001b[K     |████████████████████████████████| 596 kB 44.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from ftfy->flair) (0.2.5)\n",
            "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.7/dist-packages (from requests->bpemb>=0.3.2->flair) (1.7.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=4.0.0->flair) (7.1.2)\n",
            "Building wheels for collected packages: gdown, mpld3, overrides, sqlitedict, ftfy, langdetect, wikipedia-api\n",
            "  Building wheel for gdown (PEP 517) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gdown: filename=gdown-3.12.2-py3-none-any.whl size=9704 sha256=e2027190da12c490ba04d60911311baf3c947cc973091f669d87a86d2f20c353\n",
            "  Stored in directory: /root/.cache/pip/wheels/ba/e0/7e/726e872a53f7358b4b96a9975b04e98113b005cd8609a63abc\n",
            "  Building wheel for mpld3 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for mpld3: filename=mpld3-0.3-py3-none-any.whl size=116702 sha256=4d909c36ea6af19db40a032946e37ef257a748fc18dcf65fff64b6de1d80f1e6\n",
            "  Stored in directory: /root/.cache/pip/wheels/26/70/6a/1c79e59951a41b4045497da187b2724f5659ca64033cf4548e\n",
            "  Building wheel for overrides (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for overrides: filename=overrides-3.1.0-py3-none-any.whl size=10186 sha256=eba3a3c3d9412c5bf292c6695e556648e3494bc140d6a0c04f0eecbced7f699e\n",
            "  Stored in directory: /root/.cache/pip/wheels/3a/0d/38/01a9bc6e20dcfaf0a6a7b552d03137558ba1c38aea47644682\n",
            "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sqlitedict: filename=sqlitedict-1.7.0-py3-none-any.whl size=14392 sha256=7d2fb6216d3b21ab022a1e9ffb5259a0524d034288d74c2c21a0b4e2c887842a\n",
            "  Stored in directory: /root/.cache/pip/wheels/af/94/06/18c0e83e9e227da8f3582810b51f319bbfd181e508676a56c8\n",
            "  Building wheel for ftfy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ftfy: filename=ftfy-6.0.3-py3-none-any.whl size=41933 sha256=823f4714a9055844882aa0256c2381fa2322f15c9add6821574647402b834504\n",
            "  Stored in directory: /root/.cache/pip/wheels/19/f5/38/273eb3b5e76dfd850619312f693716ac4518b498f5ffb6f56d\n",
            "  Building wheel for langdetect (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for langdetect: filename=langdetect-1.0.9-py3-none-any.whl size=993242 sha256=c103be749838636331ec92ed6097c4858f9ef58abbbca6ced0cdb4b46a42c084\n",
            "  Stored in directory: /root/.cache/pip/wheels/c5/96/8a/f90c59ed25d75e50a8c10a1b1c2d4c402e4dacfa87f3aff36a\n",
            "  Building wheel for wikipedia-api (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wikipedia-api: filename=Wikipedia_API-0.5.4-py3-none-any.whl size=13475 sha256=568d412ff80569e1d2ed778d9654a5e5247518b6d386fcf3d5d4085cec2859fd\n",
            "  Stored in directory: /root/.cache/pip/wheels/d3/24/56/58ba93cf78be162451144e7a9889603f437976ef1ae7013d04\n",
            "Successfully built gdown mpld3 overrides sqlitedict ftfy langdetect wikipedia-api\n",
            "Installing collected packages: requests, pyyaml, importlib-metadata, tokenizers, sentencepiece, sacremoses, overrides, huggingface-hub, wikipedia-api, transformers, sqlitedict, segtok, mpld3, more-itertools, langdetect, konoha, janome, gdown, ftfy, deprecated, conllu, bpemb, flair\n",
            "  Attempting uninstall: requests\n",
            "    Found existing installation: requests 2.23.0\n",
            "    Uninstalling requests-2.23.0:\n",
            "      Successfully uninstalled requests-2.23.0\n",
            "  Attempting uninstall: pyyaml\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "  Attempting uninstall: importlib-metadata\n",
            "    Found existing installation: importlib-metadata 4.8.2\n",
            "    Uninstalling importlib-metadata-4.8.2:\n",
            "      Successfully uninstalled importlib-metadata-4.8.2\n",
            "  Attempting uninstall: more-itertools\n",
            "    Found existing installation: more-itertools 8.12.0\n",
            "    Uninstalling more-itertools-8.12.0:\n",
            "      Successfully uninstalled more-itertools-8.12.0\n",
            "  Attempting uninstall: gdown\n",
            "    Found existing installation: gdown 3.6.4\n",
            "    Uninstalling gdown-3.6.4:\n",
            "      Successfully uninstalled gdown-3.6.4\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "markdown 3.3.6 requires importlib-metadata>=4.4; python_version < \"3.10\", but you have importlib-metadata 3.10.1 which is incompatible.\n",
            "google-colab 1.0.0 requires requests~=2.23.0, but you have requests 2.26.0 which is incompatible.\n",
            "datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "Successfully installed bpemb-0.3.3 conllu-4.4.1 deprecated-1.2.13 flair-0.10 ftfy-6.0.3 gdown-3.12.2 huggingface-hub-0.2.1 importlib-metadata-3.10.1 janome-0.4.1 konoha-4.6.5 langdetect-1.0.9 more-itertools-8.8.0 mpld3-0.3 overrides-3.1.0 pyyaml-6.0 requests-2.26.0 sacremoses-0.0.46 segtok-1.5.11 sentencepiece-0.1.95 sqlitedict-1.7.0 tokenizers-0.10.3 transformers-4.15.0 wikipedia-api-0.5.4\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t_CwrUxTZ6TI"
      },
      "source": [
        "from flair.data import Sentence\n",
        "from flair.models import SequenceTagger"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_mR9zDdq8nP5"
      },
      "source": [
        "# Download our ready-made model.\n",
        "We created a sample model using 30000 sentences from OND. The model size is 738MB. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WHyleLl8axOP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b93a0d00-8ab0-4d11-d04d-1ba7110ff27d"
      },
      "source": [
        "!wget https://crch.dl.saga-u.ac.jp/nikki/dataset/final-model.pt\n",
        "model = SequenceTagger.load('final-model.pt')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-12-23 07:59:25--  https://crch.dl.saga-u.ac.jp/nikki/dataset/final-model.pt\n",
            "Resolving crch.dl.saga-u.ac.jp (crch.dl.saga-u.ac.jp)... 153.126.194.247\n",
            "Connecting to crch.dl.saga-u.ac.jp (crch.dl.saga-u.ac.jp)|153.126.194.247|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 773901813 (738M)\n",
            "Saving to: ‘final-model.pt’\n",
            "\n",
            "final-model.pt      100%[===================>] 738.05M  11.4MB/s    in 65s     \n",
            "\n",
            "2021-12-23 08:00:31 (11.3 MB/s) - ‘final-model.pt’ saved [773901813/773901813]\n",
            "\n",
            "2021-12-23 08:00:31,924 loading file final-model.pt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dQU4-cqfK732"
      },
      "source": [
        "If you change some words in a sentence, the model tries to answer NER results. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FIe2-qjJhxKZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f9f98568-f2a9-4699-b763-1a6cc7ead885"
      },
      "source": [
        "strings = \"七島要太夫・秀島徳次郎役方被仰付候事\"\n",
        "str_space = \" \".join(strings)\n",
        "sentence = Sentence(str_space)\n",
        "model.predict(sentence)\n",
        "print(sentence.to_tagged_string())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "七 <B-JINMEI> 島 <I-JINMEI> 要 <I-JINMEI> 太 <I-JINMEI> 夫 <I-JINMEI> ・ 秀 <B-JINMEI> 島 <I-JINMEI> 徳 <I-JINMEI> 次 <I-JINMEI> 郎 <I-JINMEI> 役 <B-ROLE> 方 <I-ROLE> 被 <B-TERMS> 仰 <I-TERMS> 付 <I-TERMS> 候 <I-TERMS> 事 <I-TERMS>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_VMJv4Wr2AgD"
      },
      "source": [
        "# Sample sentences from OND\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "*   災変ニ付増田与四右衛門深川善五郎野田新助古賀文左衛門西岡九兵衛御褒美之事\n",
        "*   光野卯左衛門徳見傳助同伴帰崎之事\n",
        "*   光野ピコ左衛門石川五右衛門同伴帰崎之事\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "（You can try more examples from [Ogihan Nikki Database](https://crch.dl.saga-u.ac.jp/nikki/). ）"
      ]
    }
  ]
}