paddlenlp_updated_notes_English/WordEmbedding/2-calculating text cosine similarity in paddlenlp.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"2-calculating text cosine similarity in paddlenlp.ipynb","provenance":[],"collapsed_sections":[],"authorship_tag":"ABX9TyMp/o2CwhJMGA3NmAgBE4Y6"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"code","metadata":{"id":"X0L_wjb_SZQm"},"source":["# Author: Zhengxiang (Jack) Wang \n","# Date: created on 2021-07-18; modified on 2021-07-20\n","# GitHub: https://github.com/jaaack-wang \n","# About: calculating text similarity in paddlenlp"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"wJ7RmFfj8A0F"},"source":["# Overview\n","\n","In [loading pre-trained word embedding in paddlenp.ipynb](https://colab.research.google.com/drive/1WSyYtDiHwXe4MFTwe_X6hQ5atBqNsFax?usp=sharing), we learn how to load pre-trained word embedding models in `padddlenlp.embeddings.TokenEmbedding` as well as models predefined in `paddlenlp`. This notebook will use some simple examples to illustrate the application of word embedding in calculating text similarity between word pairs or between text (e.g., phrase, sentence) pairs. More on text similarity will be updated in my GitHub Project [dl-nlp-using-paddlenlp](https://github.com/jaaack-wang/dl-nlp-using-paddlenlp). \n","\n","\n","<br>\n","\n","\n","<table align=\"right\">\n","  <td>\n","    <a target=\"_blank\" href=\"https://colab.research.google.com/drive/1QYSJ3x6Ap5HG8O4R4yqAyw6iq18JahdO?usp=sharing\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /> Run in Google Colab </a>\n","  </td>\n","  <td>\n","    <a target=\"_blank\" href=\"https://github.com/jaaack-wang\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /> Author's GitHub </a>\n","  </td>\n","  <td>\n","    <a href=\"https://docs.google.com/uc?export=download&id=1QYSJ3x6Ap5HG8O4R4yqAyw6iq18JahdO\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /> Download this notebook </a>\n","  </td>\n","</table> \n","\n","\n","<br>\n"]},{"cell_type":"markdown","metadata":{"id":"KR9ZErQJA2N4"},"source":["# Table of Contents\n","\n","- [1. Cosine similarity](#1)\n","  - [1.1 Concept and formula](#1-1)\n","  - [1.2 Calculating cosine similarity in numpy](#1-2)\n","- [2. Calculating similarity between two words](#2)\n","- [3. Calculating similarity between two sentences](#3)\n","  - [3.1 The Bag-of-Words model](#3-1)\n","  - [3.2 Real-Wolrd examples: the performances of the Bag-of-Words model](#3-2)\n","    - [3.2.1 Chinese examples](#3-2-1)\n","    - [3.2.2 English examples](#3-2-2)\n","    - [3.2.3 Summary](#3-2-3)\n","- [4. References](#4)"]},{"cell_type":"code","metadata":{"id":"8VkmUFQLCJxf","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1626807227861,"user_tz":360,"elapsed":33095,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"ee45925d-6eb3-44b7-d953-0b2d3dbb4961"},"source":["# always installing or updating paddlepaddle and paddlenlp if you use Colab\n","\n","!pip3 install --upgrade paddlepaddle\n","!pip3 install --upgrade paddlenlp"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Collecting paddlepaddle\n","  Downloading paddlepaddle-2.1.1-cp37-cp37m-manylinux1_x86_64.whl (108.9 MB)\n","\u001b[K     |████████████████████████████████| 108.9 MB 29 kB/s \n","\u001b[?25hRequirement already satisfied: gast>=0.3.3 in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (0.4.0)\n","Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (1.15.0)\n","Requirement already satisfied: protobuf>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (3.17.3)\n","Requirement already satisfied: astor in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (0.8.1)\n","Requirement already satisfied: numpy>=1.13 in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (1.19.5)\n","Requirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (7.1.2)\n","Requirement already satisfied: decorator==4.4.2 in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (4.4.2)\n","Requirement already satisfied: requests>=2.20.0 in /usr/local/lib/python3.7/dist-packages (from paddlepaddle) (2.23.0)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.20.0->paddlepaddle) (3.0.4)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.20.0->paddlepaddle) (2.10)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.20.0->paddlepaddle) (2021.5.30)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.20.0->paddlepaddle) (1.24.3)\n","Installing collected packages: paddlepaddle\n","Successfully installed paddlepaddle-2.1.1\n","Collecting paddlenlp\n","  Downloading paddlenlp-2.0.6-py3-none-any.whl (485 kB)\n","\u001b[K     |████████████████████████████████| 485 kB 3.0 MB/s \n","\u001b[?25hRequirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from paddlenlp) (3.1.0)\n","Collecting colorama\n","  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)\n","Requirement already satisfied: multiprocess in /usr/local/lib/python3.7/dist-packages (from paddlenlp) (0.70.12.2)\n","Collecting colorlog\n","  Downloading colorlog-5.0.1-py2.py3-none-any.whl (10 kB)\n","Requirement already satisfied: jieba in /usr/local/lib/python3.7/dist-packages (from paddlenlp) (0.42.1)\n","Collecting seqeval\n","  Downloading seqeval-1.2.2.tar.gz (43 kB)\n","\u001b[K     |████████████████████████████████| 43 kB 1.4 MB/s \n","\u001b[?25hCollecting visualdl\n","  Downloading visualdl-2.2.0-py3-none-any.whl (2.7 MB)\n","\u001b[K     |████████████████████████████████| 2.7 MB 17.8 MB/s \n","\u001b[?25hRequirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py->paddlenlp) (1.5.2)\n","Requirement already satisfied: numpy>=1.14.5 in /usr/local/lib/python3.7/dist-packages (from h5py->paddlenlp) (1.19.5)\n","Requirement already satisfied: dill>=0.3.4 in /usr/local/lib/python3.7/dist-packages (from multiprocess->paddlenlp) (0.3.4)\n","Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.7/dist-packages (from seqeval->paddlenlp) (0.22.2.post1)\n","Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp) (1.0.1)\n","Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp) (1.4.1)\n","Collecting bce-python-sdk\n","  Downloading bce_python_sdk-0.8.61-py3-none-any.whl (197 kB)\n","\u001b[K     |████████████████████████████████| 197 kB 32.3 MB/s \n","\u001b[?25hRequirement already satisfied: flask>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (1.1.4)\n","Requirement already satisfied: six>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (1.15.0)\n","Collecting Flask-Babel>=1.0.0\n","  Downloading Flask_Babel-2.0.0-py3-none-any.whl (9.3 kB)\n","Requirement already satisfied: protobuf>=3.11.0 in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (3.17.3)\n","Collecting flake8>=3.7.9\n","  Downloading flake8-3.9.2-py2.py3-none-any.whl (73 kB)\n","\u001b[K     |████████████████████████████████| 73 kB 1.4 MB/s \n","\u001b[?25hRequirement already satisfied: Pillow>=7.0.0 in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (7.1.2)\n","Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (2.23.0)\n","Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (1.1.5)\n","Collecting shellcheck-py\n","  Downloading shellcheck_py-0.7.2.1-py2.py3-none-manylinux1_x86_64.whl (2.0 MB)\n","\u001b[K     |████████████████████████████████| 2.0 MB 41.5 MB/s \n","\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from visualdl->paddlenlp) (3.2.2)\n","Collecting pre-commit\n","  Downloading pre_commit-2.13.0-py2.py3-none-any.whl (190 kB)\n","\u001b[K     |████████████████████████████████| 190 kB 44.3 MB/s \n","\u001b[?25hCollecting pycodestyle<2.8.0,>=2.7.0\n","  Downloading pycodestyle-2.7.0-py2.py3-none-any.whl (41 kB)\n","\u001b[K     |████████████████████████████████| 41 kB 512 kB/s \n","\u001b[?25hCollecting pyflakes<2.4.0,>=2.3.0\n","  Downloading pyflakes-2.3.1-py2.py3-none-any.whl (68 kB)\n","\u001b[K     |████████████████████████████████| 68 kB 5.3 MB/s \n","\u001b[?25hRequirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from flake8>=3.7.9->visualdl->paddlenlp) (4.6.1)\n","Collecting mccabe<0.7.0,>=0.6.0\n","  Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)\n","Requirement already satisfied: Jinja2<3.0,>=2.10.1 in /usr/local/lib/python3.7/dist-packages (from flask>=1.1.1->visualdl->paddlenlp) (2.11.3)\n","Requirement already satisfied: click<8.0,>=5.1 in /usr/local/lib/python3.7/dist-packages (from flask>=1.1.1->visualdl->paddlenlp) (7.1.2)\n","Requirement already satisfied: itsdangerous<2.0,>=0.24 in /usr/local/lib/python3.7/dist-packages (from flask>=1.1.1->visualdl->paddlenlp) (1.1.0)\n","Requirement already satisfied: Werkzeug<2.0,>=0.15 in /usr/local/lib/python3.7/dist-packages (from flask>=1.1.1->visualdl->paddlenlp) (1.0.1)\n","Requirement already satisfied: Babel>=2.3 in /usr/local/lib/python3.7/dist-packages (from Flask-Babel>=1.0.0->visualdl->paddlenlp) (2.9.1)\n","Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from Flask-Babel>=1.0.0->visualdl->paddlenlp) (2018.9)\n","Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from Jinja2<3.0,>=2.10.1->flask>=1.1.1->visualdl->paddlenlp) (2.0.1)\n","Requirement already satisfied: future>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from bce-python-sdk->visualdl->paddlenlp) (0.16.0)\n","Collecting pycryptodome>=3.8.0\n","  Downloading pycryptodome-3.10.1-cp35-abi3-manylinux2010_x86_64.whl (1.9 MB)\n","\u001b[K     |████████████████████████████████| 1.9 MB 45.9 MB/s \n","\u001b[?25hRequirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->flake8>=3.7.9->visualdl->paddlenlp) (3.7.4.3)\n","Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->flake8>=3.7.9->visualdl->paddlenlp) (3.5.0)\n","Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->visualdl->paddlenlp) (1.3.1)\n","Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->visualdl->paddlenlp) (2.8.1)\n","Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->visualdl->paddlenlp) (2.4.7)\n","Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->visualdl->paddlenlp) (0.10.0)\n","Collecting virtualenv>=20.0.8\n","  Downloading virtualenv-20.6.0-py2.py3-none-any.whl (5.3 MB)\n","\u001b[K     |████████████████████████████████| 5.3 MB 41.9 MB/s \n","\u001b[?25hCollecting identify>=1.0.0\n","  Downloading identify-2.2.11-py2.py3-none-any.whl (98 kB)\n","\u001b[K     |████████████████████████████████| 98 kB 7.1 MB/s \n","\u001b[?25hRequirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from pre-commit->visualdl->paddlenlp) (0.10.2)\n","Collecting nodeenv>=0.11.1\n","  Downloading nodeenv-1.6.0-py2.py3-none-any.whl (21 kB)\n","Collecting pyyaml>=5.1\n","  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n","\u001b[K     |████████████████████████████████| 636 kB 50.9 MB/s \n","\u001b[?25hCollecting cfgv>=2.0.0\n","  Downloading cfgv-3.3.0-py2.py3-none-any.whl (7.3 kB)\n","Requirement already satisfied: filelock<4,>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from virtualenv>=20.0.8->pre-commit->visualdl->paddlenlp) (3.0.12)\n","Collecting distlib<1,>=0.3.1\n","  Downloading distlib-0.3.2-py2.py3-none-any.whl (338 kB)\n","\u001b[K     |████████████████████████████████| 338 kB 48.6 MB/s \n","\u001b[?25hCollecting backports.entry-points-selectable>=1.0.4\n","  Downloading backports.entry_points_selectable-1.1.0-py2.py3-none-any.whl (6.2 kB)\n","Collecting platformdirs<3,>=2\n","  Downloading platformdirs-2.0.2-py2.py3-none-any.whl (10 kB)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->visualdl->paddlenlp) (1.24.3)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->visualdl->paddlenlp) (2021.5.30)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->visualdl->paddlenlp) (2.10)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->visualdl->paddlenlp) (3.0.4)\n","Building wheels for collected packages: seqeval\n","  Building wheel for seqeval (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16181 sha256=1a31931646fbd84f559764b95f146c3b0dfae2c5802331f75f4c16c055bc6a37\n","  Stored in directory: /root/.cache/pip/wheels/05/96/ee/7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7\n","Successfully built seqeval\n","Installing collected packages: platformdirs, distlib, backports.entry-points-selectable, virtualenv, pyyaml, pyflakes, pycryptodome, pycodestyle, nodeenv, mccabe, identify, cfgv, shellcheck-py, pre-commit, Flask-Babel, flake8, bce-python-sdk, visualdl, seqeval, colorlog, colorama, paddlenlp\n","  Attempting uninstall: pyyaml\n","    Found existing installation: PyYAML 3.13\n","    Uninstalling PyYAML-3.13:\n","      Successfully uninstalled PyYAML-3.13\n","Successfully installed Flask-Babel-2.0.0 backports.entry-points-selectable-1.1.0 bce-python-sdk-0.8.61 cfgv-3.3.0 colorama-0.4.4 colorlog-5.0.1 distlib-0.3.2 flake8-3.9.2 identify-2.2.11 mccabe-0.6.1 nodeenv-1.6.0 paddlenlp-2.0.6 platformdirs-2.0.2 pre-commit-2.13.0 pycodestyle-2.7.0 pycryptodome-3.10.1 pyflakes-2.3.1 pyyaml-5.4.1 seqeval-1.2.2 shellcheck-py-0.7.2.1 virtualenv-20.6.0 visualdl-2.2.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"uKNNqe8UErKN"},"source":["<a name=\"1\"></a>\n","# Cosine similarity\n","\n","<a name=\"1-1\"></a>\n","### 1.1 Concept and formula\n","\n","There are several ways to measure the similarity or dissimilarity between two embeddings (which in essessence are vectors), such as calculating the distance between two vectors in a given vector space, but cosine similarity remains the most poopular choice among other things. The formula for coin similarity is given as follows:\n","\n","$$CosineSim((\\vec{u}), \\vec{v}) = cos(\\theta) = \\frac{\\vec{u}\\cdot \\vec{v} }{\\vert\\vec{u}\\vert * \\vert \\vec{v}\\vert} \\tag{1}$$\n","\n","\n","where $\\vec{u}\\cdot \\vec{v}$ is the inner/dot product of vector $\\vec{u}$ and vector $\\vec{v}$. \n","\n","<br>\n","\n","Cosine similarity basically can be seen as representing the cosine value of the angel between the two vectors $\\vec{u}\\cdot \\vec{v}$. The bigger the cosine value, the smaller the angel. (To get the $\\theta$, we can the inverse function of cosine, namely: $\\theta = arcos (\\frac{\\vec{u}\\cdot \\vec{v} }{\\vert\\vec{u}\\vert * \\vert \\vec{v}\\vert})$). The basic interpretation of for the results of cosine similarity formula can be illustrated in the figure below:\n","\n","![](https://datascience-enthusiast.com/figures/cosine_sim.png)\n"]},{"cell_type":"markdown","metadata":{"id":"IXqi2aw0h-fv"},"source":["<a name=\"1-2\"></a>\n","### 1.2 Calculating cosine similarity in numpy\n","\n","\n","In numpy, the dot product of two vectors, two matrices or one vector and one matrix can be calculated by using `numpy.dot` or the special operator `@`, i.e. (suppose $\\vec{u}$ and $vec{v}$ are both a (n, 1) dimensional vector):\n","\n","\n","```python\n","import numpy as np\n","\n",">>> dot_prod = np.dot(u.T, v)\n","# Or:\n",">>> dot_prod = u.T @ v\n","```\n","\n","The norm of a vector can be calculated in the following two ways:\n","\n","```python\n",">>> norm_u, norm_v = np.sqrt(np.dot(u.T, u)), np.sqrt(np.dot(v.T, v))\n","# Or\n",">>> norm_u, norm_v = np.linalg.norm(u, ord=2), np.linalg.norm(v, ord=2)\n","```\n","\n","<br>\n","\n","However, using `np.dot` is less powerful compared to its equivalent form given by the following equation (2) because if the vector is instead (1, n) dimensional, we will get an unwanted matrix of (n, n) dimensions but not a scalar value:\n","\n","\n","$$\\vec{u}\\cdot \\vec{v} = \\sum_{i=1}^{n} u_i * v_i \\tag{2}$$\n","\n","We thus can use euqation (2) to rewrite the dot_prod and (one of the) norm_u/v given above as follows:\n","\n","```python\n",">>> dot_prod = np.sum(u * v)\n",">>> norm_u, norm_v = np.sqrt(np.sum(v * v)), np.sqrt(np.sum(v * v)\n","```\n","\n","<br>\n","\n","Taken together, we can define a consine similarity function in numpy as follows:\n","\n","```python\n","import numpy as np\n","\n","def cosine_similarity(u, v):\n","  return np.sum(u * v) / (np.sqrt(np.sum(u * u)) * np.sqrt(np.sum(v * v)))\n","```"]},{"cell_type":"markdown","metadata":{"id":"H8GN16A0sodG"},"source":["<a name=\"2\"></a>\n","# 2. Calculating similarity between two words\n","\n","- First, we need to get the embeddings of two words, which can be done by utilizing `padddlenlp.embeddings.TokenEmbedding` class, as described in [loading pre-trained word embedding in paddlenp.ipynb](https://colab.research.google.com/drive/1WSyYtDiHwXe4MFTwe_X6hQ5atBqNsFax?usp=sharing). \n","- Then, we can use the `cosine_similarity` function built above to get the cosine similarity value.\n","\n","<br>\n","\n","In paddlenlp, however, we can also use an inbuilt `cosine_sim` function inside the `TokenEmbedding` class to calculate the cosine similarity between two words. <ins>But please note that, this function cannot be used to measure the cosine similarity between two texts.</ins>\n","\n","<br>\n","\n","**Reference**: [TokenEmbedding](https://paddlenlp.readthedocs.io/zh/latest/source/paddlenlp.embeddings.token_embedding.html) \n","\n","**Functions of interest inside `TokenEmbedding`**:\n","- **`vocab.to_tokens(indices)`**: To get the vocabulary of the pre-trained model\n","  - indices: `list` or `int`. \n","  - Rerturns: corresponding tokens in the vocabulary of the model.\n","- **`search(words)`**: To get the pre-trained embedding of a given word/token\n","  - words: `list` or `str` or `int`. \n","  - Retruns: the vectors of specified words.\n","- **`cosine_sim(word_a, word_b)`**: Cosine simlarity between two words (**!! words, NOT texts**)\n","  - word_a (str), word_b (str) -- The first word string.\n","  - Returns: The cosine similarity value of the 2 words."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Vkf9qztvqRNC","executionInfo":{"status":"ok","timestamp":1626807345893,"user_tz":360,"elapsed":106526,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"3bd8b115-c4cb-4613-9fa9-ddb7eba01f56"},"source":["# import padddlenlp.embeddings.TokenEmbedding\n","from paddlenlp.embeddings import TokenEmbedding\n","\n","\n","# model name: w2v.baidu_encyclopedia.target.word-character.char1-4.dim300\t\n","# model size: 679.51 MB, vocab size: 636038\n","token_embedding = TokenEmbedding(embedding_name=\"w2v.baidu_encyclopedia.target.word-character.char1-4.dim300\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["100%|██████████| 695822/695822 [01:18<00:00, 8878.92it/s] \n","\u001b[32m[2021-07-20 18:55:38,829] [    INFO]\u001b[0m - Loading token embedding...\u001b[0m\n","\u001b[32m[2021-07-20 18:55:45,778] [    INFO]\u001b[0m - Finish loading embedding vector.\u001b[0m\n","\u001b[32m[2021-07-20 18:55:45,780] [    INFO]\u001b[0m - Token Embedding info:             \n","Unknown index: 636036             \n","Unknown token: [UNK]             \n","Padding index: 636037             \n","Padding token: [PAD]             \n","Shape :[636038, 300]\u001b[0m\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"mZaAzu-Fwb6T","executionInfo":{"status":"ok","timestamp":1626664995638,"user_tz":360,"elapsed":654,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"f59144b0-8a62-4c63-ec52-8fc39b213a51"},"source":["# # uncomment if you are interested \n","# # check the vocab\n","# vocab = token_embedding.vocab.to_tokens(list(range(636038)))\n","# # print 50 randomly selected words from the vocab\n","# print(vocab[1000:1050])"],"execution_count":null,"outputs":[{"output_type":"stream","text":["['普通', '形象', '客户', '容易', '),', '那些', '级', '现任', '1953', '出生', '仍', '因素', 'g', '中央', '计算', '90', '平台', '迅速', '400', '而是', '行政', '举办', '运行', '23', '----', '若', '自由', '设施', '*', '所谓', '更加', '感觉', '1937', '训练', '副书记', '亿元', '演员', '此外', '1938', '论', '强', '成果', '几乎', '0', '调整', '自然保护区', '1954', '最终', '科研', '1500']\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"IvLFQ4Hfxp6G"},"source":["# # uncomment if you are interested \n","# # Try to get the embeddings of any given words, \n","# # say cat = 猫, dog = 狗, human = 人\n","\n","# cat_em, dog_em, human_em = token_embedding.search(['猫', '狗', '人'])\n","\n","# # check the embedding for cat \n","# print('This is the word embedding for cat in the chosen model:\\n\\n', cat_em)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ZrEiHg7hzYqT","executionInfo":{"status":"ok","timestamp":1626665315004,"user_tz":360,"elapsed":126,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"ace2a3d6-17cc-4ee9-80a5-0defbb126680"},"source":["# calculate the cosine similary between two words, say, cat and dog versus cat and human\n","# we would expecr that cat is more simlar to dog than cat to human\n","\n","cos_sim1 = token_embedding.cosine_sim('猫', '狗')\n","cos_sim2 = token_embedding.cosine_sim('猫', '人')\n","print('This is the cosine similarity score for cat and dog: ', cos_sim1)\n","print('This is the cosine similarity score for cat and human: ', cos_sim2)\n","print('Is cos_sim1 greater than cos_sim2 as expected?: ', cos_sim1 > cos_sim2)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["This is the cosine similarity score for cat and dog:  0.77376914\n","This is the cosine similarity score for cat and human:  0.36149243\n","Is cos_sim1 greater than cos_sim2 as expected?:  True\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"nvgXcdyXzul-","executionInfo":{"status":"ok","timestamp":1626666189619,"user_tz":360,"elapsed":137,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"c25ee7d2-3112-4a83-ef1f-38f5a10db075"},"source":["# # # uncomment if you are interested (you need to uncomment the above cells as well)\n","# # Let's also check whether the cosine_similarity function we built above is correct\n","\n","# import numpy as np\n","\n","# def cosine_similarity(u, v):\n","#   return np.sum(u * v) / (np.sqrt(np.sum(u * u)) * np.sqrt(np.sum(v * v)))\n","\n","\n","# cat_dog_sim = cosine_similarity(cat_em, dog_em)\n","# cat_human_sim = cosine_similarity(cat_em, human_em)\n","# assert cos_sim1==cat_dog_sim, f'A bug here! cat_dog_sim({cat_dog_sim}) != {cos_sim1}'\n","# assert cos_sim2==cat_human_sim, f'A bug here! cat_dog_sim({cat_human_sim}) != {cos_sim2}'\n","# print('Your function works! Congratulations!')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Your function works! Congratulations!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Lh81LI123Oft"},"source":["<a name=\"3\"></a>\n","# 3. Calculating similarity between two sentences\n","\n","<a name=\"3-1\"></a>\n","### 3.1 The Bag-of-Words model\n","\n","- To calculate the cosine similarity between two sentences, we need first to calculate the embeddings of the sentence pair(s) of interest. \n","- A sentence's (or a larger text) embedding can usually be seen as a summation of the embeddings of the words/tokens in it. This is also known as [Bag-of-Words model](https://en.wikipedia.org/wiki/Bag-of-words_model). \n","- Since `cosine_sim` function provided by `padddlenlp.embeddings.TokenEmbedding` only applys to a pair of words, we need to use the `cosine_similarity` function instead. \n","\n","<br>\n","\n","---\n","\n","<br>\n","\n","In short, we can calculate the cosine similarity between two sentences in the following five steps:\n","\n","- Tokenize a sentences into a lisr of words/tokens (use [`jieba`](https://github.com/fxsjy/jieba));\n","- Retrieve the embeddings of these words (use `search` function inside `TokenEmbedding` class, see above); \n","- Sum up these embeddings to get the sentence's embedding;\n","- Do the same thing to get the of the embedding other sentence;\n","- Calculate the cosine similarity between the two sentences (use the `cosine_similarity` defined above). \n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"5Gw_wMgv2GsR","executionInfo":{"status":"ok","timestamp":1626766965232,"user_tz":360,"elapsed":3208,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"c7f64d6c-1977-486a-bad0-90237c3862ff"},"source":["# first install jieba \n","!pip3 install jieba"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: jieba in /usr/local/lib/python3.7/dist-packages (0.42.1)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"5lpqhrry-27U","executionInfo":{"status":"ok","timestamp":1626807448234,"user_tz":360,"elapsed":1071,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"9e5ddb6b-f902-486b-d46a-e4f7b314abd8"},"source":["# use jieba.lcut as the Chinese tokenizer\n","# you can also the inbuilt JiebaTokenizer in paddlenlp\n","# which can directly load vocab from ''token_embedding'\n","# from paddlenlp.data import JiebaTokenizer\n","# tokenizer = JiebaTokenizer(vocab=token_embedding.vocab)\n","\n","\n","import jieba\n","\n","exmp_sent = '我来自中国' # I = 我 [come from] = 来自 China = 中国\n","jieba.lcut(exmp_sent) # This will return a list of tokens directly"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Building prefix dict from the default dictionary ...\n","Dumping model to file cache /tmp/jieba.cache\n","Loading model cost 1.025 seconds.\n","Prefix dict has been built successfully.\n"],"name":"stderr"},{"output_type":"execute_result","data":{"text/plain":["['我', '来自', '中国']"]},"metadata":{"tags":[]},"execution_count":4}]},{"cell_type":"code","metadata":{"id":"_Z8UaCtnACMe"},"source":["import numpy as np\n","\n","# then define a function to get sentence embedding \n","\n","\n","def sentence_embedding(sentence, tokenizer=jieba.lcut, embedder=token_embedding):\n","  '''Return sentence embedding(s) given a sentence or a list of sentences. \n","\n","  Arguments:\n","    sentence: str or list of str\n","    tokenizer: a tokenizer function that returns a list of tokens. Defaults to \n","      jieba.lcut. \n","    embedder: defaults to padddlenlp.embeddings.TokenEmbedding as we defined above. \n","  Returns: sentence embedding(s)\n","  '''\n","  def embedding(sent):\n","    tokens = tokenizer(sent)\n","    return np.sum(embedder.search(tokens), axis=0) # sum vertically\n","\n","  if isinstance(sentence, str):\n","    return embedding(sentence)\n","  elif isinstance(sentence, list):\n","    return [embedding(sent) for sent in sentence]\n","  else:\n","    raise TypeError(f'sentence should be either a str or a list.' \n","    '{type(sentence)} not supported. ')\n","\n","\n","def cosine_similarity(u, v):\n","  return np.sum(u * v) / (np.sqrt(np.sum(u * u)) * np.sqrt(np.sum(v * v)))\n","\n","\n","def sentence_cosine_sim(sent_a, sent_b, tokenizer=jieba.lcut, embedder=token_embedding):\n","  '''Returns the cosine similarity between two sentences.\n","\n","  Arguments:\n","    sent_a, sent_b: str\n","  Returns:\n","    cosine simlarity between two sentence embeddings\n","  '''\n","\n","  sent_a_em, sent_b_em = sentence_embedding([sent_a, sent_b], tokenizer, embedder)\n","  return cosine_similarity(sent_a_em, sent_b_em)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"179H7KrfCChy","executionInfo":{"status":"ok","timestamp":1626807458518,"user_tz":360,"elapsed":120,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"9cb1ff40-06f0-4ba3-d871-07445dcf4fa2"},"source":["# Let's calculate two made-up sentences!\n","# sent_a = I enjoy learning deep learning\n","# sent_b = I am interested in learning deep learning\n","\n","sent_a, sent_b = '我喜欢学习深度学习', '我对学习深度学习很有兴趣'\n","sentence_cosine_sim(sent_a, sent_b)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.94084466"]},"metadata":{"tags":[]},"execution_count":6}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"YEhd-fH6CClN","executionInfo":{"status":"ok","timestamp":1626671044969,"user_tz":360,"elapsed":213,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"41186773-b553-4b26-cdd2-591abb66e7cb"},"source":["# These two sentences have very different structures,\n","# and there are also five non-overlapping words (喜欢, 对, 很, 有, 兴趣),\n","# but the cosine similarity socre is over 0.94! Nice work!\n","\n","print('Tokens of Sentence_a:', jieba.lcut(sent_a))\n","print('Tokens of Sentence_b:', jieba.lcut(sent_b))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Tokens of Sentence_a: ['我', '喜欢', '学习', '深度', '学习']\n","Tokens of Sentence_b: ['我', '对', '学习', '深度', '学习', '很', '有', '兴趣']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"eSQVoviIJoS0"},"source":["<br>\n","\n","**Side project: Calculating the cosine similarity between two English sentences**\n","\n","<br>\n","\n","As the [TokenEmbedding API](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/embeddings.html) in `paddlenlp` also includes some pre-trained English word embedding models (see [loading pre-trained word embedding in paddlenp.ipynb](https://colab.research.google.com/drive/1WSyYtDiHwXe4MFTwe_X6hQ5atBqNsFax?usp=sharing)), we can easily calculate the cosine similarity score of two English sentences using the function we just built. "]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"n4Mshdh7KscD","executionInfo":{"status":"ok","timestamp":1626807537513,"user_tz":360,"elapsed":75057,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"3cfcdb16-d640-458e-b385-fb287720c227"},"source":["# model name: glove.wiki2014-gigaword.target.word-word.dim300.en\n","# model size: 422.83 MB; vocab size: 400002\n","\n","tk_embedding_en = TokenEmbedding(embedding_name='glove.wiki2014-gigaword.target.word-word.dim300.en')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["100%|██████████| 432979/432979 [00:58<00:00, 7338.69it/s]\n","\u001b[32m[2021-07-20 18:58:53,383] [    INFO]\u001b[0m - Loading token embedding...\u001b[0m\n","\u001b[32m[2021-07-20 18:58:57,444] [    INFO]\u001b[0m - Finish loading embedding vector.\u001b[0m\n","\u001b[32m[2021-07-20 18:58:57,452] [    INFO]\u001b[0m - Token Embedding info:             \n","Unknown index: 400000             \n","Unknown token: [UNK]             \n","Padding index: 400001             \n","Padding token: [PAD]             \n","Shape :[400002, 300]\u001b[0m\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"vz-N0ucSLQmw","executionInfo":{"status":"ok","timestamp":1626807537514,"user_tz":360,"elapsed":23,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"8de14294-c071-4143-cbc9-8460ec52912e"},"source":["# The cosine similarity score is over 0.87! Nice work!\n","sent_a, sent_b = 'I enjoy learning deep learning', 'I am interested in learning deep learning'\n","en_tokenizer = lambda x: x.split() # here, we simply use space to tokenize English text\n","\n","sentence_cosine_sim(sent_a, sent_b, en_tokenizer, tk_embedding_en)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.8740044"]},"metadata":{"tags":[]},"execution_count":8}]},{"cell_type":"markdown","metadata":{"id":"JG77RVdy1rq7"},"source":["<a name=\"3-2\"></a>\n","### 3.2 Real-Wolrd examples: the performances of the Bag-of-Words model\n","\n","We have just used two pre-trained word embedding models (one in Chinese, one in English) to build a Bag-of-Words model that can calculate the cosine similarity between two sentences. **The results of the two made-up sentences above look nice, but how well can the models we built work for real-world datasets?** Let's give a casual try! \n","\n","<br>\n","\n","<a name=\"3-2-1\"></a>\n","#### 3.2.1 Chinese examples\n","\n","Here, we will use 50 examples extracted from the Large-scale Chinese Question Matching Corpus ([lcqmc](https://aclanthology.org/C18-1166/)) to see how well the simple Bog-of-Words model can give us clues regarding the similarity between Chinese sentences pairs. In this dataset, a similar sentence pair is labelled with \"1\" whereas a dissimilar sentence pari is labelled with \"0\". You can download the file [lcqmc_sample.tsv](https://drive.google.com/file/d/1-HVs8IEYqXX9Z63zLFODYETucW35Eq_T/view?usp=sharing) here.\n","\n","<br>\n","\n","**More on how to handle external files in Colab can be found [here](https://colab.research.google.com/notebooks/io.ipynb).**"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"kDan05ZJ6VVZ","executionInfo":{"status":"ok","timestamp":1626808666561,"user_tz":360,"elapsed":1163,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"4255e0c7-868e-469e-b27c-e1e86edb3555"},"source":["from google.colab import drive\n","\n","drive.mount('/drive', force_remount=True)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Mounted at /drive\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"4k0EYLjo7H2_"},"source":["# define a data loader that will load the dataset \n","# in the structure of [sent_a, sent_b, (similarity) label]\n","\n","def data_loader(filepath):\n","  data = open(filepath, 'r')\n","  out = [] # output\n","  for line in data:\n","    line = line.split('\\t')\n","    sent_a, sent_b, label = line[0], line[1], line[2].strip()\n","    out.append([sent_a, sent_b, label])\n","\n","  return out \n","\n","# the compare() function will calculate the cosine similarity between sentence paris\n","# and print them out along with the senntence pairs and given labels\n","# tokenizer= and embedder allows you to switch from Chinese model to English model.  \n","def compare(data, tokenizer=jieba.lcut, embedder=token_embedding):\n","  for item in data:\n","    sent_a, sent_b, label = item\n","    sim_score = sentence_cosine_sim(sent_a, sent_b, tokenizer=tokenizer, embedder=embedder)\n","    print('sent_a: ', sent_a)\n","    print('sent_b: ', sent_b)\n","    print('similarity score: ', sim_score)\n","    print('Given label: ', label)\n","    print()"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"rqNLmnox-7Jn","executionInfo":{"status":"ok","timestamp":1626808370000,"user_tz":360,"elapsed":152,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"6371470f-05ce-45d2-db66-b8747235d0ad"},"source":["file_path1 = '/drive/My Drive/lcqmc_sample.tsv'\n","data1 = data_loader(file_path1)\n","print('first five examples:\\n')\n","for item in data1[:5]:\n","  print(item)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["first five examples:\n","\n","['喜欢打篮球的男生喜欢什么样的女生', '爱打篮球的男生喜欢什么样的女生', '1']\n","['我手机丢了，我想换个手机', '我想买个新手机，求推荐', '1']\n","['大家觉得她好看吗', '大家觉得跑男好看吗？', '0']\n","['求秋色之空漫画全集', '求秋色之空全集漫画', '1']\n","['晚上睡觉带着耳机听音乐有什么害处吗？', '孕妇可以戴耳机听音乐吗?', '0']\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"5xazd7gKAczo","executionInfo":{"status":"ok","timestamp":1626808388289,"user_tz":360,"elapsed":398,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"e2705d11-89da-4c1b-d4b8-3a61b0ea491d"},"source":["# It appears that this raw model does well on similar sentence pairs, but not dissimilar ones!\n","# The raw model needs to be further trained in order to make more accurate predictions \n","compare(data1)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["sent_a:  喜欢打篮球的男生喜欢什么样的女生\n","sent_b:  爱打篮球的男生喜欢什么样的女生\n","similarity score:  0.9883471\n","Given label:  1\n","\n","sent_a:  我手机丢了，我想换个手机\n","sent_b:  我想买个新手机，求推荐\n","similarity score:  0.7669853\n","Given label:  1\n","\n","sent_a:  大家觉得她好看吗\n","sent_b:  大家觉得跑男好看吗？\n","similarity score:  0.94471115\n","Given label:  0\n","\n","sent_a:  求秋色之空漫画全集\n","sent_b:  求秋色之空全集漫画\n","similarity score:  0.99999994\n","Given label:  1\n","\n","sent_a:  晚上睡觉带着耳机听音乐有什么害处吗？\n","sent_b:  孕妇可以戴耳机听音乐吗?\n","similarity score:  0.9046495\n","Given label:  0\n","\n","sent_a:  学日语软件手机上的\n","sent_b:  手机学日语的软件\n","similarity score:  0.98783\n","Given label:  1\n","\n","sent_a:  打印机和电脑怎样连接，该如何设置\n","sent_b:  如何把带无线的电脑连接到打印机上\n","similarity score:  0.9338498\n","Given label:  0\n","\n","sent_a:  侠盗飞车罪恶都市怎样改车\n","sent_b:  侠盗飞车罪恶都市怎么改车\n","similarity score:  0.9842851\n","Given label:  1\n","\n","sent_a:  什么花一年四季都开\n","sent_b:  什么花一年四季都是开的\n","similarity score:  0.96798384\n","Given label:  1\n","\n","sent_a:  看图猜一电影名\n","sent_b:  看图猜电影！\n","similarity score:  0.9307659\n","Given label:  1\n","\n","sent_a:  这上面写的是什么？\n","sent_b:  胃上面是什么\n","similarity score:  0.8654834\n","Given label:  0\n","\n","sent_a:  建议您重新注册，辛苦您了。\n","sent_b:  可以的，您注销成功后，可以重新注册的，辛苦您了。\n","similarity score:  0.95875376\n","Given label:  0\n","\n","sent_a:  小草有什么的特点,可以象征什么?\n","sent_b:  小草有什么特点可以象征什么\n","similarity score:  0.9662234\n","Given label:  1\n","\n","sent_a:  校验失败了，\n","sent_b:  您好，您还是访客状态呢\n","similarity score:  0.632664\n","Given label:  0\n","\n","sent_a:  尼玛什么意思\n","sent_b:  尼玛啊是什么意思？\n","similarity score:  0.92650205\n","Given label:  0\n","\n","sent_a:  自找苦吃的地方是哪儿？\n","sent_b:  自找苦吃的地方是哪儿\n","similarity score:  0.9862572\n","Given label:  1\n","\n","sent_a:  尾号4位多少\n","sent_b:  尾号是多少后4位\n","similarity score:  0.9624234\n","Given label:  1\n","\n","sent_a:  谢文东能在哪里看\n","sent_b:  谢文东在哪里看\n","similarity score:  0.9781221\n","Given label:  1\n","\n","sent_a:  新概念英语第二册练习册41课答案\n","sent_b:  新概念英语第二册练习册21练习答案\n","similarity score:  0.9681953\n","Given label:  0\n","\n","sent_a:  过年送礼送什么好\n","sent_b:  过年前什么时候送礼？\n","similarity score:  0.9155301\n","Given label:  0\n","\n","sent_a:  壮丁是什么生肖\n","sent_b:  欲钱买壮丁指的是什么生肖？\n","similarity score:  0.8849146\n","Given label:  0\n","\n","sent_a:  这是什么蔬菜啊\n","sent_b:  这是什么蔬菜呀？\n","similarity score:  0.97229975\n","Given label:  1\n","\n","sent_a:  最近有什么好电视剧看？\n","sent_b:  现在看电影都是有哪些号网站啊\n","similarity score:  0.9040536\n","Given label:  0\n","\n","sent_a:  性与爱有什么区别？\n","sent_b:  性与爱有什么区别\n","similarity score:  0.9811924\n","Given label:  1\n","\n","sent_a:  网上卖的烟有真的吗？\n","sent_b:  网上帮你刷钻是真的吗\n","similarity score:  0.9081539\n","Given label:  0\n","\n","sent_a:  什么印象\n","sent_b:  什么印象了\n","similarity score:  0.9574336\n","Given label:  1\n","\n","sent_a:  面相上看，耳垂上有痣代表什么？\n","sent_b:  男性耳朵上长痣代表什么（耳垂这里）\n","similarity score:  0.9047133\n","Given label:  0\n","\n","sent_a:  男生喜欢女生送他什么礼物呢\n","sent_b:  男生最喜欢女生送什么礼物\n","similarity score:  0.9816338\n","Given label:  1\n","\n","sent_a:  如何快速忘记一个人\n","sent_b:  怎样能快速忘记一个人\n","similarity score:  0.96622247\n","Given label:  1\n","\n","sent_a:  周杰伦女友是谁\n","sent_b:  周杰伦的现女友是谁\n","similarity score:  0.9536592\n","Given label:  1\n","\n","sent_a:  月初和月末是会出现这个延迟的情况的，小二建议您耐心等待一下\n","sent_b:  因周六周日不属于工作日所以建议您耐心等待\n","similarity score:  0.81008536\n","Given label:  0\n","\n","sent_a:  支付宝手机支付密码是什么\n","sent_b:  支付宝里手机支付密码是什么\n","similarity score:  0.9929968\n","Given label:  1\n","\n","sent_a:  关于网页设计的问题\n","sent_b:  关于网页设计的一个问题\n","similarity score:  0.9833492\n","Given label:  1\n","\n","sent_a:  有没有专门为房地产行业做咨询的公司啊？\n","sent_b:  国内咨询公司中哪家的房地产咨询做的比较好啊？\n","similarity score:  0.9593285\n","Given label:  0\n","\n","sent_a:  这系列的漫画叫什么名字\n","sent_b:  这个系列的漫画叫什么名字？\n","similarity score:  0.9863512\n","Given label:  1\n","\n","sent_a:  无线路由器怎么无线上网\n","sent_b:  无线上网卡和无线路由器怎么用\n","similarity score:  0.95629627\n","Given label:  0\n","\n","sent_a:  左眼皮跳是什么原因\n","sent_b:  女的右眼皮跳是什么原因啊？\n","similarity score:  0.9371212\n","Given label:  0\n","\n","sent_a:  部落战争怎么更新？\n","sent_b:  部落战争，怎样更新\n","similarity score:  0.94233054\n","Given label:  1\n","\n","sent_a:  这个空气净化器有用吗？\n","sent_b:  空气刘海是怎么样的\n","similarity score:  0.76361257\n","Given label:  0\n","\n","sent_a:  看图猜成语，这是什么？\n","sent_b:  【看图猜成语】请问这个是？\n","similarity score:  0.91001284\n","Given label:  1\n","\n","sent_a:  找工作网站哪个好？\n","sent_b:  找工作哪个网站好\n","similarity score:  0.9870691\n","Given label:  1\n","\n","sent_a:  问发型，男生。\n","sent_b:  关于男生发型\n","similarity score:  0.8764574\n","Given label:  1\n","\n","sent_a:  安卓手机怎么设置来电自动接听\n","sent_b:  三星手机怎么自动接听\n","similarity score:  0.9166412\n","Given label:  1\n","\n","sent_a:  部落冲突游戏怎么找回\n","sent_b:  部落冲突重置游戏\n","similarity score:  0.8610685\n","Given label:  0\n","\n","sent_a:  怎么才能下载穿越火线\n","sent_b:  穿越火线的挂怎么下载\n","similarity score:  0.953341\n","Given label:  0\n","\n","sent_a:  你点击详情\n","sent_b:  您点击详情\n","similarity score:  0.96551734\n","Given label:  1\n","\n","sent_a:  我要找一个人怎么找\n","sent_b:  我要找一个人\n","similarity score:  0.9457908\n","Given label:  1\n","\n","sent_a:  中国四大古城是哪些？\n","sent_b:  中国的四大古城有哪些?5\n","similarity score:  0.9439541\n","Given label:  1\n","\n","sent_a:  脸上痣太多怎么办？\n","sent_b:  脸上痣太多怎么办\n","similarity score:  0.98747754\n","Given label:  1\n","\n","sent_a:  福州哪家装修公司好\n","sent_b:  转运公司哪家好\n","similarity score:  0.8310891\n","Given label:  0\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"eRFvpCOuRZMO"},"source":["<a name=\"3-2-2\"></a>\n","#### 3.2.2 English examples\n","\n","Here, we will use 50 examples extracted from the Semantic Textual Similarity benchmark ([sts](http://ixa2.si.ehu.eus/stswiki/index.php/Main_Page)) to see how well the simple Bog-of-Words model can give us clues regarding the similarity between English sentences pairs. In this dataset, a similar sentence pair is labelled with a similarity score ranging from 0 to 5, which has been normailzed to [0, 1] in our test sample. You can download the file [sts_sample.tsv](https://drive.google.com/file/d/1TJgl4WtKY4JlcCVZtd9CQK-1w26wNo9D/view?usp=sharing) here."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"-3lCHPouV1eJ","executionInfo":{"status":"ok","timestamp":1626808717437,"user_tz":360,"elapsed":1403,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"7c4c38c5-3251-450b-bf39-854a11a86081"},"source":["drive.mount('/drive', force_remount=True)\n","\n","file_path2 = '/drive/My Drive/sts_sample.tsv'\n","data2 = data_loader(file_path2)\n","print('first five examples:\\n')\n","for item in data2[:5]:\n","  print(item)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Mounted at /drive\n","first five examples:\n","\n","['A multi-colored bird clings to a wire fence.', 'A bird holding on to a metal gate.', '0.64']\n","['A woman is mixing meat.', 'A woman is feeding a man.', '0.2']\n","['Two men sailing in a small sailboat.', 'Two brown horses standing in grassy field.', '0.0']\n","['the united states and other nato members have refused to do ratify the updates cfe.', 'the united states and other nato members have refused to ratify the amended treaty. ', '0.64']\n","[\"Thieves steal Channel swimmer's wheelchair\", \"Thieves snatch English Channel swimmer's custom-made wheelchair\", '0.8']\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"297cmPjIW0ly","executionInfo":{"status":"ok","timestamp":1626808961624,"user_tz":360,"elapsed":280,"user":{"displayName":"Jack Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiQ21gEldWUHBSnZER8H46ttdc4XkKI1LW2QpkDzg=s64","userId":"06786829715817144955"}},"outputId":"1fbda641-003b-4697-e693-5579b1c72dd8"},"source":["#Again, it appears that this raw model does well on similar sentence pairs, but not dissimilar ones!\n","# The raw model needs to be further trained in order to make more accurate predictions \n","en_tokenizer = lambda x: x.split() # here, we simply use space to tokenize English text\n","compare(data2, en_tokenizer, tk_embedding_en)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["sent_a:  A multi-colored bird clings to a wire fence.\n","sent_b:  A bird holding on to a metal gate.\n","similarity score:  0.7458005\n","Given label:  0.64\n","\n","sent_a:  A woman is mixing meat.\n","sent_b:  A woman is feeding a man.\n","similarity score:  0.81256783\n","Given label:  0.2\n","\n","sent_a:  Two men sailing in a small sailboat.\n","sent_b:  Two brown horses standing in grassy field.\n","similarity score:  0.62510234\n","Given label:  0.0\n","\n","sent_a:  the united states and other nato members have refused to do ratify the updates cfe.\n","sent_b:  the united states and other nato members have refused to ratify the amended treaty. \n","similarity score:  0.980547\n","Given label:  0.64\n","\n","sent_a:  Thieves steal Channel swimmer's wheelchair\n","sent_b:  Thieves snatch English Channel swimmer's custom-made wheelchair\n","similarity score:  0.69551796\n","Given label:  0.8\n","\n","sent_a:  Men are falling into a pool.\n","sent_b:  People flip into a swimming pool.\n","similarity score:  0.7382532\n","Given label:  0.55\n","\n","sent_a:  Fire destroys Tibetan town in China\n","sent_b:  Fire destroys ancient Tibetan town in China\n","similarity score:  0.9057428\n","Given label:  0.8800000000000001\n","\n","sent_a:  It's also a strategic win for Overture, given that Knight Ridder had been using Google's advertising services.\n","sent_b:  It's also a strategic win for Overture, given that Knight Ridder had the option of signing on Google's services.\n","similarity score:  0.9442394\n","Given label:  0.8\n","\n","sent_a:  Obama mulls limited military action in Syria\n","sent_b:  Obama made last-minute decision on Syria approval\n","similarity score:  0.57910675\n","Given label:  0.4\n","\n","sent_a:  A woman is riding a horse.\n","sent_b:  A woman rides a horse.\n","similarity score:  0.8838857\n","Given label:  1.0\n","\n","sent_a:  A man is tying a noose.\n","sent_b:  A man is preparing a hanging rope.\n","similarity score:  0.8802036\n","Given label:  0.9334\n","\n","sent_a:  A brown dog carrying a Frisbee runs through the snow.\n","sent_b:  A skier speeds through the snow.\n","similarity score:  0.60724825\n","Given label:  0.04\n","\n","sent_a:  The man spanked the other man with a stick.\n","sent_b:  The man hit the other man with a stick.\n","similarity score:  0.9601374\n","Given label:  0.8400000000000001\n","\n","sent_a:  Australian researchers believe they have found a trigger of type 1 diabetes in children - their mothers eating potatoes and other tuberous vegetables during pregnancy.\n","sent_b:  Women who eat potatoes and other tuberous vegetables during pregnancy may be at risk of triggering type 1 diabetes in their children, Melbourne researchers believe.\n","similarity score:  0.9639508\n","Given label:  0.85\n","\n","sent_a:  A key player in former state Treasurer Paul Silvester's corruption scheme testified on Tuesday about kickbacks and bribes Silvester traded for state business.\n","sent_b:  A key figure in former state Treasurer Paul Silvester's bribery scheme was accused Wednesday of changing his story about Silvester's alleged corrupt dealings with a Boston investment firm.\n","similarity score:  0.90715486\n","Given label:  0.4\n","\n","sent_a:  indian troops already fighting rebels are now also engaged in a massive operation to destroy poppy crops in kashmir.\n","sent_b:  indian troops already fighting rebels in kashmir are now engaged in a massive campaign to destroy poppy crops in the region. \n","similarity score:  0.9830573\n","Given label:  1.0\n","\n","sent_a:  A boy holds a net with a snake in it close to the ground.\n","sent_b:  A black dog is walking on the grass with a stick in it's mouth.\n","similarity score:  0.87802607\n","Given label:  0.0\n","\n","sent_a:  The ideology was built around violent expansion and race superiority.\n","sent_b:  The ideology was built around violent expansion, so yes - VIOLENT expansion.\n","similarity score:  0.8893103\n","Given label:  0.8\n","\n","sent_a:  Australian PM defiant under pressure over leadership\n","sent_b:  Australia can eclipse England in Windies\n","similarity score:  0.41926008\n","Given label:  0.08\n","\n","sent_a:  The young boy is jumping into the pool on his belly.\n","sent_b:  a boy jumps into the blue pool water.\n","similarity score:  0.89565516\n","Given label:  0.6799999999999999\n","\n","sent_a:  Iran, six world powers clinch breakthrough nuclear deal\n","sent_b:  Iran, six global powers sign landmark nuclear deal\n","similarity score:  0.8872206\n","Given label:  1.0\n","\n","sent_a:  A woman is applying eye liner.\n","sent_b:  A man is playing a flute.\n","similarity score:  0.7270311\n","Given label:  0.0\n","\n","sent_a:  The dog ran in the water at the beach.\n","sent_b:  The man drove his little red car around the traffic.\n","similarity score:  0.8025419\n","Given label:  0.0\n","\n","sent_a:  The technology-laced Nasdaq Composite Index .IXIC was up 7.42 points, or 0.45 percent, at 1,653.44.\n","sent_b:  The broader Standard & Poor's 500 Index <.SPX> was 0.46 points lower, or 0.05 percent, at 997.02.\n","similarity score:  0.8651397\n","Given label:  0.1778\n","\n","sent_a:  Congress twice passed similar bills, but then-President Clinton vetoed them both times.\n","sent_b:  Congress twice before passed similar restrictions only to see President Bill Clinton veto them.\n","similarity score:  0.9087729\n","Given label:  0.9199999999999999\n","\n","sent_a:  A bird standing on top of a wooden fencepost.\n","sent_b:  A buddha statue sits on a table with a wooden table runner.\n","similarity score:  0.8423585\n","Given label:  0.08\n","\n","sent_a:  Bangladesh: Fugitive Sentenced to Death by War Crimes Tribunal\n","sent_b:  Bangladesh Opposition Leader Sentenced to Death for War Crimes\n","similarity score:  0.77602375\n","Given label:  0.72\n","\n","sent_a:  Enron's stock comprised as much as 61% of the workers' 401(k) portfolios.\n","sent_b:  Enron's 401(k) plan covered about 20,000 workers, retirees and beneficiaries.\n","similarity score:  0.67108893\n","Given label:  0.5\n","\n","sent_a:  western countries' failure to sufficiently compensate libya will deter iran and north korea from disarming.\n","sent_b:  kadhafi warns that us and britain's failure to compensate libya for nuclear disarmament will deter iran and north korea from disarming. \n","similarity score:  0.94981986\n","Given label:  0.6799999999999999\n","\n","sent_a:  His first Father's Day as a dad, his first major as a champion.\n","sent_b:  Jim Furyk celebrated his first Father's Day as a father by winning his first major golf championship.\n","similarity score:  0.9235847\n","Given label:  0.6\n","\n","sent_a:  It's also a stellar performer in terms of blithering idiocy.\n","sent_b:  Atlas Shrugged It's also a stellar performer in terms of blithering idiocy.\n","similarity score:  0.99959546\n","Given label:  0.72\n","\n","sent_a:  A woman is cutting a lemon.\n","sent_b:  A man is mowing a lawn.\n","similarity score:  0.85009885\n","Given label:  0.04\n","\n","sent_a:  India voters kick off world‚Äôs biggest election\n","sent_b:  India starts voting in world's largest election\n","similarity score:  0.76879835\n","Given label:  1.0\n","\n","sent_a:  A cruise ship docked at a coast.\n","sent_b:  A large cruise ship in harbor with trees behind.\n","similarity score:  0.77899265\n","Given label:  0.6799999999999999\n","\n","sent_a:  So which system isn't working for us?\n","sent_b:  Tell me exactly which system isn't working right now?\n","similarity score:  0.85461843\n","Given label:  0.8\n","\n","sent_a:  Bombs in Thailand kill 14, wound 340\n","sent_b:  Bombs in southern Thailand kill 5, wound 50\n","similarity score:  0.86454767\n","Given label:  0.6\n","\n","sent_a:  Three cows grazing in a field.\n","sent_b:  Three cows are grazing in a grassy field.\n","similarity score:  0.9263041\n","Given label:  0.9199999999999999\n","\n","sent_a:  U.S. drone strikes kill 16 suspected militants in Yemen\n","sent_b:  US drone strike kills 5 militants in Pakistan\n","similarity score:  0.92309624\n","Given label:  0.36\n","\n","sent_a:  Five Killed in Blasts in India\n","sent_b:  Twelve killed as gunmen raid village\n","similarity score:  0.49369946\n","Given label:  0.04\n","\n","sent_a:  Fundamental difference?\n","sent_b:  Fundimental difference, please?\n","similarity score:  0.9999998\n","Given label:  0.9199999999999999\n","\n","sent_a:  Syrian Opposition Council to Establish Envoy in France\n","sent_b:  Syrian opposition in reconciliation talks\n","similarity score:  0.5763809\n","Given label:  0.27999999999999997\n","\n","sent_a:  The large beige dog is running through the grass.\n","sent_b:  A golden dog is running on the grass.\n","similarity score:  0.8873029\n","Given label:  0.6799999999999999\n","\n","sent_a:  A red and white double decker bus is parked.\n","sent_b:  Parked white double decker bus.\n","similarity score:  0.84219354\n","Given label:  0.8400000000000001\n","\n","sent_a:  The Connecticut Hospital Association joined the Red Cross Monday in calling for more blood donors.\n","sent_b:  Hospitals and the Red Cross appealed to blood donors yesterday.\n","similarity score:  0.8328539\n","Given label:  0.75\n","\n","sent_a:  Westwood in front a Dubai Desert Classic\n","sent_b:  Golf: Westwood takes lead in Dubai Desert Classic\n","similarity score:  0.7208514\n","Given label:  0.8\n","\n","sent_a:  On Tuesday, before Byrd's speech, Fleischer said Bush wanted ''to see an aircraft landing the same way that the pilots saw an aircraft landing.\n","sent_b:  Bush wanted \"to see an aircraft landing the same way that the pilots saw an aircraft landing,\" White House press secretary Ari Fleischer said yesterday.\n","similarity score:  0.9841639\n","Given label:  0.8\n","\n","sent_a:  A woman is feeding a man.\n","sent_b:  A man is smelling two pizzas.\n","similarity score:  0.74575436\n","Given label:  0.04\n","\n","sent_a:  Urgent: Death toll from NE China fire rises to 112\n","sent_b:  Death toll from west Chinas violence rises to 35\n","similarity score:  0.9024217\n","Given label:  0.24\n","\n","sent_a:  The Rev. John Johnston, 64, was charged with aggravated harassment in the phone call case and with criminal possession of a weapon, according to a police statement.\n","sent_b:  The priest, the Rev. John F. Johnston, 64, of 35th Avenue in Jackson Heights, was charged with aggravated harassment and criminal possession of a weapon.\n","similarity score:  0.9443637\n","Given label:  0.75\n","\n","sent_a:  a woman sitting on a sofa holding a baby.\n","sent_b:  A smiling woman holding a small baby.\n","similarity score:  0.87575614\n","Given label:  0.6799999999999999\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"-U5_tuiDZ7KP"},"source":["<a name=\"3-2-3\"></a>\n","#### 3.2.3 Summary\n","\n","It is quite understandable that the bog-of-word models do not work well on dissimilar sentence pairs both in Chinese dataset and English dataset. **Our models are too simple to be powerful**. What we did is only use the pre-trained word embeddings and sum them up to get the sentence embeddings, which still needs further fine-tuned in order to have predictive powers.\n","\n","<br>\n","\n","As the sentence embedding is just sum of its tokens' embeddings, two sentences whose tokens' embeddings are similar but whose meaninsg are different, can have a very high cosine similarity score. Moreover, if two sentences have identical words except one being negative (e.g., \"I am happy\" versus \"I am not happy\"), then their cosine similarity score will be very high as well. \n","\n","<br>\n","\n","More on text similarity will be updated in my GitHub Project [dl-nlp-using-paddlenlp](https://github.com/jaaack-wang/dl-nlp-using-paddlenlp).  "]},{"cell_type":"markdown","metadata":{"id":"FxVuGgM04mBy"},"source":["<a name=\"4\"></a>\n","# 4. References\n"]},{"cell_type":"markdown","metadata":{"id":"gvV2AjpkVD8A"},"source":["- [PaddleNLP Embedding API](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/embeddings.html)"]}]}