paddlenlp_updated_notes_English/Text Similarity/2.2 - preprocess_data_using_paddle.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2db48e77",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Author: Zhengxiang (Jack) Wang \n",
    "# Date: 2022-01-20\n",
    "# GitHub: https://github.com/jaaack-wang "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "423b97cc",
   "metadata": {},
   "source": [
    "## Quick start\n",
    "\n",
    "With wrapped up functions that we will gradually learn throughout this tutorial, preprocessing the text data into one that is ready for model training can be as simple as following. Does it really work? Let's explore!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "84f29b37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Two vocabulary dictionaries have been built!\n",
      "Please call \u001b[1mX.vocab_to_idx | X.idx_to_vocab\u001b[0m to find out more where [X] stands for the name you used for this TextVectorizer class.\n"
     ]
    }
   ],
   "source": [
    "from utils import load_dataset, gather_text\n",
    "from paddle_utils import * \n",
    "\n",
    "train_set = load_dataset('train.txt')\n",
    "\n",
    "text = gather_text(train_set)\n",
    "V = TextVectorizer()\n",
    "V.build_vocab(text)\n",
    "\n",
    "trans_fn = get_trans_fn(V, include_seq_len=False)\n",
    "batchify_fn = get_batchify_fn(include_seq_len=False)\n",
    "train_loader = create_dataloader(train_set, trans_fn, batchify_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ebf68a00",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Tensor(shape=[64, 55], dtype=int64, place=CPUPlace, stop_gradient=True,\n",
      "       [[6   , 47  , 7   , ..., 0   , 0   , 0   ],\n",
      "        [26  , 208 , 3702, ..., 0   , 0   , 0   ],\n",
      "        [4   , 3   , 22  , ..., 0   , 0   , 0   ],\n",
      "        ...,\n",
      "        [17  , 13  , 23  , ..., 0   , 0   , 0   ],\n",
      "        [20  , 11  , 19  , ..., 0   , 0   , 0   ],\n",
      "        [6   , 15  , 7   , ..., 0   , 0   , 0   ]]), Tensor(shape=[64, 32], dtype=int64, place=CPUPlace, stop_gradient=True,\n",
      "       [[6  , 11 , 7  , ..., 0  , 0  , 0  ],\n",
      "        [5  , 24 , 321, ..., 0  , 0  , 0  ],\n",
      "        [6  , 15 , 7  , ..., 0  , 0  , 0  ],\n",
      "        ...,\n",
      "        [17 , 13 , 23 , ..., 0  , 0  , 0  ],\n",
      "        [4  , 38 , 8  , ..., 0  , 0  , 0  ],\n",
      "        [26 , 651, 78 , ..., 0  , 0  , 0  ]]), Tensor(shape=[64], dtype=int64, place=CPUPlace, stop_gradient=True,\n",
      "       [1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0])]\n"
     ]
    }
   ],
   "source": [
    "for example in train_loader:\n",
    "    print(example)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e704c8e6",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In this tutorial, we will use functions from `paddle` to help us preprocess and numericalize datasets. As these functions are native to `paddle`, so there is an advantage in training models constructed by using `paddle`, especially when the datasets are large. You will also need [`paddlenlp`](https://github.com/PaddlePaddle/PaddleNLP), a nlp package designed by the `paddle` team, to get everything going. To download it, simply run in command `pip3 install --upgrade paddlenlp`.\n",
    "\n",
    "If you need more intuition about the ins and outs of this process, please refer to `2 - preprocess_data.ipynb` in the same folder.\n",
    "\n",
    "Below are the structure of this tutoiral:\n",
    "\n",
    "- [Load dataset](#1)\n",
    "- [Create vocab_to_idx mapping dictionary](#2)\n",
    "- [Text encoder](#3)\n",
    "- [Example converter](#4)\n",
    "    - [Coverting multiple examples](#4-1)\n",
    "- [Creating dataloader](#5)\n",
    "    - [Transform the dataset into  Dataset class using MapDataset](#5-1)\n",
    "    - [A data sampler](#5-2)\n",
    "    - [Building a batchify method](#5-3)\n",
    "    - [Now the dataloader](#5-4)\n",
    "- [A quick test](#6)\n",
    "- [Wrapped up functions](#7)\n",
    "    - [TextVectorizer](#7-1)\n",
    "    - [Example converter](#7-2)\n",
    "    - [Get trans_fn](#7-3)\n",
    "    - [Get batchify_fn](#7-4)\n",
    "    - [Create dataloader](#7-5)\n",
    "- [More thorough tests](#8)\n",
    "    - [Initializations](#8-1)\n",
    "    - [Test One: CNN](#8-2)\n",
    "    - [Test Two: RNN](#8-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee0077be",
   "metadata": {},
   "source": [
    "<a name=\"1\"></a>\n",
    "## Load dataset\n",
    "\n",
    "As usual, let's first use the `load_dataset` function compiled in the last two tutorials to load the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e3616160",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3000"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils import load_dataset\n",
    "\n",
    "train_set = load_dataset('train.txt')\n",
    "\n",
    "# check. should be 3000 (recall `1 - get_data.ipynb`)\n",
    "len(train_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfd1e127",
   "metadata": {},
   "source": [
    "<a name=\"2\"></a>\n",
    "\n",
    "## Create `vocab_to_idx` mapping dictionary\n",
    "\n",
    "The purpose of creating a `vocab_to_idx` mapping dictionary is for later encoding or numeralizing text data for model training. In the `2.1 - wrapped_up_data_preprocessor` tutorial, we have learnt how to use `TextVectorizer` to conveniently do this job. \n",
    "\n",
    "In this tutorial, we will use `paddlenlp.data.Vocab` to do a simlar job. Fortunately, there are many English tokenizers that come with `paddlenlp` and we will use a simple one from `from paddlenlp.transformers.bert.tokenizer.BasicTokenizer` (a lot of tokenizers are stored in the [transformers folder](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/paddlenlp/transformers)). Apparently, no one is going to rememberize so a long path to import the tokenizer. Alternatively, you can just use the `.split` function to tokenize English or use the `tokenize` function from `utils.py`.\n",
    "\n",
    "Unfortunately, unlike the `TextVectorize` we have built, `paddlenlp.data.Vocab` do not have a text_encoder function for us to use directly, but we can built it ourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ceb66d62",
   "metadata": {},
   "outputs": [],
   "source": [
    "from paddlenlp.data import Vocab\n",
    "# such a long path! sucks!\n",
    "from paddlenlp.transformers.bert.tokenizer import BasicTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1adc7b9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, we need a tokenize func, \"()\" is needed here!\n",
    "tokenize = BasicTokenizer().tokenize\n",
    "\n",
    "# Then we need a list of tokenized texts\n",
    "from utils import gather_text\n",
    "text = gather_text(train_set) # ---> gather text from the train_set\n",
    "tokens = list(map(tokenize, text)) # ---> a list of tokenized texts ([[w1, w2...], [w1, w2...]...])\n",
    "\n",
    "# build the vocabulary which will give us the mapping dictionaries for encoding\n",
    "V = Vocab.build_vocab(tokens, unk_token='[UNK]', pad_token='[PAD]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a98442d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bos_token',\n",
       " 'build_vocab',\n",
       " 'eos_token',\n",
       " 'from_dict',\n",
       " 'from_json',\n",
       " 'idx_to_token',\n",
       " 'load_vocabulary',\n",
       " 'pad_token',\n",
       " 'to_indices',\n",
       " 'to_json',\n",
       " 'to_tokens',\n",
       " 'token_to_idx',\n",
       " 'unk_token']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's see what we have \n",
    "# The most useful will be: V.token_to_idx and V.idx_to_token\n",
    "[d for d in dir(V) if not d.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0a7cb67f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The first 10 examples from the token_to_idx dictionary\n",
      "\n",
      "[PAD]               0\n",
      "[UNK]               1\n",
      "?                   2\n",
      "the                 3\n",
      "what                4\n",
      "is                  5\n",
      "how                 6\n",
      "i                   7\n",
      "a                   8\n",
      "to                  9\n",
      "\n",
      "\n",
      "The first 10 random examples from the idx_to_token dictionary\n",
      "\n",
      "0                   [PAD]\n",
      "1                   [UNK]\n",
      "2                   ?\n",
      "3                   the\n",
      "4                   what\n",
      "5                   is\n",
      "6                   how\n",
      "7                   i\n",
      "8                   a\n",
      "9                   to\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "\n",
    "tmp = \"{:20}{}\"\n",
    "print(\"The first 10 examples from the token_to_idx dictionary\\n\")\n",
    "\n",
    "for item in list(V.token_to_idx.items())[:10]:\n",
    "    print(tmp.format(*item))\n",
    "    \n",
    "    \n",
    "print(\"\\n\\nThe first 10 random examples from the idx_to_token dictionary\\n\")\n",
    "\n",
    "for idx, tk in list(V.idx_to_token.items())[:10]:\n",
    "    print(tmp.format(str(idx), tk))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c5bd4b60",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index for \u001b[1mThis_Word_Does_Not_Exist\u001b[0m: 1\n"
     ]
    }
   ],
   "source": [
    "# for unseen token_to_idx\n",
    "\n",
    "print(\"Index for \\033[1mThis_Word_Does_Not_Exist\\033[0m:\", V.token_to_idx[\"This_Word_Does_Not_Exist\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff694234",
   "metadata": {},
   "source": [
    "<a name=\"3\"></a>\n",
    "## Text encoder\n",
    "\n",
    "If you are interested, you can also build a `text_decoder` as we did before. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5e9f2b30",
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_encoder(text, \n",
    "                 tokenize=tokenize, \n",
    "                 token_to_idx=V.token_to_idx):\n",
    "    \n",
    "    tokens = tokenize(text)\n",
    "    out = []\n",
    "    for tk in tokens:\n",
    "        out.append(token_to_idx[tk])\n",
    "        \n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c1af9d04",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original text: How do I write a good essay?\n",
      "Encoded text: [6, 11, 7, 247, 8, 51, 932, 2]\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "\n",
    "print(\"Original text:\", text[0])\n",
    "print(\"Encoded text:\", text_encoder(text[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c96a5128",
   "metadata": {},
   "source": [
    "<a name=\"4\"></a>\n",
    "## Example converter \n",
    "\n",
    "We will see why this is needed shortly. The purpose of an example converter is to transform a given example into the data we need for training models. Later, we will use `map` function to map the `example_converter` to an entire dataset.\n",
    "\n",
    "If you see the previous tutorials, we will know that since the `RNN` models takes as an input the text seq length, we also need to take that into consideration. More generally, as different tasks or different models have different needs, you need to tailor the `example_converter` based on your specific needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "10371a5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def example_converter(example, text_encoder, include_seq_len):\n",
    "    \n",
    "    text_a, text_b, label = example\n",
    "    encoded_a = text_encoder(text_a)\n",
    "    encoded_b = text_encoder(text_b)\n",
    "    if include_seq_len:\n",
    "        len_a, len_b = len(encoded_a), len(encoded_b)\n",
    "        return encoded_a, encoded_b, len_a, len_b, label\n",
    "    return encoded_a, encoded_b, label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "eabeccb8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original text_a: How do I write a good essay?\n",
      "Encoded text_a: [6, 11, 7, 247, 8, 51, 932, 2]\n",
      "\n",
      "Original text_b: How do I write an essay in English?\n",
      "Encoded text_b: [6, 11, 7, 247, 40, 932, 10, 107, 2]\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "\n",
    "encoded_a, encoded_b, label = example_converter(train_set[0], text_encoder, False)\n",
    "a, b, l = train_set[0]\n",
    "\n",
    "print(\"Original text_a:\", a)\n",
    "print(\"Encoded text_a:\", encoded_a)\n",
    "\n",
    "print(\"\\nOriginal text_b:\", b)\n",
    "print(\"Encoded text_b:\", encoded_b)\n",
    "\n",
    "# nothing changes for label"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "698dfcd8",
   "metadata": {},
   "source": [
    "<a name=\"4-1\"></a>\n",
    "### Coverting multiple examples\n",
    "\n",
    "To convert multiple examples, we can either loop through the examples one by one or use the `map` function. The `map` function takes a function as the first input and then one or several iterable (e.g., list or tuple) corresponding to the parameters of that function. In our `example_converter` we have three parameters, that means we will need to do something like the following:\n",
    "\n",
    "```python\n",
    ">>> examples = 'a list of examples'\n",
    ">>> n = len(examples)\n",
    ">>> E = text_encoder\n",
    ">>> B = include_seq_len\n",
    ">>> converted = map(example_converter, examples, [E] * n, [B] * n)\n",
    "```\n",
    "\n",
    "That looks bad, right? If you do not multiply the later two parameters, making them iterable, we will see something like the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b4a18de3",
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'function' object is not iterable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/w9/d_nplhzj4qx35xxlgljgdtjh0000gn/T/ipykernel_66770/1024683464.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mexamples\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_set\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mconverted\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexample_converter\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexamples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtext_encoder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: 'function' object is not iterable"
     ]
    }
   ],
   "source": [
    "examples = train_set[:2]\n",
    "converted = map(example_converter, examples, text_encoder, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efd1f8cf",
   "metadata": {},
   "source": [
    "**There are three ways to get around this:**\n",
    "\n",
    "- First, assign a default value to the latter two parameters (text_encoder & include_seq_len) so that you only need to pass the \"example\" paramters when using `map`. If we want to change the default values, we need to do it every time by changing the `example_converter` function directly.\n",
    "\n",
    "- Second, use the handy built-in function `partial` from `functools` or `lambda` to assign default values for a function's parameters, which will create another encapsulated function for you. \n",
    "\n",
    "- Third, in our case, we can simply make the `example_converter` return the text seq length regardless, but that means, you need to construct all your models taking this into account (some models that do not need seq length info will have the related parameters but never use them). \n",
    "\n",
    "\n",
    "Below, we will show the second way because it is what is commonly adopted by the deep learning community when doing natural language processing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56af8f06",
   "metadata": {},
   "source": [
    "#### Use `lambda`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e1bafe72",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example #1....\n",
      "Encoded text_a: [6, 11, 7, 247, 8, 51, 932, 2]\n",
      "Encoded text_b: [6, 11, 7, 247, 40, 932, 10, 107, 2]\n",
      "Label: 1\n",
      "Example #2....\n",
      "Encoded text_a: [30, 5, 3, 22, 4101, 210, 2]\n",
      "Encoded text_b: [30, 13, 3, 22, 725, 7069, 3809, 4101, 210, 92, 230, 2]\n",
      "Label: 0\n"
     ]
    }
   ],
   "source": [
    "trans_fn = lambda example: example_converter(example, text_encoder, False)\n",
    "converted = map(trans_fn, examples)\n",
    "\n",
    "for idx, conv in enumerate(converted):\n",
    "    print(f\"Example #{idx+1}....\")\n",
    "    e_a, e_b, l = conv\n",
    "    print(\"Encoded text_a:\", e_a)\n",
    "    print(\"Encoded text_b:\", e_b)\n",
    "    print(\"Label:\", l)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "751883c9",
   "metadata": {},
   "source": [
    "#### Use `partial`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "62969002",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example # 1 ...\n",
      "Encoded text_a: [6, 11, 7, 247, 8, 51, 932, 2]\n",
      "Encoded text_b: [6, 11, 7, 247, 40, 932, 10, 107, 2]\n",
      "Label: 1\n",
      "Example # 2 ...\n",
      "Encoded text_a: [30, 5, 3, 22, 4101, 210, 2]\n",
      "Encoded text_b: [30, 13, 3, 22, 725, 7069, 3809, 4101, 210, 92, 230, 2]\n",
      "Label: 0\n"
     ]
    }
   ],
   "source": [
    "from functools import partial \n",
    "\n",
    "# names for paramters you want to set default must be given\n",
    "trans_fn = partial(example_converter, \n",
    "                   text_encoder=text_encoder, \n",
    "                   include_seq_len=False)\n",
    "\n",
    "converted = map(trans_fn, examples)\n",
    "\n",
    "for idx, conv in enumerate(converted):\n",
    "    print(f\"Example # {idx+1} ...\")\n",
    "    e_a, e_b, l = conv\n",
    "    print(\"Encoded text_a:\", e_a)\n",
    "    print(\"Encoded text_b:\", e_b)\n",
    "    print(\"Label:\", l)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a58376c",
   "metadata": {},
   "source": [
    "<a name=\"5\"></a>\n",
    "## Creating dataloader\n",
    "\n",
    "Now comes with the most important points! **I figure that detailed explanations may not help you to understand what will be shown below, because you may need to practice again and again, and compare with what we have done previously to build a solid intuition.** Let's simply take a dataloder as a black box. All you need to know is what needs to go in and what will come out. Here are some of the points you need to know:\n",
    "\n",
    "- A dataloader is something iterable and will work more efficiently with the models constructed by a deep learning framework, especially when trained on GPUs because they can load data asynchronously.\n",
    "\n",
    "\n",
    "- For a dataloader, you need to pass the dataset you want to train on, the `example_converter` (or the encapsulated `trans_fn`) , a data `sampler` (i.e., how to build batches), as well as a `batchify` method to preprocess a given batch (or sample).\n",
    "\n",
    "\n",
    "- For the dataset, its type needs to be what is called `Dataset` (map-style dataset) or `IterableDataset` (iterable-style dataset) in order to make everything work. \n",
    "\n",
    "\n",
    "Enough words. Let's just see what this can be done."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbe09ae3",
   "metadata": {},
   "source": [
    "<a name=\"5-1\"></a>\n",
    "### Transform the dataset into  `Dataset` class using `MapDataset`\n",
    "\n",
    "Some properties of the `Dataset`:\n",
    "\n",
    "- It is iterable both by a for loop and by a slicing index (just like a list!)\n",
    "- It has a `Dataset.map` method that will transform (namely numericalize) the entire dataset given a `trans_fn` (with only one paratemter allowed)\n",
    "- The transformed `Dataset` will be iterable by a for loop but can not be indexed anymore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8cd1f2d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type of train <class 'paddlenlp.datasets.dataset.MapDataset'>\n",
      "Is train's type a Dataset? True\n"
     ]
    }
   ],
   "source": [
    "import paddle\n",
    "from paddlenlp.datasets import MapDataset\n",
    "\n",
    "train = MapDataset(train_set)\n",
    "\n",
    "print(\"Type of train\", type(train))\n",
    "print(\"Is train's type a Dataset?\", isinstance(train, paddle.io.Dataset))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "60a75285",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number 1 example:  ([6, 11, 7, 247, 8, 51, 932, 2], [6, 11, 7, 247, 40, 932, 10, 107, 2], 1)\n",
      "Number 2 example:  ([30, 5, 3, 22, 4101, 210, 2], [30, 13, 3, 22, 725, 7069, 3809, 4101, 210, 92, 230, 2], 0)\n",
      "Number 3 example:  ([6, 11, 7, 2476, 8, 3456, 36, 82, 1887, 14, 2456, 7239, 10, 24, 2], [32, 7, 21, 159, 1404, 18, 15, 7, 2770, 247, 23, 3211, 325, 10, 8, 5102, 57, 3456, 2, 5, 24, 51, 29, 321, 2], 0)\n",
      "\n",
      "The first three examples...\n",
      " [('How do I write a good essay?', 'How do I write an essay in English?', 1), ('Which is the best thriller movies ?', 'Which are the best brain twisting psychological thriller movies ever made?', 0), ('How do I preserve a journal that has pencil and pen writings in it?', \"If I'm depressed, can I atleast write my feelings down in a diary/journal? Is it good or bad?\", 0)]\n"
     ]
    }
   ],
   "source": [
    "from copy import deepcopy # we need \"deepcopy\" to protect the orignal data\n",
    "\n",
    "train_copy = deepcopy(train)\n",
    "train_copy.map(trans_fn)\n",
    "\n",
    "for idx, emp in enumerate(train_copy):\n",
    "    print(f\"Number {idx+1} example: \", emp)\n",
    "    if idx == 2:\n",
    "        break\n",
    "        \n",
    "# versus (remeber, after transformation, we can't use slicing index!)\n",
    "print(\"\\nThe first three examples...\\n\", train[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c54fea3",
   "metadata": {},
   "source": [
    "<a name=\"5-2\"></a>\n",
    "### A data sampler\n",
    "\n",
    "There are two types of data sampler in `paddle`. One is `paddle.io.DistributedBatchSampler` for distributed multiple GPU training. And the other is `paddle.io.BatchSampler` for the normal use. We will use the later here.\n",
    "\n",
    "Please note that the batched dataset will be only iterable in a for loop, but we can use the `list` function nested within a `len` function to count how many batches there are. For every batch inside this iterable output are a list of indices that index the dataset. For example, an index \"1\" would mean \"dataset\\[1\\]\". The indices will later be used by the dataloader to retrieve examples from the transformed/numerilized dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "80fcdde6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of batches: 47\n",
      "Number of items in the last batch: 56\n",
      "Everything all right? True\n"
     ]
    }
   ],
   "source": [
    "from paddle.io import BatchSampler\n",
    "\n",
    "# do not forget to map the \"trans_fn\" to the \"train\" dataset. with the \"map\" func applied,\n",
    "# you can no longer run this cell again, as the \"train\" has been transformed and no longer has \"map\" (see above)\n",
    "bacth_sampler = BatchSampler(dataset=train.map(trans_fn), shuffle=True, batch_size=64)\n",
    "\n",
    "# check: 3000 == 46 * 64 + 56\n",
    "print(\"Number of batches:\", len(list(bacth_sampler)))\n",
    "print(\"Number of items in the last batch:\",  len(list(bacth_sampler)[-1]))\n",
    "print(\"Everything all right?\", 3000 == 46 * 64 + 56)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "fa10af5d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first 10 examples from the first batch: [2472, 2285, 646, 725, 1637, 2020, 2471, 1059, 2078, 2132]\n",
      "The indices are connecting with the transformed/numericalized dataset.\n"
     ]
    }
   ],
   "source": [
    "first_10 = list(bacth_sampler)[0][:10]\n",
    "\n",
    "print(\"This is the first 10 examples from the first batch:\", first_10)\n",
    "print(\"The indices are connecting with the transformed/numericalized dataset.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f345bd18",
   "metadata": {},
   "source": [
    "<a name=\"5-3\"></a>\n",
    "### Building a `batchify` method \n",
    "\n",
    "The purpose of the `batchify` method is to provide a set of methods to further preprocess the bacthed dataset in a way that make possible model training. More concretely, the following are three things we need to consider:\n",
    "\n",
    "- for the text ids (numericalized text) within a batch and of same kind (text_a versus text_b), we need to make sure that they are of same length/dimension (aligned with the max length in a batch). We can use the `paddlenlp.data.Pad` function to do that.\n",
    "- for the label values or the likes within a batch, we want them to be stacked together in a array. We will use the `paddlenlp.data.Stack` to do that. \n",
    "- for every bacthed element (e.g., text_a, text_b, label), we want them to be separated. We can use the `paddlenlp.data.Tuple` function to do that.\n",
    "\n",
    "Please note that, all these class methods are callable. You can use use them by calling, for example, `Pad()(YourInput)`. But you can also initialize certain values when calling, for example `Pad(axis=0, pad_val=0)(YourInput)`. Moreover, for those RNN models, we will need to create anthoer two `Stack()` for the text_a_seq_len and text_b_seq_len respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7df60525",
   "metadata": {},
   "outputs": [],
   "source": [
    "from paddlenlp.data import Pad, Stack, Tuple\n",
    "\n",
    "# The Tuple() will pass the \"samples\" to the three funcs inside one by one\n",
    "# then a list of three outputs returned by the three funcs (in this case)\n",
    "batchify_fn = lambda samples, fn=Tuple(\n",
    "        Pad(axis=0, pad_val=0),  # for text_a_ids; axis=0, pad_val=0 are default just so you know\n",
    "        Pad(axis=0, pad_val=0),  # for text_b_ids\n",
    "        Stack(dtype=\"int64\")  # for label\n",
    "    ): fn(samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3e9618e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of text_a_ids preprocessed: (3000, 67)\n",
      "Shape of text_b_ids preprocessed: (3000, 74)\n",
      "Shape of labels preprocessed: (3000,)\n"
     ]
    }
   ],
   "source": [
    "# check. Note the following has not been batched.\n",
    "\n",
    "t_a, t_b, l = batchify_fn(list(map(trans_fn, train_set)))\n",
    "print(\"Shape of text_a_ids preprocessed:\", t_a.shape)\n",
    "print(\"Shape of text_b_ids preprocessed:\", t_b.shape)\n",
    "print(\"Shape of labels preprocessed:\", l.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b4c81c0",
   "metadata": {},
   "source": [
    "<a name=\"5-4\"></a>\n",
    "### Now the dataloader\n",
    "\n",
    "The dataloader will \"load\" the numericalized dataset according to our instructions and return an iterable object for the model to loop through during training. As mentioned, we will teach the dataloader how to sample (build batches for) the dataset, and how to further preprocess the built batches. The dataloader will also transform the numericalized dataset into `paddle.Tensor`, making model training quicker (for larger datasets trained on GPUs).\n",
    "\n",
    "Again, the output cannot be retrieved by index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c7419629",
   "metadata": {},
   "outputs": [],
   "source": [
    "from paddle.io import DataLoader\n",
    "\n",
    "dataloader = DataLoader(\n",
    "    train,\n",
    "    batch_sampler=bacth_sampler, \n",
    "    collate_fn=batchify_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f67ad149",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Tensor(shape=[64, 56], dtype=int64, place=CPUPlace, stop_gradient=True,\n",
      "       [[4 , 5 , 24, ..., 0 , 0 , 0 ],\n",
      "        [6 , 15, 7 , ..., 0 , 0 , 0 ],\n",
      "        [4 , 13, 35, ..., 0 , 0 , 0 ],\n",
      "        ...,\n",
      "        [32, 7 , 21, ..., 0 , 0 , 0 ],\n",
      "        [6 , 11, 7 , ..., 0 , 0 , 0 ],\n",
      "        [6 , 15, 7 , ..., 0 , 0 , 0 ]]), Tensor(shape=[64, 32], dtype=int64, place=CPUPlace, stop_gradient=True,\n",
      "       [[4   , 26  , 24  , ..., 0   , 0   , 0   ],\n",
      "        [6   , 38  , 7   , ..., 0   , 0   , 0   ],\n",
      "        [6   , 1928, 15  , ..., 0   , 0   , 0   ],\n",
      "        ...,\n",
      "        [32  , 7   , 1056, ..., 0   , 0   , 0   ],\n",
      "        [6   , 13  , 3   , ..., 0   , 0   , 0   ],\n",
      "        [26  , 651 , 78  , ..., 0   , 0   , 0   ]]), Tensor(shape=[64], dtype=int64, place=CPUPlace, stop_gradient=True,\n",
      "       [1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0])]\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "\n",
    "for d in dataloader:\n",
    "    print(d)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ff29605",
   "metadata": {},
   "source": [
    "<a name=\"6\"></a>\n",
    "## A quick test\n",
    "\n",
    "It works!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "59ae45c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from paddle_models.BoW import BoW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c0de4f98",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model(model):\n",
    "    model = paddle.Model(model)\n",
    "    optimizer = paddle.optimizer.Adam(\n",
    "    parameters=model.parameters(), learning_rate=5e-4)\n",
    "    criterion = paddle.nn.CrossEntropyLoss()\n",
    "    metric = paddle.metric.Accuracy()\n",
    "    model.prepare(optimizer, criterion, metric)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "58befa4e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The loss value printed in the log is the current step, and the metric is the average value of previous steps.\n",
      "Epoch 1/1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/paddle/fluid/layers/utils.py:77: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working\n",
      "  return (isinstance(seq, collections.Sequence) and\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 47/47 [==============================] - loss: 0.6177 - acc: 0.6017 - 35ms/step          \n",
      "CPU times: user 1.62 s, sys: 61.1 ms, total: 1.68 s\n",
      "Wall time: 1.63 s\n"
     ]
    }
   ],
   "source": [
    "model = BoW(len(V.token_to_idx), 2)\n",
    "model = get_model(model)\n",
    "%time model.fit(dataloader, epochs=1, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c522aaac",
   "metadata": {},
   "source": [
    "<a name=\"7\"></a>\n",
    "## Wrapped up functions\n",
    "\n",
    "Before heading to the next section, you can test the following functions/class methods up and see if you can utilize them to do a quick start yourself!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e56762e",
   "metadata": {},
   "source": [
    "<a name=\"7-1\"></a>\n",
    "### TextVectorizer\n",
    "\n",
    "The following wrapped up class method remsembles the one that we built in the `2.1 - wrapped_up_data_preprocessor.ipynb`, but here we are using as many functions as from `paddle`. If you are interested, you can look a look back at the `TextVectorizer` inside the `utils.py` and see if you can create some additional functions (such as save the results into json file for later re-loading). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c447ef33",
   "metadata": {},
   "outputs": [],
   "source": [
    "from paddlenlp.data import Vocab\n",
    "from paddlenlp.transformers.bert.tokenizer import BasicTokenizer\n",
    "from collections.abc import Iterable\n",
    "\n",
    "\n",
    "class TextVectorizer:\n",
    "     \n",
    "    def __init__(self, tokenizer=None):\n",
    "        self.tokenize = tokenizer if tokenizer \\\n",
    "        else BasicTokenizer().tokenize\n",
    "        self.vocab_to_idx = None\n",
    "        self.idx_to_vocab = None\n",
    "        self._V = None\n",
    "    \n",
    "    def build_vocab(self, text):\n",
    "        tokens = list(map(tokenize, text))\n",
    "        self._V = Vocab.build_vocab(tokens, unk_token='[UNK]', pad_token='[PAD]')\n",
    "        self.vocab_to_idx = self._V.token_to_idx\n",
    "        self.idx_to_vocab = self._V.idx_to_token\n",
    "        \n",
    "        print('Two vocabulary dictionaries have been built!\\n' \\\n",
    "             + 'Please call \\033[1mX.vocab_to_idx | X.idx_to_vocab\\033[0m to find out more' \\\n",
    "             + ' where [X] stands for the name you used for this TextVectorizer class.')\n",
    "        \n",
    "    def text_encoder(self, text):\n",
    "        if isinstance(text, list):\n",
    "            return [self(t) for t in text]\n",
    "        \n",
    "        tks = self.tokenize(text)\n",
    "        out = [self.vocab_to_idx[tk] for tk in tks]\n",
    "        return out\n",
    "            \n",
    "    def text_decoder(self, text_ids, sep=\" \"):\n",
    "        if all(isinstance(ids, Iterable) for ids in text_ids):\n",
    "            return [self.text_decoder(ids, sep) for ids in text_ids]\n",
    "            \n",
    "        out = []\n",
    "        for text_id in text_ids:\n",
    "            out.append(self.idx_to_vocab[text_id])\n",
    "            \n",
    "        return f'{sep}'.join(out)\n",
    "    \n",
    "    def __call__(self, text):\n",
    "        if self.vocab_to_idx:\n",
    "            return self.text_encoder(text)\n",
    "        raise ValueError(\"No vocab is built!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb8b6ae6",
   "metadata": {},
   "source": [
    "<a name=\"7-2\"></a>\n",
    "### Example converter\n",
    "\n",
    "Nothing to change here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "9349300f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def example_converter(example, text_encoder, include_seq_len):\n",
    "    \n",
    "    text_a, text_b, label = example\n",
    "    encoded_a = text_encoder(text_a)\n",
    "    encoded_b = text_encoder(text_b)\n",
    "    if include_seq_len:\n",
    "        len_a, len_b = len(encoded_a), len(encoded_b)\n",
    "        return encoded_a, encoded_b, len_a, len_b, label\n",
    "    return encoded_a, encoded_b, label"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64700eca",
   "metadata": {},
   "source": [
    "<a name=\"7-3\"></a>\n",
    "### Get trans_fn\n",
    "\n",
    "Let's customize a method to return trans_fn for us for this series of tutorial!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "6027fcf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_trans_fn(text_encoder, include_seq_len):\n",
    "    return lambda ex: example_converter(ex, text_encoder, include_seq_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e693dd1",
   "metadata": {},
   "source": [
    "<a name=\"7-4\"></a>\n",
    "### Get batchify_fn\n",
    "\n",
    "Let's customize a method to return batchify_fn for us for this series of tutorial!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7f2609b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from paddlenlp.data import Pad, Stack, Tuple\n",
    "\n",
    "\n",
    "def get_batchify_fn(include_seq_len):\n",
    "    \n",
    "    if include_seq_len:\n",
    "        stack = [Stack(dtype=\"int64\")] * 3\n",
    "    else:\n",
    "        stack = [Stack(dtype=\"int64\")]\n",
    "    \n",
    "    batchify_fn = lambda samples, fn=Tuple(\n",
    "        Pad(axis=0, pad_val=0),  \n",
    "        Pad(axis=0, pad_val=0),  \n",
    "        *stack\n",
    "    ): fn(samples)\n",
    "    \n",
    "    return batchify_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d68e897c",
   "metadata": {},
   "source": [
    "<a name=\"7-5\"></a>\n",
    "### Create dataloader "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "4fd4631a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from paddlenlp.datasets import MapDataset\n",
    "from paddle.io import BatchSampler, DataLoader\n",
    "\n",
    "\n",
    "def create_dataloader(dataset, \n",
    "                      trans_fn, \n",
    "                      batchify_fn, \n",
    "                      batch_size=64, \n",
    "                      shuffle=True, \n",
    "                      sampler=BatchSampler):\n",
    "    \n",
    "    \n",
    "    if not isinstance(dataset, MapDataset):\n",
    "        dataset = MapDataset(dataset)\n",
    "        \n",
    "    dataset.map(trans_fn)\n",
    "    batch_sampler = sampler(dataset, \n",
    "                            shuffle=shuffle, \n",
    "                            batch_size=batch_size)\n",
    "    \n",
    "    dataloder = DataLoader(dataset, \n",
    "                           batch_sampler=batch_sampler, \n",
    "                           collate_fn=batchify_fn)\n",
    "    \n",
    "    return dataloder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b5e3363",
   "metadata": {},
   "source": [
    "<a name=\"8\"></a>\n",
    "## More thorough tests \n",
    "\n",
    "This time, we will include the dev_set for validation and the test_set for evaluation!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "450e0a12",
   "metadata": {},
   "source": [
    "<a name=\"8-1\"></a>\n",
    "### Initializations "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "1f2be1d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Two vocabulary dictionaries have been built!\n",
      "Please call \u001b[1mX.vocab_to_idx | X.idx_to_vocab\u001b[0m to find out more where [X] stands for the name you used for this TextVectorizer class.\n"
     ]
    }
   ],
   "source": [
    "from utils import load_dataset, gather_text\n",
    "\n",
    "train_set, dev_set, test_set = load_dataset(['train.txt', 'dev.txt', 'test.txt'])\n",
    "\n",
    "text = gather_text(train_set)\n",
    "V = TextVectorizer()\n",
    "V.build_vocab(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f71a018c",
   "metadata": {},
   "source": [
    "<a name=\"8-2\"></a>\n",
    "### Test One: CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "a4301a8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "trans_fn = get_trans_fn(V, False)\n",
    "batchify_fn = get_batchify_fn(False)\n",
    "train_loader = create_dataloader(train_set, trans_fn, batchify_fn)\n",
    "dev_loader = create_dataloader(dev_set, trans_fn, batchify_fn)\n",
    "test_loader = create_dataloader(test_set, trans_fn, batchify_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "9432896d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The loss value printed in the log is the current step, and the metric is the average value of previous steps.\n",
      "Epoch 1/5\n",
      "step 47/47 [==============================] - loss: 0.6780 - acc: 0.5490 - 130ms/step          \n",
      "Eval begin...\n",
      "step 16/16 [==============================] - loss: 0.6978 - acc: 0.5750 - 63ms/step          \n",
      "Eval samples: 1000\n",
      "Epoch 2/5\n",
      "step 47/47 [==============================] - loss: 0.5683 - acc: 0.7730 - 135ms/step          \n",
      "Eval begin...\n",
      "step 16/16 [==============================] - loss: 0.6176 - acc: 0.6480 - 67ms/step          \n",
      "Eval samples: 1000\n",
      "Epoch 3/5\n",
      "step 47/47 [==============================] - loss: 0.4185 - acc: 0.8713 - 132ms/step          \n",
      "Eval begin...\n",
      "step 16/16 [==============================] - loss: 0.6410 - acc: 0.6740 - 61ms/step          \n",
      "Eval samples: 1000\n",
      "Epoch 4/5\n",
      "step 47/47 [==============================] - loss: 0.1479 - acc: 0.9557 - 126ms/step          \n",
      "Eval begin...\n",
      "step 16/16 [==============================] - loss: 0.5570 - acc: 0.6440 - 60ms/step          \n",
      "Eval samples: 1000\n",
      "Epoch 5/5\n",
      "step 47/47 [==============================] - loss: 0.0307 - acc: 0.9907 - 128ms/step          \n",
      "Eval begin...\n",
      "step 16/16 [==============================] - loss: 0.5416 - acc: 0.6460 - 60ms/step          \n",
      "Eval samples: 1000\n",
      "CPU times: user 36 s, sys: 380 ms, total: 36.4 s\n",
      "Wall time: 35.6 s\n"
     ]
    }
   ],
   "source": [
    "from paddle_models.CNN import CNN\n",
    "\n",
    "model = CNN(len(V.vocab_to_idx), 2)\n",
    "model = get_model(model)\n",
    "%time model.fit(train_loader, dev_loader, epochs=5, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "e3754351",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eval begin...\n",
      "step 10/16 - loss: 0.8315 - acc: 0.6312 - 71ms/step\n",
      "step 16/16 - loss: 0.7882 - acc: 0.6340 - 60ms/step\n",
      "Eval samples: 1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'loss': [0.7882101], 'acc': 0.634}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83ecf331",
   "metadata": {},
   "source": [
    "<a name=\"8-3\"></a>\n",
    "### Test Two: RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "aa25bbd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "trans_fn = get_trans_fn(V, True)\n",
    "batchify_fn = get_batchify_fn(True)\n",
    "train_loader = create_dataloader(train_set, trans_fn, batchify_fn)\n",
    "dev_loader = create_dataloader(dev_set, trans_fn, batchify_fn)\n",
    "test_loader = create_dataloader(test_set, trans_fn, batchify_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "ae1407a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The loss value printed in the log is the current step, and the metric is the average value of previous steps.\n",
      "Epoch 1/5\n",
      "step 47/47 [==============================] - loss: 0.6910 - acc: 0.5030 - 53ms/step          \n",
      "Eval begin...\n",
      "step 16/16 [==============================] - loss: 0.6964 - acc: 0.5000 - 34ms/step          \n",
      "Eval samples: 1000\n",
      "Epoch 2/5\n",
      "step 47/47 [==============================] - loss: 0.6739 - acc: 0.5770 - 50ms/step          \n",
      "Eval begin...\n",
      "step 16/16 [==============================] - loss: 0.6740 - acc: 0.5230 - 33ms/step          \n",
      "Eval samples: 1000\n",
      "Epoch 3/5\n",
      "step 47/47 [==============================] - loss: 0.3798 - acc: 0.8153 - 52ms/step          \n",
      "Eval begin...\n",
      "step 16/16 [==============================] - loss: 0.7092 - acc: 0.5620 - 34ms/step          \n",
      "Eval samples: 1000\n",
      "Epoch 4/5\n",
      "step 47/47 [==============================] - loss: 0.2007 - acc: 0.9477 - 51ms/step          \n",
      "Eval begin...\n",
      "step 16/16 [==============================] - loss: 1.2131 - acc: 0.5470 - 34ms/step          \n",
      "Eval samples: 1000\n",
      "Epoch 5/5\n",
      "step 47/47 [==============================] - loss: 0.0307 - acc: 0.9867 - 52ms/step          \n",
      "Eval begin...\n",
      "step 16/16 [==============================] - loss: 0.9891 - acc: 0.5470 - 34ms/step          \n",
      "Eval samples: 1000\n",
      "CPU times: user 15.6 s, sys: 333 ms, total: 16 s\n",
      "Wall time: 14.8 s\n"
     ]
    }
   ],
   "source": [
    "from paddle_models.S_RNN import SimpleRNN\n",
    "\n",
    "model = SimpleRNN(len(V.vocab_to_idx), 2)\n",
    "model = get_model(model)\n",
    "%time model.fit(train_loader, dev_loader, epochs=5, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "8ee02705",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eval begin...\n",
      "step 10/16 - loss: 1.1164 - acc: 0.5547 - 44ms/step\n",
      "step 16/16 - loss: 1.5635 - acc: 0.5440 - 33ms/step\n",
      "Eval samples: 1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'loss': [1.5634658], 'acc': 0.544}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(test_loader)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}