2-NMT-Training.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "mount_file_id": "1QFWTTPHQYjMEMPCqzrS4ayT5Z2nD1WI_",
      "authorship_tag": "ABX9TyObE91Y3T+1kwaL2GlL0Zkm",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ymoslem/OpenNMT-Tutorial/blob/main/2-NMT-Training.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vSUyCs23M_H2"
      },
      "source": [
        "# Install OpenNMT-py 3.x\n",
        "!pip3 install OpenNMT-py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Prepare Your Datasets\n",
        "Please make sure you have completed the [first exercise](https://colab.research.google.com/drive/1rsFPnAQu9-_A6e2Aw9JYK3C8mXx9djsF?usp=sharing)."
      ],
      "metadata": {
        "id": "vhgIdJn-cLqu"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dWVOWYedzZ_G"
      },
      "source": [
        "# Open the folder where you saved your prepapred datasets from the first exercise\n",
        "# You might need to mount your Google Drive first\n",
        "%cd /content/drive/MyDrive/nmt/\n",
        "!ls"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MPlmhd426B7l"
      },
      "source": [
        "# Create the Training Configuration File\n",
        "\n",
        "The following config file matches most of the recommended values for the Transformer model [Vaswani et al., 2017](https://arxiv.org/abs/1706.03762). As the current dataset is small, we reduced the following values: \n",
        "* `train_steps` - for datasets with a few millions of sentences, consider using a value between 100000 and 200000, or more! Enabling the option `early_stopping` can help stop the training when there is no considerable improvement.\n",
        "* `valid_steps` - 10000 can be good if the value `train_steps` is big enough. \n",
        "* `warmup_steps` - obviously, its value must be less than `train_steps`. Try 4000 and 8000 values.\n",
        "\n",
        "Refer to [OpenNMT-py training parameters](https://opennmt.net/OpenNMT-py/options/train.html) for more details. If you are interested in further explanation of the Transformer model, you can check this article, [Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qbW7Xek6UDlY"
      },
      "source": [
        "# Create the YAML configuration file\n",
        "# On a regular machine, you can create it manually or with nano\n",
        "# Note here we are using some smaller values because the dataset is small\n",
        "# For larger datasets, consider increasing: train_steps, valid_steps, warmup_steps, save_checkpoint_steps, keep_checkpoint\n",
        "\n",
        "config = '''# config.yaml\n",
        "\n",
        "\n",
        "## Where the samples will be written\n",
        "save_data: run\n",
        "\n",
        "# Training files\n",
        "data:\n",
        "    corpus_1:\n",
        "        path_src: UN.en-fr.fr-filtered.fr.subword.train\n",
        "        path_tgt: UN.en-fr.en-filtered.en.subword.train\n",
        "        transforms: [filtertoolong]\n",
        "    valid:\n",
        "        path_src: UN.en-fr.fr-filtered.fr.subword.dev\n",
        "        path_tgt: UN.en-fr.en-filtered.en.subword.dev\n",
        "        transforms: [filtertoolong]\n",
        "\n",
        "# Vocabulary files, generated by onmt_build_vocab\n",
        "src_vocab: run/source.vocab\n",
        "tgt_vocab: run/target.vocab\n",
        "\n",
        "# Vocabulary size - should be the same as in sentence piece\n",
        "src_vocab_size: 50000\n",
        "tgt_vocab_size: 50000\n",
        "\n",
        "# Filter out source/target longer than n if [filtertoolong] enabled\n",
        "src_seq_length: 150\n",
        "src_seq_length: 150\n",
        "\n",
        "# Tokenization options\n",
        "src_subword_model: source.model\n",
        "tgt_subword_model: target.model\n",
        "\n",
        "# Where to save the log file and the output models/checkpoints\n",
        "log_file: train.log\n",
        "save_model: models/model.fren\n",
        "\n",
        "# Stop training if it does not imporve after n validations\n",
        "early_stopping: 4\n",
        "\n",
        "# Default: 5000 - Save a model checkpoint for each n\n",
        "save_checkpoint_steps: 1000\n",
        "\n",
        "# To save space, limit checkpoints to last n\n",
        "# keep_checkpoint: 3\n",
        "\n",
        "seed: 3435\n",
        "\n",
        "# Default: 100000 - Train the model to max n steps \n",
        "# Increase to 200000 or more for large datasets\n",
        "# For fine-tuning, add up the required steps to the original steps\n",
        "train_steps: 3000\n",
        "\n",
        "# Default: 10000 - Run validation after n steps\n",
        "valid_steps: 1000\n",
        "\n",
        "# Default: 4000 - for large datasets, try up to 8000\n",
        "warmup_steps: 1000\n",
        "report_every: 100\n",
        "\n",
        "# Number of GPUs, and IDs of GPUs\n",
        "world_size: 1\n",
        "gpu_ranks: [0]\n",
        "\n",
        "# Batching\n",
        "bucket_size: 262144\n",
        "num_workers: 0  # Default: 2, set to 0 when RAM out of memory\n",
        "batch_type: \"tokens\"\n",
        "batch_size: 4096   # Tokens per batch, change when CUDA out of memory\n",
        "valid_batch_size: 2048\n",
        "max_generator_batches: 2\n",
        "accum_count: [4]\n",
        "accum_steps: [0]\n",
        "\n",
        "# Optimization\n",
        "model_dtype: \"fp16\"\n",
        "optim: \"adam\"\n",
        "learning_rate: 2\n",
        "# warmup_steps: 8000\n",
        "decay_method: \"noam\"\n",
        "adam_beta2: 0.998\n",
        "max_grad_norm: 0\n",
        "label_smoothing: 0.1\n",
        "param_init: 0\n",
        "param_init_glorot: true\n",
        "normalization: \"tokens\"\n",
        "\n",
        "# Model\n",
        "encoder_type: transformer\n",
        "decoder_type: transformer\n",
        "position_encoding: true\n",
        "enc_layers: 6\n",
        "dec_layers: 6\n",
        "heads: 8\n",
        "hidden_size: 512\n",
        "word_vec_size: 512\n",
        "transformer_ff: 2048\n",
        "dropout_steps: [0]\n",
        "dropout: [0.1]\n",
        "attention_dropout: [0.1]\n",
        "'''\n",
        "\n",
        "with open(\"config.yaml\", \"w+\") as config_yaml:\n",
        "  config_yaml.write(config)"
      ],
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vsL4zycvLMUx"
      },
      "source": [
        "# [Optional] Check the content of the configuration file\n",
        "!cat config.yaml"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F0bcqYkEXhRY"
      },
      "source": [
        "# Build Vocabulary\n",
        "\n",
        "For large datasets, it is not feasable to use all words/tokens found in the corpus. Instead, a specific set of vocabulary is extracted from the training dataset, usually betweeen 32k and 100k words. This is the main purpose of the vocabulary building step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AuwltKp_VhnQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4d9d5e5e-df7b-474b-b281-369424c47603"
      },
      "source": [
        "# Find the number of CPUs/cores on the machine\n",
        "!nproc --all"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P2GV1PgyUsJr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e1749505-d1f6-41cd-9420-1876db27e405"
      },
      "source": [
        "# Build Vocabulary\n",
        "\n",
        "# -config: path to your config.yaml file\n",
        "# -n_sample: use -1 to build vocabulary on all the segment in the training dataset\n",
        "# -num_threads: change it to match the number of CPUs to run it faster\n",
        "\n",
        "!onmt_build_vocab -config config.yaml -n_sample -1 -num_threads 2"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Corpus corpus_1's weight should be given. We default it to 1 for you.\n",
            "[2022-11-19 20:00:16,415 INFO] Counter vocab from -1 samples.\n",
            "[2022-11-19 20:00:16,415 INFO] n_sample=-1: Build vocab on full datasets.\n",
            "[2022-11-19 20:00:18,967 INFO] * Transform statistics for corpus_1(50.00%):\n",
            "\t\t\t* FilterTooLongStats(filtered=2138)\n",
            "\n",
            "[2022-11-19 20:00:18,976 INFO] * Transform statistics for corpus_1(50.00%):\n",
            "\t\t\t* FilterTooLongStats(filtered=2032)\n",
            "\n",
            "[2022-11-19 20:00:19,035 INFO] Counters src:14705\n",
            "[2022-11-19 20:00:19,035 INFO] Counters tgt:11884\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ncWyNtxiO_Ov"
      },
      "source": [
        "From the **Runtime menu** > **Change runtime type**, make sure that the \"**Hardware accelerator**\" is \"**GPU**\".\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TMMPeS-pSV8I",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ea51133a-beaa-4642-e8ba-7bd9159ada68"
      },
      "source": [
        "# Check if the GPU is active\n",
        "!nvidia-smi -L"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GPU 0: Tesla T4 (UUID: GPU-1759f39f-df0c-a03f-3066-463f5fec7c38)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_3rVQhd4NXNG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "181eb6a3-cc09-45b6-de4e-b1e88e45f97b"
      },
      "source": [
        "# Check if the GPU is visable to PyTorch\n",
        "\n",
        "import torch\n",
        "\n",
        "print(torch.cuda.is_available())\n",
        "print(torch.cuda.get_device_name(0))\n",
        "\n",
        "gpu_memory = torch.cuda.mem_get_info(0)\n",
        "print(\"Free GPU memory:\", gpu_memory[0]/1024**2, \"out of:\", gpu_memory[1]/1024**2)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "True\n",
            "Tesla T4\n",
            "Free GPU memory: 15007.75 out of: 15109.75\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8aCxETSnXcL-"
      },
      "source": [
        "# Training\n",
        "\n",
        "Now, start training your NMT model! 🎉 🎉 🎉"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!rm -rf drive/MyDrive/nmt/models/"
      ],
      "metadata": {
        "id": "HZd1o1kIb6Nv"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "prJCKA2CP-dl"
      },
      "source": [
        "# Train the NMT model\n",
        "!onmt_train -config config.yaml"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# For error debugging try:\n",
        "# !dmesg -T"
      ],
      "metadata": {
        "id": "XUYAvE8ffK2k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eShpS01j-Jcp"
      },
      "source": [
        "# Translation\n",
        "\n",
        "Translation Options:\n",
        "* `-model` - specify the last model checkpoint name; try testing the quality of multiple checkpoints\n",
        "* `-src` - the subworded test dataset, source file\n",
        "* `-output` - give any file name to the new translation output file\n",
        "* `-gpu` - GPU ID, usually 0 if you have one GPU. Otherwise, it will translate on CPU, which would be slower.\n",
        "* `-min_length` - [optional] to avoid empty translations\n",
        "* `-verbose` - [optional] if you want to print translations\n",
        "\n",
        "Refer to [OpenNMT-py translation options](https://opennmt.net/OpenNMT-py/options/translate.html) for more details."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MbQEGTj4TybH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b2181f89-47a1-46d3-888c-9facf296bf61"
      },
      "source": [
        "# Translate the \"subworded\" source file of the test dataset\n",
        "# Change the model name, if needed.\n",
        "!onmt_translate -model models/model.fren_step_3000.pt -src UN.en-fr.fr-filtered.fr.subword.test -output UN.en.translated -gpu 0 -min_length 1"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[2022-11-19 21:00:48,673 INFO] PRED SCORE: -0.2185, PRED PPL: 1.24 NB SENTENCES: 2000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XHYihrgfIrIO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "980d6dad-3467-4157-a398-8fe05509cb54"
      },
      "source": [
        "# Check the first 5 lines of the translation file\n",
        "!head -n 5 UN.en.translated"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "▁ 1 1 . ▁Reaffirms ▁the ▁commitments ▁made ▁at ▁the ▁Fourth ▁M inisterial ▁Conference ▁of ▁the ▁World ▁Trade ▁Organization , ▁held ▁in ▁Doha , ▁and ▁the ▁Thir d ▁Unit ed ▁Nations ▁Conference ▁on ▁the ▁Lea st ▁Developed ▁Countries , ▁held ▁in ▁Brussels ▁from ▁ 1 4 ▁to ▁ 2 0 ▁May ▁ 2 0 0 1 , See ▁A / CONF . 1 9 1 / 1 1 1 ▁and ▁ 1 2 . ▁and ▁in ▁this ▁regard ▁calls ▁upon ▁developed ▁countries ▁that ▁have ▁not ▁ye t ▁done ▁so ▁to ▁work ▁together ▁to ▁work ▁towards ▁the ▁objective ▁of ▁duty - free ▁and ▁quota - free ▁market ▁access ▁for ▁all ▁least ▁developed ▁countries , ▁and ▁also ▁notes ▁that ▁the ▁proposals ▁for ▁developing ▁countries ▁would ▁be ▁helpful ;\n",
            "▁ 1 0 . ▁Recognizes , ▁where ▁appropriate , ▁international ▁cooperation ▁in ▁the ▁field ▁of ▁critical ▁information ▁infrastructures , ▁including ▁the ▁establishment ▁of ▁emergency ▁information ▁and ▁coordination ▁mechanisms , ▁as ▁well ▁as ▁the ▁sharing ▁of ▁experience , ▁information ▁on ▁and ▁ te lecommunications , ▁response ▁and ▁solutions ▁to ▁such ▁incidents , ▁and ▁the ▁identification ▁of ▁incidents ▁in ▁accord ance ▁with ▁domestic ▁law .\n",
            "▁Recalling ▁its ▁relevant ▁resolutions , ▁including ▁resolution ▁ 5 8 / 2 9 2 ▁of ▁ 6 ▁May ▁ 2 0 0 4 , ▁as ▁well ▁as ▁th ose ▁adopted ▁at ▁its ▁tenth ▁emergency ▁special ▁session ,\n",
            "▁ 1 6 . ▁Further ▁requests ▁the ▁Secretary - General ▁to ▁report ▁to ▁the ▁General ▁Assembly ▁at ▁its ▁ resumed ▁fifty - ninth ▁session ▁on ▁the ▁experience ▁of ▁human ▁resources ▁management ;\n",
            "▁ 6 . ▁Encourages ▁all ▁States ▁of ▁the ▁region ▁to ▁foster ▁conditions ▁conduc ive ▁to ▁strengthening ▁mutual ▁confidence - building ▁measures ▁and ▁genuine ▁openness , ▁transparency ▁in ▁all ▁military ▁matters , ▁in ▁particular ▁through ▁the ▁Unit ed ▁Nations ▁system ▁for ▁the ▁standardiz ed ▁reporting ▁of ▁military ▁expenditures ▁and ▁the ▁Unit ed ▁Nations ▁Register ▁of ▁Convention al ▁Arms ▁and ▁L ight ▁Weapons ; See ▁resolution ▁ 4 6 / 3 6 .\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zRsJm6UET2C_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f9a410d7-e753-4c43-e5cb-82c62ed00a53"
      },
      "source": [
        "# If needed install/update sentencepiece\n",
        "!pip3 install --upgrade -q sentencepiece\n",
        "\n",
        "# Desubword the translation file\n",
        "!python3 MT-Preparation/subwording/3-desubword.py target.model UN.en.translated"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l\r\u001b[K     |▎                               | 10 kB 27.3 MB/s eta 0:00:01\r\u001b[K     |▌                               | 20 kB 13.1 MB/s eta 0:00:01\r\u001b[K     |▊                               | 30 kB 16.9 MB/s eta 0:00:01\r\u001b[K     |█                               | 40 kB 14.2 MB/s eta 0:00:01\r\u001b[K     |█▎                              | 51 kB 12.4 MB/s eta 0:00:01\r\u001b[K     |█▌                              | 61 kB 14.3 MB/s eta 0:00:01\r\u001b[K     |█▉                              | 71 kB 15.6 MB/s eta 0:00:01\r\u001b[K     |██                              | 81 kB 17.1 MB/s eta 0:00:01\r\u001b[K     |██▎                             | 92 kB 17.3 MB/s eta 0:00:01\r\u001b[K     |██▋                             | 102 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██▉                             | 112 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███                             | 122 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███▍                            | 133 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███▋                            | 143 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███▉                            | 153 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████                            | 163 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████▍                           | 174 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████▋                           | 184 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████▉                           | 194 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████▏                          | 204 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████▍                          | 215 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████▋                          | 225 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████                          | 235 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████▏                         | 245 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████▍                         | 256 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████▊                         | 266 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████                         | 276 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████▏                        | 286 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████▍                        | 296 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████▊                        | 307 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████                        | 317 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████▏                       | 327 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████▌                       | 337 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████▊                       | 348 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████                       | 358 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████▎                      | 368 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████▌                      | 378 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████▊                      | 389 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████                      | 399 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████▎                     | 409 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████▌                     | 419 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████▉                     | 430 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████                     | 440 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████▎                    | 450 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████▌                    | 460 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████▉                    | 471 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████                    | 481 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████▎                   | 491 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████▋                   | 501 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████▉                   | 512 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████                   | 522 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████▍                  | 532 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████▋                  | 542 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████▉                  | 552 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████▏                 | 563 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████▍                 | 573 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████▋                 | 583 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████▉                 | 593 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████▏                | 604 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████▍                | 614 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████▋                | 624 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████                | 634 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████▏               | 645 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████▍               | 655 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████▊               | 665 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████               | 675 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████▏              | 686 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████▌              | 696 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████▊              | 706 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████              | 716 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████▎             | 727 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████▌             | 737 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████▊             | 747 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████             | 757 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████▎            | 768 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████▌            | 778 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████▊            | 788 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████            | 798 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 808 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████▌           | 819 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████▉           | 829 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████           | 839 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████▎          | 849 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████▋          | 860 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████▉          | 870 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████          | 880 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████▎         | 890 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████▋         | 901 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████▉         | 911 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████         | 921 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████▍        | 931 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████▋        | 942 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████▉        | 952 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████████▏       | 962 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████████▍       | 972 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████████▋       | 983 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████       | 993 kB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▏      | 1.0 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▍      | 1.0 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▊      | 1.0 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████      | 1.0 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▏     | 1.0 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▍     | 1.1 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▊     | 1.1 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████     | 1.1 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▏    | 1.1 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▌    | 1.1 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▊    | 1.1 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████    | 1.1 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▎   | 1.1 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▌   | 1.1 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▊   | 1.1 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████   | 1.2 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▎  | 1.2 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▌  | 1.2 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▊  | 1.2 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 1.2 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▎ | 1.2 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▌ | 1.2 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▉ | 1.2 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████ | 1.2 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▎| 1.2 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▋| 1.3 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▉| 1.3 MB 15.7 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 1.3 MB 15.7 MB/s \n",
            "\u001b[?25hDone desubwording! Output: UN.en.translated.desubword\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ai4RhhGaKBp1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a833acd0-99d0-48ed-ad63-dac8f91ff4da"
      },
      "source": [
        "# Check the first 5 lines of the desubworded translation file\n",
        "!head -n 5 UN.en.translated.desubword"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "11. Reaffirms the commitments made at the Fourth Ministerial Conference of the World Trade Organization, held in Doha, and the Third United Nations Conference on the Least Developed Countries, held in Brussels from 14 to 20 May 2001,See A/CONF.191/111 and 12. and in this regard calls upon developed countries that have not yet done so to work together to work towards the objective of duty-free and quota-free market access for all least developed countries, and also notes that the proposals for developing countries would be helpful;\n",
            "10. Recognizes, where appropriate, international cooperation in the field of critical information infrastructures, including the establishment of emergency information and coordination mechanisms, as well as the sharing of experience, information on and telecommunications, response and solutions to such incidents, and the identification of incidents in accordance with domestic law.\n",
            "Recalling its relevant resolutions, including resolution 58/292 of 6 May 2004, as well as those adopted at its tenth emergency special session,\n",
            "16. Further requests the Secretary-General to report to the General Assembly at its resumed fifty-ninth session on the experience of human resources management;\n",
            "6. Encourages all States of the region to foster conditions conducive to strengthening mutual confidence-building measures and genuine openness, transparency in all military matters, in particular through the United Nations system for the standardized reporting of military expenditures and the United Nations Register of Conventional Arms and Light Weapons;See resolution 46/36.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kOUWB4r3OFOV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c16d31db-0643-4e05-c72a-f3982b4c6ebb"
      },
      "source": [
        "# Desubword the target file (reference) of the test dataset\n",
        "# Note: You might as well have split files *before* subwording during dataset preperation, \n",
        "# but sometimes datasets have tokeniztion issues, so this way you are sure the file is really untokenized.\n",
        "!python3 MT-Preparation/subwording/3-desubword.py target.model UN.en-fr.en-filtered.en.subword.test"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Done desubwording! Output: UN.en-fr.en-filtered.en.subword.test.desubword\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0jULN0MwOFeH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c240f715-d7f8-451f-9bf6-1cf344af6810"
      },
      "source": [
        "# Check the first 5 lines of the desubworded reference\n",
        "!head -n 5 UN.en-fr.en-filtered.en.subword.test.desubword"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "11. Reaffirms the commitments made at the Fourth Ministerial Conference of the World Trade Organization held at Doha and at the Third United Nations Conference on the Least Developed Countries, held at Brussels from 14 to 20 May 2001,See A/CONF.191/11 and 12. and in this regard calls upon developed countries that have not already done so to work towards the objective of duty-free, quota-free market access for all least developed countries' exports, and notes that consideration of proposals for developing countries to contribute to improved market access for least developed countries would also be helpful;\n",
            "10. Engage in international cooperation, when appropriate, to secure critical information infrastructures, including by developing and coordinating emergency warning systems, sharing and analysing information regarding vulnerabilities, threats and incidents and coordinating investigations of attacks on such infrastructures in accordance with domestic laws.\n",
            "Recalling its relevant resolutions, including resolution 58/292 of 6 May 2004, as well as those resolutions adopted at its tenth emergency special session,\n",
            "16. Further requests the Secretary-General to report to the General Assembly at its resumed fifty-ninth session on the implications of the experiment for human resources management policies;\n",
            "6. Encourages all States of the region to favour the necessary conditions for strengthening the confidence-building measures among them by promoting genuine openness and transparency on all military matters, by participating, inter alia, in the United Nations system for the standardized reporting of military expenditures and by providing accurate data and information to the United Nations Register of Conventional Arms;See resolution 46/36 L.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bHMumxqvLDDc"
      },
      "source": [
        "# MT Evaluation\n",
        "\n",
        "There are several MT Evaluation metrics such as BLEU, TER, METEOR, COMET, BERTScore, among others.\n",
        "\n",
        "Here we are using BLEU. Files must be detokenized/desubworded beforehand."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w-9XGYnaJ-Nj"
      },
      "source": [
        "# Download the BLEU script\n",
        "!wget https://github.com/raw/ymoslem/MT-Evaluation/main/BLEU/compute-bleu.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rYDG0x0KLk_O"
      },
      "source": [
        "# Install sacrebleu\n",
        "!pip3 install sacrebleu"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W3V3tZphTzK9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c2a85bb9-9a25-420e-98fd-800a554aae79"
      },
      "source": [
        "# Evaluate the translation (without subwording)\n",
        "!python3 compute-bleu.py UN.en-fr.en-filtered.en.subword.test.desubword UN.en.translated.desubword"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reference 1st sentence: 11. Reaffirms the commitments made at the Fourth Ministerial Conference of the World Trade Organization held at Doha and at the Third United Nations Conference on the Least Developed Countries, held at Brussels from 14 to 20 May 2001,See A/CONF.191/11 and 12. and in this regard calls upon developed countries that have not already done so to work towards the objective of duty-free, quota-free market access for all least developed countries' exports, and notes that consideration of proposals for developing countries to contribute to improved market access for least developed countries would also be helpful;\n",
            "MTed 1st sentence: 11. Reaffirms the commitments made at the Fourth Ministerial Conference of the World Trade Organization, held in Doha, and the Third United Nations Conference on the Least Developed Countries, held in Brussels from 14 to 20 May 2001,See A/CONF.191/111 and 12. and in this regard calls upon developed countries that have not yet done so to work together to work towards the objective of duty-free and quota-free market access for all least developed countries, and also notes that the proposals for developing countries would be helpful;\n",
            "BLEU:  63.13825360368016\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IBi1PhRv4bX9"
      },
      "source": [
        "# More Features and Directions to Explore\n",
        "\n",
        "Experiment with the following ideas:\n",
        "* Icrease `train_steps` and see to what extent new checkpoints provide better translation, in terms of both BLEU and your human evaluation.\n",
        "\n",
        "* Check other MT Evaluation mentrics other than BLEU such as [TER](https://github.com/mjpost/sacrebleu#ter), [WER](https://blog.machinetranslation.io/compute-wer-score/), [METEOR](https://blog.machinetranslation.io/compute-bleu-score/#meteor), [COMET](https://github.com/Unbabel/COMET), and [BERTScore](https://github.com/Tiiiger/bert_score). What are the conceptual differences between them? Is there special cases for using a specific metric?\n",
        "\n",
        "* Continue training from the last model checkpoint using the `-train_from` option, only if the training stopped and you want to continue it. In this case, `train_steps` in the config file should be larger than the steps of the last checkpoint you train from.\n",
        "```\n",
        "!onmt_train -config config.yaml -train_from models/model.fren_step_3000.pt\n",
        "```\n",
        "\n",
        "* **Ensemble Decoding:** During translation, instead of adding one model/checkpoint to the `-model` argument, add multiple checkpoints. For example, try the two last checkpoints. Does it improve quality of translation? Does it affect translation seepd?\n",
        "\n",
        "* **Averaging Models:** Try to average multiple models into one model using the [average_models.py](https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/bin/average_models.py) script, and see how this affects translation quality.\n",
        "```\n",
        "python3 average_models.py -models model_step_xxx.pt model_step_yyy.pt -output model_avg.pt\n",
        "```\n",
        "* **Release the model:** Try this command and see how it reduce the model size.\n",
        "```\n",
        "onmt_release_model --model \"model.pt\" --output \"model_released.pt\n",
        "```\n",
        "* **Use CTranslate2:** For efficient translation, consider using [CTranslate2](https://github.com/OpenNMT/CTranslate2), a fast inference engine. Check out an [example](https://gist.github.com/ymoslem/60e1d1dc44fe006f67e130b6ad703c4b).\n",
        "\n",
        "* **Work on low-resource languages:** Find out more details about [how to train NMT models for low-resource languages](https://blog.machinetranslation.io/low-resource-nmt/).\n",
        "\n",
        "* **Train a multilingual model:** Find out helpful notes about [training multilingual models](https://blog.machinetranslation.io/multilingual-nmt).\n",
        "\n",
        "* **Publish a demo:** Show off your work through a [simple demo with CTranslate2 and Streamlit](https://blog.machinetranslation.io/nmt-web-interface/).\n"
      ]
    }
  ]
}