This is a set of scripts for downloading parallel corpora, fixing and cleaning them. It uses MTData to download and then some rule-based filtering. It started using the cleaning scripts from Bergamot students repository but I added a few more things.
Clone the repository and install python modules:
git clone --recursive https://github.com/ZJaume/clean
cd clean
pip install -r requirements.txt
pip install -r tools/bifixer/requirements.txt
Install zstd for compression of intermediate files:
sudo apt install zstd
Usage: create-corpus.sh [options]
Options:
-B BLOCK Block size of parallel
-c CORPORA Comma-separated list of corpora
from 'mtdata list -l SRC-TRG'
-C CACHE mtdata cache directory
Default: ./cache
-j JOBS Number of jobs of parallel
-l SRC-TRG Language pair
-s SIZE Number of sentences
First of all look at the available corpora with mtdata
:
$ mtdata list -l en-mt
INFO:root:Loaded entries: Statmt.org:355 Paracrawl:59 Tilde:519 JoshuaIndianCoprus:29 GlobalVoices:812 UnitedNations:30 OPUS:53,321 OPUS_JW300:44,663 OPUS100:302 WikiMatrix:1,617 Other:7 Neulab_TEDTalksv1:4,455 Total:106,169
WARNING:root:Suggestion: Use ISO 639_3 codes eng-mlt instead of en-mt. Let's make a little space for all 7000+ languages of our planet 😢.
INFO:root:Found 24
paracrawl_v6 eng-mlt https://s3.amazonaws.com/web-language-models/paracrawl/release6/en-mt.txt.gz
paracrawl_v7_1 eng-mlt https://s3.amazonaws.com/web-language-models/paracrawl/release7.1/en-mt.txt.gz
EESC2017 eng-mlt https://tilde-model.s3-eu-west-1.amazonaws.com/EESC2017.en-mt.tmx.zip *.tmx
EMA2016 eng-mlt https://tilde-model.s3-eu-west-1.amazonaws.com/EMA2016.en-mt.tmx.zip *.tmx
ecb2017 eng-mlt https://tilde-model.s3-eu-west-1.amazonaws.com/ecb2017.en-mt.tmx.zip *.tmx
rapid2016 eng-mlt https://tilde-model.s3-eu-west-1.amazonaws.com/rapid2016.en-mt.tmx.zip *.tmx
OPUS_EUconst_v1 eng-mlt http://opus.nlpl.eu/download.php?f=EUconst/v1/moses/en-mt.txt.zip *.en,*.mt
OPUS_EUbookshop_v2 eng-mlt http://opus.nlpl.eu/download.php?f=EUbookshop/v2/moses/en-mt.txt.zip *.en,*.mt
OPUS_EMEA_v3 eng-mlt http://opus.nlpl.eu/download.php?f=EMEA/v3/moses/en-mt.txt.zip *.en,*.mt
OPUS_ECB_v1 eng-mlt http://opus.nlpl.eu/download.php?f=ECB/v1/moses/en-mt.txt.zip *.en,*.mt
OPUS_DGT_v2019 eng-mlt http://opus.nlpl.eu/download.php?f=DGT/v2019/moses/en-mt.txt.zip *.en,*.mt
OPUS_wikimedia_v20190628 eng-mlt http://opus.nlpl.eu/download.php?f=wikimedia/v20190628/moses/en-mt.txt.zip *.en,*.mt
OPUS_Ubuntu_v14_10 eng-mlt http://opus.nlpl.eu/download.php?f=Ubuntu/v14.10/moses/en-mt.txt.zip *.en,*.mt
OPUS_TildeMODEL_v2018 eng-mlt http://opus.nlpl.eu/download.php?f=TildeMODEL/v2018/moses/en-mt.txt.zip *.en,*.mt
OPUS_Tatoeba_v20190709 eng-mlt http://opus.nlpl.eu/download.php?f=Tatoeba/v20190709/moses/en-mt.txt.zip *.en,*.mt
OPUS_QED_v2_0a eng-mlt http://opus.nlpl.eu/download.php?f=QED/v2.0a/moses/en-mt.txt.zip *.en,*.mt
OPUS_ParaCrawl_v5 eng-mlt http://opus.nlpl.eu/download.php?f=ParaCrawl/v5/moses/en-mt.txt.zip *.en,*.mt
OPUS_KDE4_v2 eng-mlt http://opus.nlpl.eu/download.php?f=KDE4/v2/moses/en-mt.txt.zip *.en,*.mt
OPUS_JRC_Acquis eng-mlt http://opus.nlpl.eu/download.php?f=JRC-Acquis/en-mt.txt.zip *.en,*.mt
OPUS_GNOME_v1 eng-mlt http://opus.nlpl.eu/download.php?f=GNOME/v1/moses/en-mt.txt.zip *.en,*.mt
JW300 eng-mlt http://opus.nlpl.eu/download.php?f=JW300/v1/xml/en-mt.xml.gz http://opus.nlpl.eu/download.php?f=JW300/v1/xml/en.zip,http://opus.nlpl.eu/download.php?f=JW300/v1/xml/mt.zip
Then, run the main script specifying the desired corpora passing their mtdata
id's separated by commas:
./create-corpus.sh -l en-mt -c OPUS_TildeMODEL_v2018,JW300,OPUS_Tatoeba_v20190709,OPUS_ECB_v1
The script will download the corpora with mtdata
, apply some fixes, clean, concatenation and near-deduplication.
Specific corpus fixes can be applied adding custom executable scripts at fixes/
directoryi that will be called inside the pipeline.
These scripts must follow the naming fixes/corpus_id.sh
for processing parallel tab-separated input or fixes/corpus_id.lang.sh
for processing monolingual data.
For example, the fixes/JW300.mt.sh
reads monolingual data and fixes some tokenization issues present in the JW300 corpus of Maltese:
#!/bin/bash
# Fix Maltese tokenization in JW300 that detokenizer cannot fix
sed "s/ - $(echo -ne \u200b) /-/g" \
| sed 's/ - /-/g'
Or the fixes/JW300.sh
that reads the tab-separated input, detokenizes it and then it prints to stdout in the same tab-separated format and fixed:
#!/bin/bash
set -e
# Detokenize JW300
SRC=$1
TRG=$2
temp=$(mktemp -d)
tee >(cut -f1 | sacremoses -j 6 -l $SRC detokenize >$temp/$SRC.detok) \
>(cut -f2 | sacremoses -j 6 -l $TRG detokenize >$temp/$TRG.detok)
paste $temp/$SRC.detok $temp/$TRG.detok
rm -r $temp
Note that the scripts that process parallel data will be called with the language identifiers as arguments.