Would you release the tutorial about how to generate bpe.codes and dict.txt file?

[RyanHuangNLP]

Would you release the tutorial about how to generate bpe.codes and dict.txt file, and the preprocess pipeline about how generate pretrain data?

I want to train a another language bertweet

[datquocnguyen]

Regrading bpe.codes: see https://github.com/glample/fastBPE

W.r.t. dict.txt: see https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md

To generate dict.txt you would have to REMOVE: --srcdict gpt2_bpe/dict.txt \ out of the data binarizing step, i.e. use:

fairseq-preprocess \
    --only-source \
    --trainpref wikitext-103-raw/wiki.train.bpe \
    --validpref wikitext-103-raw/wiki.valid.bpe \
    --testpref wikitext-103-raw/wiki.test.bpe \
    --destdir data-bin/wikitext-103 \
    --workers 60

[RyanHuangNLP]

mkdir -p gpt2_bpe
wget -O gpt2_bpe/encoder.json https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/encoder.json
wget -O gpt2_bpe/vocab.bpe https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/vocab.bpe
for SPLIT in train valid test; do \
    python -m examples.roberta.multiprocessing_bpe_encoder \
        --encoder-json gpt2_bpe/encoder.json \
        --vocab-bpe gpt2_bpe/vocab.bpe \
        --inputs wikitext-103-raw/wiki.${SPLIT}.raw \
        --outputs wikitext-103-raw/wiki.${SPLIT}.bpe \
        --keep-empty \
        --workers 60; \
done

for the encode step, is it necessary to replace the encoder to fastbpe? Would you give more details about encode steps?

Regrading bpe.codes: see https://github.com/glample/fastBPE

W.r.t. dict.txt: see https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md

To generate dict.txt you would have to REMOVE: --srcdict gpt2_bpe/dict.txt \ out of the data binarizing step, i.e. use:

fairseq-preprocess \
    --only-source \
    --trainpref wikitext-103-raw/wiki.train.bpe \
    --validpref wikitext-103-raw/wiki.valid.bpe \
    --testpref wikitext-103-raw/wiki.test.bpe \
    --destdir data-bin/wikitext-103 \
    --workers 60

[datquocnguyen]

Yes, you should use fastbpe. See: facebookresearch/fairseq#1186