This repository presents on overview of pretrained language models on the British Library corpus.
Project is part of the 🌼 BigScience "Language models for historical texts" working group.
- 27.01.2023: Initial version of this repo.
The British Library Corpus - available from here and from the Datasets Hub - is used to pretrain various language models.
The following filtering steps were performed:
langdetectis used to extract English texts only- Only texts from >= 1800 and < 1900 are used
The final corpus has a size of 24GB and tokens. An overview of the complete filtering steps can be found here.
For BERT/ELECTRA and ConvBERT we use the same 32k wordpiece vocabulary, trained on the whole corpus.
For T5 a 32k vocabulary is trained with sentencepiece.
All details can be found here.
All pretraining steps (incl. training data generation) are document in the model specific cheatsheets:
We pretrain all models on a v3-32 TPU pod from the awesome TPU Research Cloud program.
The following models are available on the Hugging Face Model Hub (currently flagged as private):
| Model Name | Pretraining Time | Parameters |
|---|---|---|
bigscience-historical-texts/bert-base-blbooks-cased |
1.64d | 110,617,344 |
bigscience-historical-texts/electra-base-blbooks-cased-discriminator |
2.69d | 110,026,752 |
bigscience-historical-texts/electra-base-blbooks-cased-generator |
2.69d | 34,646,272 |
bigscience-historical-texts/convbert-base-blbooks-cased |
3.83d | 106,815,624 |
bigscience-historical-texts/t5-efficient-blbooks-small-el32 |
0.81d | 142,322,176 |
bigscience-historical-texts/t5-efficient-blbooks-base-nl36 |
1.98d | 619,357,440 |
bigscience-historical-texts/t5-efficient-blbooks-large-nl36 |
2.98d | 1,090,051,072 |
All models are evaluated on the AjMC dataset from HIPE-2022 Shared Task.
The Flair library is used to load the dataset and a basic hyper-parameter search is performed.
Here's an overview of the results on the development split - F1-Score over 5 runs is reported:
| Model | Best Configuration | F1-Score |
|---|---|---|
| BERT | bs8-e10-lr5e-05 |
85.92 ± 0.53 |
| ELECTRA | bs4-e10-lr5e-05 |
85.53 ± 0.61 |
| ConvBERT | bs4-e10-lr5e-05 |
86.43 ± 0.82 |
| T5-Small | bs4-e10-lr0.00016 |
84.12 ± 1.11 |
| T5-Base | bs4-e10-lr0.00016 |
85.58 ± 0.62 |
| T5-Large | bs4-e10-lr0.00016 |
85.91 ± 1.09 |
For T5, encoder-only fine-tuning is performed. More details can be found here.
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs ❤️