ELMo embeddings were presented by Peters et al. in 2018. They are using
a bidirectional recurrent neural network to predict the next word in a text.
We are using the implementation of AllenNLP. As this implementation comes with a lot of
sub-dependencies, which we don't want to include in Flair, you need to first install the library via
pip install allennlp==0.9.0 before you can use it in Flair.
Using the embeddings is as simple as using any other embedding type:
from flair.embeddings import ELMoEmbeddings
# init embedding
embedding = ELMoEmbeddings()
# create a sentence
sentence = Sentence('The grass is green .')
# embed words in sentence
embedding.embed(sentence)ELMo word embeddings can be constructed by combining ELMo layers in different ways. The available combination strategies are:
"all": Use the concatenation of the three ELMo layers."top": Use the top ELMo layer."average": Use the average of the three ELMo layers.
By default, the top 3 layers are concatenated to form the word embedding.
AllenNLP provides the following pre-trained models. To use any of the following models inside Flair
simple specify the embedding id when initializing the ELMoEmbeddings.
| ID | Language | Embedding |
|---|---|---|
| 'small' | English | 1024-hidden, 1 layer, 14.6M parameters |
| 'medium' | English | 2048-hidden, 1 layer, 28.0M parameters |
| 'original' | English | 4096-hidden, 2 layers, 93.6M parameters |
| 'large' | English | |
| 'pt' | Portuguese | |
| 'pubmed' | English biomedical data | more information |