BERT vs Word2vec

[vikaschib7]

Hello All,

Can you please help me out in getting similar words from the BERT model, as we do in Word2Vec?

Best regards,
Vikas

[TinaB19]

https://github.com/hanxiao/bert-as-service

[crapthings]

@Tina-19 is that repo just do encoding?

[TinaB19]

Yes, it provides fixed-length vectors for sentences using BERT that can be used instead of Word2Vec.

[vikaschib7]

Is it possible to see similar words in BERT like if I search for "radar sensor companies", so I can get the similar words related to above query @Tina-19 @andrewluchen @jacobdevlin-google

[HoaiDuyLe]

I think this link can help you.
#60

[apogiatzis]

One thing to realise is that word2vec provides context-free embeddings (static) whereas BERT gives contextualised embeddings (dynamic). For instance, take these two sentences "I like apples", "I like apple MacBooks". Word2vec will give the same embedding for the word apple in both sentences whereas BERT will give you a different one depending on the context.

Now coming back to your question, here is a step by step tutorial I wrote to obtain contextualised embeddings from BERT:
https://towardsdatascience.com/nlp-extract-contextualized-word-embeddings-from-bert-keras-tf-67ef29f60a7b