You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Same data no. I have trained with openwebtext (a open version of web text, part of Roberta training data) with a base architecture Roberta. It does better on sentence similarity task but not on retrieval tasks, when compared with Bert condenser. As a side note, we observed previously that vanilla Roberta base is typically inferior to vanilla Bert base on retrieval tasks.
We have just started test runs with condenser-roberta-large and therefore not much to say there yet.
I pretrained a condeser-roberta-base on the same data and hyperparameters, but the results on downstream tasks were not high.
Have you ever tried condenser pretraining on RoBERTa-base ?
Thank you
The text was updated successfully, but these errors were encountered: