CPT pretrain problem

[SunyanGu]

when I use roberta_zh to pretrain CPT model , raise error "Error(s) in loading state_dict for BertModel". So what pretrained model shoud i use? Roberta or BERT ?

[choosewhatulike]

If you use roberta_zh with transformers, it would have the same architecture as BERT. And it should work.

[SunyanGu]

I'm sure that I use transformers to load roberta_zh.But the model which I download have error parameter name. Can you give me a link to download right roberta_zh?

[choosewhatulike]

This should work: https://huggingface.co/hfl/chinese-roberta-wwm-ext