AutoModel abstraction fails for pre-training initialization

[g-karthik]

Environment info

• transformers version: 4.5.1
• Python version: 3.6
• PyTorch version: 1.4+
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help

@patrickvonplaten, @LysandreJik

Information

Model I am using: GPT-2

The problem arises when using:

• [Y] my own modified scripts: (give details below)

To reproduce

Steps to reproduce the behavior:

from transformers import AutoConfig, AutoModelForCausalLM, GPT2LMHeadModel
config = AutoConfig.from_pretrained("gpt2", return_dict=True, gradient_checkpointing=False)

model_class = GPT2LMHeadModel
model = model_class(config)  # WORKS FINE

model_class = AutoModelForCausalLM
model = model_class(config)  # FAILS, stack trace below

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() takes 1 positional argument but 2 were given

Expected behavior

Both cases should work fine. The latter case should pull the former class internally.

[LysandreJik]

Hello! We recommend you read the docs regarding the AutoModel. I have linked you the from_config method which should be used in this use case.

[LysandreJik]

However, it is indeed unexpected for you to receive this error message. The message should be more explicit, investigating now.

[LysandreJik]

Opened #11956 for a more explicit error, and opening your use case for discussion.