如何加载改过模型结构之后的qwen2呢？

[BobTsang1995]

我现在想验证yarn的外推性能，所以我将qwen2-7b的Qwen2RotaryEmbedding在模型加载后replace成了yarn embedding，训练完成之后，我不太清楚在opencompass上如何加载一个修改过模型结构的qwen2，有人做过这方面的实验吗？

[rationalspark]

参见我在#1261中回复的配置文件（最后一行有问题的不加），即可。
但是，qwen2已经支持到128k上下文了，他本身应该是用个大base的ntk调出来的，换个yarn，训练数据没有官方调时多，估计不会有明显改进。
而且，如果想长上下文，你会遇到和我一样的oom问题，如能解决，还请分享经验，先感谢

[tonysy]

Reduce the batch size to avoid OOM.
Also yuo can try vLLM or LMDeploy to speedup the inference

[rationalspark]

I want to run qwen2_7b with a context length of 32k. The batch size is already set to 1, but OOM still occurs. While there are 8 A100-80G GPUs, the task cannot evenly spread over these cards. For some tasks where the 32K context length limit is not reached, the memory costs on 2 GPUs are close to 80GB, while those for the other 6 GPUs are roughly 10GB. Consequently, when the context length is close to 32, OOM cannot be resolved by setting 'num_gpus=8' since the memory on most of the GPUs are not utilized.

I truly appreciate your help in resolving the problem.

[tonysy]

As opencompass is not designed for inference, I think you can make sure that the model can be successfully loaded and inferenced using transformers without opencomass, as the first step.

[rationalspark]

In transformers without opencomass, the model can be loaded as:
model = AutoModel.from_pretrained(model_path, trust_remote_code=True, device_map="auto")
Instead of device_map="auto", it can be set like
device_map={'transformer.wte': 0, 'transformer.drop': 0, 'transformer.h.0': 1, 'transformer.h.1': 2, ...}

By such setting device_map properly, qwen2_7b can run with a context length of 32k with 8 A100-80G GPUs.

Then, the problem is that I cannot find a interface in opencompass to set device_map