Support Qwen type llm's

[CyberTimon]

⚠️ Please check that this feature request hasn't been suggested before.

• I searched previous Ideas in Discussions didn't find any similar feature requests.
• I searched previous Issues didn't find any similar feature requests.

🔖 Feature description

With the new qwen 14b model it would be very nice to have axolotl support.

✔️ Solution

-Support finetuning of Qwen type llm's

❓ Alternatives

Normal finetuning but it's unoptimized

📝 Additional Context

No response

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this feature has not been requested yet.
• I have provided enough information for the maintainers to understand and evaluate this request.

[NanoCode012]

Could you please try with trust_remote_code: true and using AutoModelCausalLM and AutoTokenizer? Is there an error?

[Peter-Devine]

I'd like to second this. trust_remote_code: true seems to work for full finetuning, but I get errors for qlora training. Is fully tested support for Qwen on the roadmap for this project? It would be really useful for CJK languages!

[CheshireAI]

I get errors: raise ValueError("Adding unknown special tokens is not supported")
ValueError: Adding unknown special tokens is not supported
trying to use qlora with qwen

[kostum123]

Has anyone been able to fine-tune it with qlora?

[NanoCode012]

I get errors: raise ValueError("Adding unknown special tokens is not supported") ValueError: Adding unknown special tokens is not supported trying to use qlora with qwen

@CheshireAI , Hm, this is an upstream problem. If you want this feature, you should ask at transformers.

@Peter-Devine @kostum123 , what's the error with running qlora?

[NanoCode012]

Hello, I apologize for the long delay. This should be fixed here: #894

Please let me know how it goes.

[CyberTimon]

Thank you very much! Will try it out soon.