-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to convert a AutoModelForCausalLM object to a dspy model object? #1018
Comments
I see, but internally hf module internally uses AutoModel module to instantiate weights. So can you explain me why we require to use already loaded model to dspy instead of giving the weight path? |
Thanks for reply, The reason is I have a 4 bit quantized model and i want to use it directly. I have tried to save it first to hugging face so that i can load it from weight path but then error comes that the Hugging face does not support saving 4 bit quantized model. |
can you please share the full code for the loading process and your approach? Would appreciate this. |
dspy_4bitquantized_llama2_error.zip |
Hey @pawanGithub10 I have started to raise a PR by seeing the issue that you faced. Here are some of the cases of loading models would look like: from dsp.modules.hf_new import HFLocalModel
from transformers import AutoTokenizer, BitsAndBytesConfig
from transformers import AutoModelForCausalLM
model_path = "../models/llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(
model_path,
)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
def case1():
model = HFLocalModel(
model=model_path,
tokenizer=tokenizer,
load_in_4bit=True ,
bnb_config=BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="float16",
bnb_4bit_use_double_quant=False
)
)
response = model("hello", do_sample=True)
print(response)
def case2():
model = AutoModelForCausalLM.from_pretrained(
model_path,
quantization_config=BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="float16",
bnb_4bit_use_double_quant=False
)
)
model_ = HFLocalModel(
model=model,
tokenizer=tokenizer,
)
response = model_("hello", do_sample=True)
print(response)
if __name__ == "__main__":
case1()
print("---------------------------")
case2() Additionally, adding PEFT models are also now supported, with multi gpu support. Now the problem is, I would be able to test till PEFT, for multi gpu support, the tests are not possible, since have no access with multi gpu setting. |
@Anindyadeep thanks a lot for the detailed help but i feel that i have missed the documentation details. nit signature: Args: So after reading this i have made following changes and the code work import dspy |
As per the previous comment i think the issue can be closed |
import dspy
llm = dspy.HFModel(model='model')
This method takes a string as input for the model if i have a quantized model object of the class AutoModelForCausalLM How i can convert the model object to dspy object?
direct assignment gives error on inference
llm = model #previously created as AutoModelForCausalLM class object
llm("Testing testing, is anyone out there?")
Error After Code Line 4 File /opt/conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:623, in LlamaModel.forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict) 621 raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time") 622 elif input_ids is not None: --> 623 batch_size, seq_length = input_ids.shape 624 elif inputs_embeds is not None: 625 batch_size, seq_length, _ = inputs_embeds.shape
AttributeError: 'str' object has no attribute 'shape'
The text was updated successfully, but these errors were encountered: