Fix(model): Linear detected and added to target module with rope linear

[NanoCode012]

Discussion: https://discord.com/channels/1104757954588196865/1110594519226925137/1163762312247787570

ValueError: Target module LlamaLinearScalingRotaryEmbedding() is not supported. Currently, only the following modules are supported: torch.nn.Linear, torch.nn.Embedding, torch.nn.Conv2d, transformers.pytorch_utils.Conv1D

when lora + rope_scaling: linear. dynamic is ok.

This condition check must've included the linear layer from the name.

Alternative solution: add blacklist for this module.

Note: I'm not sure if by removing this would affect any current run, but the linear layers listed in cls seems to be inclusive.

[Napuh]

This PR works for Llama-2-7B.

However I can't confirm for other models.
Maybe it's safer to do something like:

if isinstance(module, cls) or "Linear" in module.__class__.__name__ and module.__class__ not in (LlamaLinearScalingRotaryEmbedding):
  names = name.split(".")
  lora_module_names.add(names[0] if len(names) == 1 else names[-1])

or:

if isinstance(module, cls) or "Linear" in module.__class__.__name__ and "Scaling" not in module.__class__.__name__:
  names = name.split(".")
  lora_module_names.add(names[0] if len(names) == 1 else names[-1])

[Napuh]

This works fine on Llama-2, Llama-2 based models and OpenLlama3B.

[NanoCode012]

Thanks @Napuh

[NanoCode012]

I just realized a code smell. I should've wrapped the and condition separately from or.