support to disable exllama for gptq

[winglian]

fixes #599

adding gptq_disable_exllama: true to the yml config should fix the issue with gptq

[NanoCode012]

The author of the linked issue mentions

    raise ValueError('Expected a cuda device, but got: {}'.format(device))
ValueError: Expected a cuda device, but got: cpu

after setting it. Is that due to cuda oom?

[Napuh]

Executing the example yaml file in this branch thows an error related to modifying the LlamaConfig object:

  File "/home/axolotl/scripts/finetune.py", line 52, in <module>
    fire.Fire(do_cli)
  File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/axolotl/scripts/finetune.py", line 48, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/axolotl/src/axolotl/train.py", line 58, in train
    model, peft_config = load_model(cfg, tokenizer, inference=cli_args.inference)
  File "/home/axolotl/src/axolotl/utils/models.py", line 202, in load_model
    model_config["disable_exllama"] = cfg.gptq_disable_exllama
TypeError: 'LlamaConfig' object does not support item assignment
Traceback (most recent call last):
  File "/opt/conda/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 986, in launch_command
    simple_launcher(args)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 628, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', 'scripts/finetune.py', 'examples/llama-2/gptq-lora.yml']' returned non-zero exit status 1.

Also:

The author of the linked issue mentions

    raise ValueError('Expected a cuda device, but got: {}'.format(device))
ValueError: Expected a cuda device, but got: cpu

after setting it. Is that due to cuda oom?

I doubt is a CUDA memory issue, I was executing on a RTX3090 24GB. Wasn't a fluke either, as the error persisted in two different machines.

[winglian]

@Napuh I updated the fix, lmk if that works.

[Napuh]

Now it's throwing the same error as in #599, ValueError: Found modules on cpu/disk. Using Exllama backend requires all the modules to be on GPU.You can deactivate exllama backend by setting `disable_exllama=True` in the quantization config object.

Im executing accelerate launch scripts/finetune.py examples/llama-2/gptq-lora.yml, am I doing something wrong?

Also script keeps loading after model downloading for about 5 minutes, and no memory is ever allocated on the gpu (monitored manually via nvidia-smi).

[winglian]

@Napuh hopefully this most recent commit resolves it.

[Napuh]

@winglian Latest commit solves the config issue with exllama, but now it's throwing ValueError: Expected a cuda device, but got: cpu, the same error as if I edit the config.json manually.

#456 may be related as it's the same error, but the scenario is different from this one.

[winglian]

@winglian Latest commit solves the config issue with exllama, but now it's throwing ValueError: Expected a cuda device, but got: cpu, the same error as if I edit the config.json manually.

#456 may be related as it's the same error, but the scenario is different from this one.

#609 should fix the issue for the device check when logging gpu utilization