Add GPTQ Quantization

[SunMarc]

What does this PR do?

This PR adds the possibility to perform GTPQ quantization. I tried to be as generic as possible to support any kind of models ( whether it is Transformer models or not). The backend relies on auto_gptq library where we use GTPQ class and QuantLinear class. With this API you can do conversion, saving and loading quantized weights. We have a dependence on accelerate to save the weights and load the quantized weights efficiently without allocating more memory than needed.

Quantize model:

model_name = "bigscience/bloom-1b7"
from transformers import AutoModelForCausalLM, AutoTokenizer
from optimum.gptq import GPTQQuantizer, load_quantized_model
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype = torch.float16, device_map = "auto")
quantizer = GPTQQuantizer(bits=4, dataset="c4", group_size=128, desc_act=False)
quantized_model = quantizer.quantize_model(quantized_model, tokenizer)

Save model

save_folder = 'bloom-1b7-quantized_optimum'
quantizer.save(model, save_folder)

Convert model and Load quantized weights

from accelerate import init_empty_weights
with init_empty_weights():
    empty_model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16)
empty_model.tie_weights()
model_from_saved = load_quantized_model(empty_model, save_folder=save_folder, device_map="auto")

PS: In a follow-up PR, I will use this API to integrate GTPQ quantization into transformers, so that we take into account GPTQ quantized model, just like what we did for bitsandbytes

TODO:

• Doc
• CI
• cpu offload (for now, if some modules are offloaded to the cpu (using device_map), the quantization will not work )
• exllama integration
• script to calculate the perplexity, maybe in a new PR ...

[younesbelkada]

Thanks a lot for this work and integration! Left some initial comments before the review of @fxmarty with respect to potential transformers integration later on

[mfuntowicz]

Do we actually need a GPU to quantize the model? I don't have the details right out of my head now, but is it a strong requirement?

[mfuntowicz]

(Not considering the actual inference process which I understand requires a GPU, maybe quantization/calibration can be done on CPU too?)

[SunMarc]

I don't think it is a strong requirement. I need to test but you can have a look at fasterquant method from GPTQ class. Looks like it uses cuda to speed up some operations (cholesky, cholesky_inverse, matmul) and I can see that he calls torch.cuda.synchronize().

[fxmarty]

It is in good shape, thank you for working on it!

I still think that using exllama kernel would be very interesting (for ref https://github.com/fxmarty/q4f16-gemm-gemv-benchmark & huggingface/text-generation-inference#553 (comment), second one is with Triton, that I see is not used here), but integrating with AutoGPTQ is already a nice step (making sure that cuda-old is used when possible).

Is there a hard requirement on accelerate? If so, why?

I think it would be good to add a documentation as well in this PR, and CI.

It is also not clear to me what should go in accelerate and optimum, if this PR puts a hard requirement on accelerate. Why not put it in accelerate directly? Why bitsandbytes in accelerate & this in optimum? I think it can be quite confusing to users.

[younesbelkada]

Amazing work @SunMarc ! All this looks great! Left few nits but the overall structure looks really nice. Thanks a lot for working on this!
Let's wait for @fxmarty's review for merging this PR :D

[fxmarty]

Looks great! I left a few questions/style comments

[fxmarty]

LGTM, thank you for adding it!

Just left taste comments / answers to the threads above. I believe this breaks with transformers 4.31 now, but maybe that's fine

[fxmarty]

Can you run make style?

[SunMarc]

I believe this breaks with transformers 4.31 now, but maybe that's fine

I've changed it so that we install from source. I will change it back after the release of transformers.

[regisss]

Very nice PR @SunMarc 🔥 🚀
Just commenting the title of the new doc section.
Also, could you add a bullet point for GPTQ in this section please?