Batched quantization

[casper-hansen]

TODO:

• Add documentation with custom dataset, e.g. Cosmopedia filtering for long samples only.

Features:

• Split samples into batches of n_parallel_calib_samples
• Allow any sample length (long context)

Fixes:

• If you modify n_samples to be higher than 128, OOM is extremely likely because all samples are currently run through the model at the same time. Closes Expose calibration dataset arguments #517 Fixes bug with block_size and exposes n_samples and block_size to the user #493
• scales being NaN or inf in some cases. Resolves qwen2-72B can not be quantized by autoawq #498 when quantize qwen2 by autoawq, it not works successful. #500 Support Qwen2 72 Awq quantization？ #509.

[WallE-Chang]

OOM often happens here. Maybe you need move partial_output to cpu by module_output.append(partial_output.detach.cpu())

[RanchiZhao]

Mark here! looking forward to the new feature!