Modifier Refactor OBCQ Implementation #1737

Satrat · 2023-09-26T20:11:52Z

Summary

Implementation of OBCQ in the new modifiers framework, tested for OPT and Llama so far. This shouldn't be merged to main until #1713 is merged in. Main changes are in src/sparseml/modifiers/obcq

A new modifier is added for the SparseGPT algorithm, building off the code in src/sparseml/experimental/sparsegpt.
Added quantization modifier to the new modifier framework. Currently this modifier only supports one-shot
Added post one-shot calibration parameter for recalibrating quantization statistics after OBCQ
Dataset registry for c4,ptb,wikitext2,open_platypus

Instructions

To run for OPT:

python src/sparseml/transformers/sparsification/obcq/obcq.py facebook/opt-1.3b c4
    --recipe src/sparseml/transformers/sparsification/obcq/example.yaml

To run for Llama, you'll need a local copy of the model

python src/sparseml/transformers/sparsification/obcq/obcq.py /local/path/to/llama open_platypus
    --recipe src/sparseml/transformers/sparsification/obcq/example_llama.yaml

Testing

To compare the perplexity evaluation between the experimental implementation and productionized version of OPT run:

python src/sparseml/experimental/sparsegpt/examples/opt/compare_obcq.py

To compare perplexity for Llama run

python src/sparseml/experimental/sparsegpt/examples/llama2/compare_obcq.py

…arsegpt/llama2 # Conflicts: # src/sparseml/experimental/sparsegpt/dispatch.py # src/sparseml/experimental/sparsegpt/layer_compressor.py # src/sparseml/experimental/sparsegpt/main.py # src/sparseml/experimental/sparsegpt/model_preprocessor.py # src/sparseml/experimental/sparsegpt/opt.py # src/sparseml/experimental/sparsegpt/sequential.py # src/sparseml/experimental/sparsegpt/sparsegpt.py

src/sparseml/modifiers/obcq/base.py

src/sparseml/modifiers/obcq/pytorch.py

src/sparseml/modifiers/obcq/utils/data.py

src/sparseml/modifiers/obcq/utils/models.py

src/sparseml/modifiers/obcq/utils/sparsegpt.py

src/sparseml/modifiers/quantization/__init__.py

… into refactor_obcq

bfineran

latest commits look great. good to go from my end

src/sparseml/modifiers/obcq/base.py

src/sparseml/transformers/data/base_llm.py

Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>

natuan and others added 30 commits August 17, 2023 23:28

First abstract structure with OPT, MPT

bd729f3

Merge branch 'main' into tuan/sparsegpt

0476a14

scaffolding

e9c4e95

First abstract structure with OPT, MPT

07a7273

Updated

73ccd2e

Updated

f742f4a

Initial implementation of Llama2 integration

7c2d3d4

Initial implementation of Llama2 integration

3f4c5d8

Fix llama-2 key

c181380

Add modelutils

dd0f5f8

Make smoothquant optional

99f8bbe

probe sequence length from model

db901e4

probe sequence length from model

48ebe81

Fix typo

2be7def

add in recipe and modifier logic

fb346ad

Catch additional arguments to load_model and load_data

c4fc152

Redifinition of seqlen

2b50607

Runable OPT main code path

15b2c10

mpt modifier

34c3d84

Merge branch 'tuan/sparsegpt' into sa/sgpt_modifier

4cac803

Initial start implementation

a3934c2

State before rebasing

6a35b7f

State before rebasing

e02abb2

Initial support for MPT

466213c

add in further completion state for session and events

d953be4

Initial commit for Llama2

7fc49ac

Return model after override Llama attn

ff5b9a1

working refactor

090af1c

More memory-efficient layer compression

594339f

Satrat requested review from dsikka and rahul-tuli October 4, 2023 14:10

fix typos

ac29d68

Base automatically changed from sparsification-refactor to main October 4, 2023 19:08

bfineran requested changes Oct 4, 2023

View reviewed changes

Satrat added 11 commits October 5, 2023 09:44

PR comments

7617f9c

small bug fix on logger

f216d53

Merge branch 'main' into refactor_obcq

dcbbbd4

fixing transformer dependency, adding registry for dataset loaders

de3dc2f

fixing bugs in dataset loading and perplexity

39ff676

cleanup

7423f20

Merge branch 'main' into refactor_obcq

25ba312

fix memory issue in comparison

13c3a6c

return perplexity

b5e9d6d

adding split to all datasets

1decfe5

Merge branch 'refactor_obcq' of https://github.com/neuralmagic/sparseml…

0a0cff9

… into refactor_obcq

Satrat requested a review from bfineran October 6, 2023 19:32

bfineran previously approved these changes Oct 6, 2023

View reviewed changes

src/sparseml/modifiers/obcq/base.py Outdated Show resolved Hide resolved

src/sparseml/transformers/data/base_llm.py Show resolved Hide resolved

Update src/sparseml/modifiers/obcq/base.py

5fa7361

Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>

Satrat dismissed bfineran’s stale review via 5fa7361 October 9, 2023 14:26

bfineran previously approved these changes Oct 9, 2023

View reviewed changes

Satrat added 2 commits October 9, 2023 10:32

Merge branch 'main' into refactor_obcq

f1441cb

fixing dataset issues

29271a3

Satrat dismissed bfineran’s stale review via 29271a3 October 10, 2023 23:12

Satrat mentioned this pull request Oct 11, 2023

SmoothQuant Modifier for OBCQ #1758

Merged

Satrat requested review from natuan and bfineran October 11, 2023 14:35

bfineran approved these changes Oct 11, 2023

View reviewed changes

bfineran merged commit 6c3f054 into main Oct 11, 2023
11 checks passed

bfineran deleted the refactor_obcq branch October 11, 2023 17:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modifier Refactor OBCQ Implementation #1737

Modifier Refactor OBCQ Implementation #1737

Satrat commented Sep 26, 2023 •

edited

Loading

bfineran left a comment

Modifier Refactor OBCQ Implementation #1737

Modifier Refactor OBCQ Implementation #1737

Conversation

Satrat commented Sep 26, 2023 • edited Loading

Summary

Instructions

Testing

bfineran left a comment

Choose a reason for hiding this comment

Satrat commented Sep 26, 2023 •

edited

Loading