Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SmoothQuant Modifier for OBCQ #1758

Merged
merged 219 commits into from
Oct 31, 2023
Merged

SmoothQuant Modifier for OBCQ #1758

merged 219 commits into from
Oct 31, 2023

Conversation

Satrat
Copy link
Contributor

@Satrat Satrat commented Oct 11, 2023 •
edited
Loading

This PR implements a productionized SmoothQuantModifier for use in OBCQ, with example recipes for OPT and Llama.

Instructions and Testing

To run for OPT, and print out perplexity results

python src/sparseml/transformers/sparsification/obcq/obcq.py facebook/opt-1.3b c4
    --recipe src/sparseml/transformers/sparsification/obcq/example.yaml --eval wikitext

To run for Llama, you'll need a local copy of the model

python src/sparseml/transformers/sparsification/obcq/obcq.py /local/path/to/llama open_platypus
    --recipe src/sparseml/transformers/sparsification/obcq/example_llama.yaml --eval open_platypus

natuan and others added 30 commits August 17, 2023 23:28
@natuan
First abstract structure with OPT, MPT
bd729f3
@Satrat
Merge branch 'main' into tuan/sparsegpt
0476a14
@Satrat
scaffolding
e9c4e95
@natuan
First abstract structure with OPT, MPT
07a7273
@natuan
Updated
73ccd2e
@natuan
Updated
f742f4a
@anmarques
Initial implementation of Llama2 integration
7c2d3d4
@anmarques
Initial implementation of Llama2 integration
3f4c5d8
@anmarques
Fix llama-2 key
c181380
@anmarques
Add modelutils
dd0f5f8
@anmarques
Make smoothquant optional
99f8bbe
@anmarques
probe sequence length from model
db901e4
@anmarques
probe sequence length from model
48ebe81
@anmarques
Fix typo
2be7def
@Satrat
add in recipe and modifier logic
fb346ad
@anmarques
Catch additional arguments to load_model and load_data
c4fc152
@anmarques
Redifinition of seqlen
2b50607
@natuan
Runable OPT main code path
15b2c10
@Satrat
mpt modifier
34c3d84
@Satrat
Merge branch 'tuan/sparsegpt' into sa/sgpt_modifier
4cac803
@markurtz
Initial start implementation
a3934c2
@anmarques
State before rebasing
6a35b7f
@anmarques
State before rebasing
e02abb2
@anmarques
Merge remote-tracking branch 'origin/tuan/sparsegpt' into research/sp…
2fb9e66 
…arsegpt/llama2

# Conflicts:
#	src/sparseml/experimental/sparsegpt/dispatch.py
#	src/sparseml/experimental/sparsegpt/layer_compressor.py
#	src/sparseml/experimental/sparsegpt/main.py
#	src/sparseml/experimental/sparsegpt/model_preprocessor.py
#	src/sparseml/experimental/sparsegpt/opt.py
#	src/sparseml/experimental/sparsegpt/sequential.py
#	src/sparseml/experimental/sparsegpt/sparsegpt.py
@natuan
Initial support for MPT
466213c
@markurtz
add in further completion state for session and events
d953be4
@natuan
Initial commit for Llama2
7fc49ac
@natuan
Return model after override Llama attn
ff5b9a1
@Satrat
working refactor
090af1c
@natuan
More memory-efficient layer compression
594339f
bfineran
bfineran previously approved these changes Oct 16, 2023
View reviewed changes
@Satrat Satrat requested a review from anmarques October 16, 2023 20:56
@Satrat Satrat requested a review from bfineran October 19, 2023 15:45
anmarques
anmarques requested changes Oct 20, 2023
View reviewed changes
src/sparseml/modifiers/smoothquant/base.py Outdated Show resolved Hide resolved
src/sparseml/modifiers/smoothquant/pytorch.py Outdated Show resolved Hide resolved
rahul-tuli
rahul-tuli previously approved these changes Oct 20, 2023
View reviewed changes
src/sparseml/core/model/pytorch.py Outdated Show resolved Hide resolved
src/sparseml/modifiers/smoothquant/base.py Outdated Show resolved Hide resolved
src/sparseml/modifiers/smoothquant/base.py Outdated Show resolved Hide resolved
src/sparseml/modifiers/smoothquant/base.py Outdated Show resolved Hide resolved
src/sparseml/modifiers/smoothquant/base.py Outdated Show resolved Hide resolved
src/sparseml/modifiers/smoothquant/base.py Outdated Show resolved Hide resolved
src/sparseml/modifiers/smoothquant/base.py Show resolved Hide resolved
src/sparseml/modifiers/smoothquant/base.py Outdated Show resolved Hide resolved
src/sparseml/modifiers/smoothquant/pytorch.py Outdated Show resolved Hide resolved
anmarques
anmarques previously approved these changes Oct 31, 2023
View reviewed changes
@bfineran bfineran merged commit 90250f2 into main Oct 31, 2023
10 of 11 checks passed
@bfineran bfineran deleted the prod_smooth_quant branch October 31, 2023 12:21
bfineran pushed a commit that referenced this pull request Nov 16, 2023
@Satrat @bfineran
SmoothQuant Modifier for OBCQ (#1758)
312f2f0
bfineran pushed a commit that referenced this pull request Nov 16, 2023
@Satrat @bfineran
SmoothQuant Modifier for OBCQ (#1758)
63aee59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bfineran bfineran bfineran approved these changes

@anmarques anmarques anmarques left review comments

@natuan natuan Awaiting requested review from natuan

@dsikka dsikka Awaiting requested review from dsikka

@rahul-tuli rahul-tuli Awaiting requested review from rahul-tuli

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@Satrat @bfineran @anmarques @rahul-tuli @natuan @markurtz