Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Layerwise Sparsity Support for SparseGPT #1777

Merged
merged 12 commits into from
Oct 26, 2023
Merged

Layerwise Sparsity Support for SparseGPT #1777

merged 12 commits into from
Oct 26, 2023

Conversation

Satrat
Copy link
Contributor

@Satrat Satrat commented Oct 19, 2023 •
edited
Loading

Support for providing a list of sparsities to the SparseGPTModifier to enable layer-wise sparsity configuration. The length must match up with the number of target layers to compress, otherwise an error is thrown.

Testing

Example Recipe

test_stage:
  obcq_modifiers:
    SparseGPTModifier:
      sparsity: [
        0.1,
        0.2,
        0.3,
        0.4,
        0.5
      ]
      block_size: 128
      sequential_update: False
      quantize: False
      percdamp: 0.01
      prunen: 0
      prunem: 0
      compress_layers: [
        "model.decoder.layers.0",
        "model.decoder.layers.1",
        "model.decoder.layers.2",
        "model.decoder.layers.3",
        "model.decoder.layers.4"
      ]
      target_ids: ["attention_mask"]
      layer_prefix: "decoder"

To Run

python src/sparseml/transformers/sparsification/obcq/obcq.py facebook/opt-1.3b c4 --recipe example_recipe.yaml --eval wikitext2

Output

2023-10-19 09:56:37 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 1/5 to sparsity 0.1 =====
2023-10-19 09:56:37 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj...
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.47
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 201.29
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj...
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.37
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 48.39
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj...
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 210.59
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj...
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 0.02
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1...
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 189.64
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2...
2023-10-19 09:56:41 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 1.58
2023-10-19 09:56:41 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 1.16
2023-10-19 09:56:41 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 2/5 to sparsity 0.2 =====
2023-10-19 09:56:42 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj...
2023-10-19 09:56:42 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.39
2023-10-19 09:56:42 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 444.99
2023-10-19 09:56:42 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj...
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.37
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 87.64
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj...
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 348.68
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj...
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 0.02
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1...
2023-10-19 09:56:44 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:44 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 1242.68
2023-10-19 09:56:44 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2...
2023-10-19 09:56:45 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 1.57
2023-10-19 09:56:45 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 0.08
2023-10-19 09:56:46 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 3/5 to sparsity 0.3 =====
2023-10-19 09:56:46 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj...
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.39
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 1737.14
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj...
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 567.80
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj...
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 1402.33
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj...
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 2.21
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1...
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 5431.26
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2...
2023-10-19 09:56:50 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 1.58
2023-10-19 09:56:50 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 6.83
2023-10-19 09:56:50 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 4/5 to sparsity 0.4 =====
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj...
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.39
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 5309.78
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj...
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.37
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 2328.10
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj...
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 4082.74
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj...
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 12.44
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1...
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 16928.26
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2...
2023-10-19 09:56:54 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 1.58
2023-10-19 09:56:54 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 26.82
2023-10-19 09:56:54 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 5/5 to sparsity 0.5 =====
2023-10-19 09:56:55 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj...
2023-10-19 09:56:55 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.39
2023-10-19 09:56:55 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 13890.12
2023-10-19 09:56:55 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj...
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.37
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 6673.23
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj...
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 12242.67
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj...
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 44.67
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1...
2023-10-19 09:56:57 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:57 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 45602.01
2023-10-19 09:56:57 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2...
2023-10-19 09:56:58 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 1.58
2023-10-19 09:56:58 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 85.04
Found cached dataset wikitext (/home/sadkins/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
2023-10-19 09:57:00 sparseml.modifiers.obcq.utils.helpers INFO     Evaluating perplexity...
2023-10-19 09:58:02 sparseml.modifiers.obcq.utils.helpers INFO     Perplexity: 14.667972

rahul-tuli
rahul-tuli reviewed Oct 19, 2023
View reviewed changes
Copy link
Member

@rahul-tuli rahul-tuli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending comment!

src/sparseml/modifiers/obcq/base.py Outdated Show resolved Hide resolved
bfineran
bfineran previously approved these changes Oct 20, 2023
View reviewed changes
src/sparseml/modifiers/obcq/base.py Show resolved Hide resolved
rahul-tuli
rahul-tuli previously approved these changes Oct 26, 2023
View reviewed changes
@Satrat Satrat dismissed stale reviews from rahul-tuli and bfineran via 1c30bd8 October 26, 2023 21:09
@bfineran bfineran merged commit e20927f into main Oct 26, 2023
10 of 11 checks passed
@bfineran bfineran deleted the layer-wise-sparsity branch October 26, 2023 22:31
bfineran pushed a commit that referenced this pull request Nov 16, 2023
@Satrat @bfineran
Layerwise Sparsity Support for SparseGPT (#1777)
f6ab063
bfineran pushed a commit that referenced this pull request Nov 16, 2023
@Satrat @bfineran
Layerwise Sparsity Support for SparseGPT (#1777)
1f66cc9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bfineran bfineran bfineran left review comments

@rahul-tuli rahul-tuli rahul-tuli left review comments

@dsikka dsikka Awaiting requested review from dsikka

@dbogunowicz dbogunowicz Awaiting requested review from dbogunowicz

@horheynm horheynm Awaiting requested review from horheynm

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@Satrat @bfineran @rahul-tuli @horheynm