Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Export Refactor] Feature Branch #1858

Merged
merged 21 commits into from
Jan 10, 2024
Merged

[Export Refactor] Feature Branch #1858

merged 21 commits into from
Jan 10, 2024

Conversation

dbogunowicz
Copy link
Contributor

@dbogunowicz dbogunowicz commented Nov 28, 2023 •
edited
Loading

Feature Description

Feature branch that implements new exporting framework. This PR roughly outlines the new design as outlined in the diagram below:
image

Features

The first order of business is to migrate Image Classification export to use the new framework. The related PRs are:

The second work item is to implement Transformers export:

After the aforementioned smaller PRs land, the next order of affairs would be to:

  • Add some basic documentation
  • Remove the part of the codebase responsible for legacy export and swap the outward-facing pathways to use the new export
  • Fix any pending tests

Also add few missing, potentially slightly orthogonal features:

  • Make sure that all create_model functions properly create sparse models (use @Satrat 's new modifiers), also apply one shot etc.
  • Validate validate_correctness function (some sparsezoo edits are required)
  • Properly implement and validate separating the exported onnx model into multiple files
  • Use device argument when creating a model
  • (Potentially) set is_tar as an export argument (to decide later)

CLI Example

sparseml.export PATH/TO/SPARSIFIED/LLM \
  --task text-generation

bfineran
bfineran reviewed Nov 29, 2023
View reviewed changes
src/sparseml/export.py Outdated Show resolved Hide resolved
src/sparseml/export.py Outdated Show resolved Hide resolved
src/sparseml/export.py Outdated Show resolved Hide resolved
src/sparseml/export.py Outdated Show resolved Hide resolved
dbogunowicz and others added 13 commits December 29, 2023 14:51
@dbogunowicz
initial commit
5058120
@dbogunowicz
respond to PR comments
bb5887e
@dbogunowicz
[Export Refactor][Image Classification] create_model function (#1878)
2930dea 
* initial commit

* looking good, time to cleanup

* Delete src/sparseml/export/helpers.py

* Delete tests/sparseml/export/test_helpers.py

* ready for review

* improve design

* tests pass

* reuse _validate_dataset_num_classes
@dbogunowicz
[Export Refactor][Image Classification] create_dummy_input function (…
5ac4d35 
…#1880)

* initial commit

* looking good, time to cleanup

* Delete src/sparseml/export/helpers.py

* Delete tests/sparseml/export/test_helpers.py

* ready for review

* improve design

* tests pass

* reuse _validate_dataset_num_classes

* initial commit

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* ready for review

* Update src/sparseml/export/export.py

* Update src/sparseml/integration_helper_functions.py
@dbogunowicz
[Export Refactor][Image Classification] export_model function (#1883)
59f3f5a 
* initial commit

* looking good, time to cleanup

* Delete src/sparseml/export/helpers.py

* Delete tests/sparseml/export/test_helpers.py

* ready for review

* improve design

* tests pass

* reuse _validate_dataset_num_classes

* initial commit

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* ready for review

* Update src/sparseml/export/export.py

* Update src/sparseml/integration_helper_functions.py

* initial commit

* fixes

* ready for review

* nit

* add return

* make export function more general
@dbogunowicz
[Export Refactor][Image Classification] apply_optimizations function (
9096b0d 
#1884)

* initial commit

* looking good, time to cleanup

* Delete src/sparseml/export/helpers.py

* Delete tests/sparseml/export/test_helpers.py

* ready for review

* improve design

* tests pass

* reuse _validate_dataset_num_classes

* initial commit

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* ready for review

* Update src/sparseml/export/export.py

* Update src/sparseml/integration_helper_functions.py

* initial commit

* fixes

* ready for review

* nit

* add return

* initial commit
@dbogunowicz
[Export Refactor][Image Classification] `export_sample_inputs_outputs…
16c9bf3 
…` function (#1888)

* initial commit

* looking good, time to cleanup

* Delete src/sparseml/export/helpers.py

* Delete tests/sparseml/export/test_helpers.py

* ready for review

* improve design

* tests pass

* reuse _validate_dataset_num_classes

* initial commit

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* ready for review

* Update src/sparseml/export/export.py

* Update src/sparseml/integration_helper_functions.py

* initial commit

* fixes

* ready for review

* nit

* add return

* initial commit

* initial commit

* PR comments

* beautification
@dbogunowicz
remove duplicated function
4d66402
@dbogunowicz
[Export Refactor][Image Classification] create_deployment_folder fu…
ed04d3f 
…nction (#1889)

* initial commit

* looking good, time to cleanup

* Delete src/sparseml/export/helpers.py

* Delete tests/sparseml/export/test_helpers.py

* ready for review

* improve design

* tests pass

* reuse _validate_dataset_num_classes

* initial commit

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* ready for review

* Update src/sparseml/export/export.py

* Update src/sparseml/integration_helper_functions.py

* initial commit

* fixes

* ready for review

* nit

* add return

* initial commit

* initial commit

* initial commit

* fix rebase, tests_work

* ready to push
@dbogunowicz
[Export Refactor][Image Classification] validate_correctness functi…
571aed8 
…on (#1890)

* initial commit

* looking good, time to cleanup

* Delete src/sparseml/export/helpers.py

* Delete tests/sparseml/export/test_helpers.py

* ready for review

* improve design

* tests pass

* reuse _validate_dataset_num_classes

* initial commit

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* ready for review

* Update src/sparseml/export/export.py

* Update src/sparseml/integration_helper_functions.py

* initial commit

* fixes

* ready for review

* nit

* add return

* initial commit

* initial commit

* initial commit

* initial commit

* Delete tests/sparseml/test_integration_helper_functions.py

* ready to merge
@dbogunowicz
[Export Refactor] End to end testing (#1898)
627ddd6 
* initial commit

* looking good, time to cleanup

* Delete src/sparseml/export/helpers.py

* Delete tests/sparseml/export/test_helpers.py

* ready for review

* improve design

* tests pass

* reuse _validate_dataset_num_classes

* initial commit

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* ready for review

* Update src/sparseml/export/export.py

* Update src/sparseml/integration_helper_functions.py

* initial commit

* fixes

* ready for review

* nit

* add return

* initial commit

* initial commit

* initial commit

* initial commit

* Delete tests/sparseml/test_integration_helper_functions.py

* ready to merge

* add structure validator

* ready for review

* Delete tests/sparseml/export/model.onnx

* Delete tests/sparseml/export/image_classification/model.onnx

* Delete tests/sparseml/export/image_classification/conftest.py

* PR comments

* remove onnx
@dbogunowicz @bfineran
[Export Refactor] Prepare the module to be more general (before inclu…
3da5f23 
…ding `transformers`) (#1908)

* adapt the export script to handle transformers

* Update src/sparseml/pytorch/image_classification/integration_helper_functions.py

* Delete tests/sparseml/export/transformers/__init__.py

* Delete tests/sparseml/export/transformers/test_generative_transformers.py

* Delete tests/sparseml/export/transformers/test_transformers.py

* Update src/sparseml/export/export.py

Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>

* addressing review comments

* [Export Refactor] Export `transformers` (#1909)

* cleanup

* Delete src/sparseml/transformers/integration_helper_functions_generative.py

* Delete src/sparseml/transformers/utils/optimizations.py

* Delete tests/sparseml/export/transformers/test_generative_transformers.py

* Delete tests/sparseml/transformers/test_integration_helper_functions_generative.py

* addressing PR reviews

* [Export Refactor] Export generative transformers(#1910)

* make tests green, remove using task to resolve the integration type

* fix all the tests after the merge, make integration resolution independent of the task name

* fold generative transformers into transformer helper functions

* complete tests for export_data.py

* Update src/sparseml/export/export.py

* add tests that confirms that kv cache injection has been added

* move applying optimizations into integration helper functions

---------

Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>
@dbogunowicz
[Export Refactor][Transformers] Enable loading SparseModels (#1921)
c65ab6e 
* initial commit

* adressing review comments
@dbogunowicz dbogunowicz force-pushed the feature/damian/feature_branch_export branch from 682fd9d to 3bd72bb Compare December 29, 2023 14:53
@dbogunowicz dbogunowicz force-pushed the feature/damian/feature_branch_export branch from 5d02f37 to 7b28881 Compare January 2, 2024 17:16
Satrat
Satrat reviewed Jan 2, 2024
View reviewed changes
Copy link
Contributor

@Satrat Satrat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest commits LGTM, just had some small comments. Lets also be sure to test this on llama 7b before landing to make sure we're getting the same perplexity results as before

src/sparseml/export/export.py Outdated Show resolved Hide resolved
src/sparseml/transformers/utils/initializers.py
)
return recipe

_LOGGER.info(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add another log statement on success that prints out the path to the recipe we are loading?

dbogunowicz and others added 3 commits January 5, 2024 21:15
@dbogunowicz
[Export][Transformers] Enable loading text-generation datasets (#1938)
6179cb2 
* add suport for past_key_values in sample-outputs

* [Export][Transformers] Implementation of correctness validation (#1935)

* fix tests with help from sara

* Update src/sparseml/transformers/utils/initializers.py

* swap sparsezoo validator for custom one (top k match)

* add more informative error message

* add correctness validation for LLMs

* remove past_key_values from outputs

* remove past_key_values from outputs (2)

* small note comment for the future
@dbogunowicz
tests fixed
7f166a1
@dbogunowicz
fix test
c3c90a4
Satrat
Satrat previously approved these changes Jan 10, 2024
View reviewed changes
Copy link
Contributor

@Satrat Satrat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just have one unresolved comment about adding an additional log message. Also looks like we need to resolve a merge conflict with the GHA file

@bfineran
[Export refactor] final manual testing fixes (#1948)
57a4dd0 
* [Export refactor] final manual testing fixes

* review
@bfineran bfineran merged commit 691cb49 into main Jan 10, 2024
12 of 13 checks passed
@bfineran bfineran deleted the feature/damian/feature_branch_export branch January 10, 2024 21:33
@dbarbuzzi dbarbuzzi mentioned this pull request Jan 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bfineran bfineran bfineran approved these changes

@Satrat Satrat Satrat left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@dbogunowicz @Satrat @bfineran