Generic model export pipeline + torch to ONNX and ONNX to deepsparse refactor #1192

dbogunowicz · 2022-12-01T10:55:25Z

Summary

This PR reworks all of the onnx optimization passes contained in src/sparseml/pytorch/sparsification/quantization/quantize_qat_export.py.

The new classes/functions are introduced:

Transforms: class BaseTransform and class OnnxTransform(BaseTransform), which represent abstract transforms that act on onnx graphs
Exporters: class BaseExporter(BaseTransform), class TorchToONNX(BaseExporter), and class ONNXToDeepsparse(BaseExporter), which represent transforms that can also save things to disk

In addition, every transform from quantize_qat_export (and other files), has been added as a OnnxTransform. This includes:

The actual transform implementation
docstring of exactly what the transform does
a unit test for the transform

The get_structural_matches utility function was also added to aid in the implementation of these transforms. Notably it allows callees to match sub-graphs in any onnx graph, which makes implementation details more clear/explicit.

Test Plan

As mentioned in the summary, every transform has a unit test associated with it. Additionally, regression tests were added against the old module exporter logic for the following models:

Resnet50
Yolov5
Bert for QA
So both the graph structure and numerical output match exactly for both the new exporters and the old exporters.

* initial commit * PR comments * initial commit * Delete test_fold_identity_initializers.py * Delete __init__.py * Delete __init__.py * Update src/sparseml/exporters/transforms/base_transform.py * fix docstrings * few improvements and tests Co-authored-by: bogunowicz@arrival.com <bogunowicz@arrival.com>

* Adding onnx graph structural matching * Styling * Adding missing init.py * Update src/sparseml/exporters/transforms/utils/matching.py Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> * Updating docstring of structural_matches * Adding __all__ * Addressing review comments * Removing extra file from merge Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com>

* initial commit * PR comments * initial commit * Delete test_fold_identity_initializers.py * Delete __init__.py * Delete __init__.py * Adding onnx graph structural matching * Styling * Update src/sparseml/exporters/transforms/base_transform.py * fix docstrings * Adding missing init.py * Update src/sparseml/exporters/transforms/utils/matching.py Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> * Updating docstring of structural_matches * Adding __all__ * ready for review * Update src/sparseml/exporters/transforms/fold_identity_initializers.py Co-authored-by: corey-nm <109536191+corey-nm@users.noreply.github.com> * some nits according to Bens comments Co-authored-by: bogunowicz@arrival.com <bogunowicz@arrival.com> Co-authored-by: Corey Lowman <corey@neuralmagic.com> Co-authored-by: corey-nm <109536191+corey-nm@users.noreply.github.com>

* Adding InitializersToUint8 transform * Update src/sparseml/exporters/transforms/initializers_to_uint8.py

* Adding onnx graph structural matching * Styling * Adding missing init.py * Update src/sparseml/exporters/transforms/utils/matching.py Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> * Updating docstring of structural_matches * Adding __all__ * initial commit * Update convert_quantizable_conv_integer.py Co-authored-by: Corey Lowman <corey@neuralmagic.com> Co-authored-by: corey-nm <109536191+corey-nm@users.noreply.github.com> Co-authored-by: bogunowicz@arrival.com <bogunowicz@arrival.com>

* Adding onnx graph structural matching * Styling * Adding missing init.py * Update src/sparseml/exporters/transforms/utils/matching.py Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> * Updating docstring of structural_matches * Adding __all__ * initial commit * Delete base_exporter.py * Update src/sparseml/exporters/transforms/convert_quantizable_matmul.py * beautify * check for initializers * add docstring Co-authored-by: Corey Lowman <corey@neuralmagic.com> Co-authored-by: corey-nm <109536191+corey-nm@users.noreply.github.com> Co-authored-by: bogunowicz@arrival.com <bogunowicz@arrival.com>

* Adding ConvToQLinearConv transform * Responding to review comments * Respond to reviews

* Adding FlattenQParams transform * Respond to review

* Adding ConvToQLinearConv transform * Responding to review comments * Adding GemmToQLinearMatMul * Styling

* initial commit * intiial commit * PR comments * fix errors * Apply suggestions from code review * upadte heleprs * matching of conv integer pass * second implementation done, needs some polishing * Adding match_structure and iter_structural_matches * Using structural matching for quantizable_conv_integer * initial commit * Adding onnx graph structural matching * Styling * Adding missing init.py * Update src/sparseml/exporters/transforms/utils/matching.py Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> * Updating docstring of structural_matches * Adding __all__ * initial commit * ready for PR * beautify * Delete test_helpers.py * Delete base_exporter.py Co-authored-by: bogunowicz@arrival.com <bogunowicz@arrival.com> Co-authored-by: Corey Lowman <corey@neuralmagic.com> Co-authored-by: corey-nm <109536191+corey-nm@users.noreply.github.com>

* Adding FoldConvDivBn * Expanding docstring

* initial commit * get transform into the correct format * ready for review * fix naming in test * Fixing trivial onnx adds Co-authored-by: bogunowicz@arrival.com <bogunowicz@arrival.com> Co-authored-by: corey-nm <109536191+corey-nm@users.noreply.github.com> Co-authored-by: Corey Lowman <corey@neuralmagic.com>

* Adding QuantizeResiduals transform * Adding tests

…1255)

* Adding DeleteRepeatedQdq transform * Adding unit test for delete repeated qdq * Using assert_node_type * Update src/sparseml/exporters/transforms/delete_repeated_qdq.py

* Adding SkipInputQuantize transform * add tests Co-authored-by: bogunowicz@arrival.com <bogunowicz@arrival.com>

* Fixing matching logic of qlinear transforms * Adding folding of input/output quants to qlinears

…_pipeline_refactor

…1249) * Initial comit of exporters * Styling * Fixing SkipInputQuantize * Adding validation methods * Clean up ONNXToDeepsparse * Moving TorchToONNX to pytorch * Adding inplace and saving pre optimized model to ONNXToDeepsparse * Adding sketch of tests * Regression tests against simple models * resnet50 regression tests passing * resnet50 exporters are all equivalent * Moving FoldConvDivBn under initializer folding * Adding yolov5 tests * Apply suggestions from code review Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> * Review response * Adding notes from review * uncomment asserts... oops * yolo & resnet tests passing Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com>

…_pipeline_refactor

…tchResult to str (#1262) * Addin any_of and MatchResult to str * Fixing docstring of get_structural_matches

…1263)

* Standardization of some transforms * Adding logging methods to OnnxTransform class

* Standardizing transforms with node removals * Using log_match

* Standardizing MatMulToQLinearMatMul * Using log_match

* Standardizing ConvToConvIntegerAddCastMul * Using log_match

* Standardizing qlinear transforms * Using log_match

#1269) * Standardizing MatMulIntegerAddCastMul transforms * Using log_match and any_of

* Standardizing QuantizeQATEmbedding * Add log_match

* initial commit * Apply suggestions from code review * Update tests/sparseml/pytorch/test_torch_to_onnx_exporter.py * Fixing bert exporters Co-authored-by: bogunowicz@arrival.com <bogunowicz@arrival.com> Co-authored-by: Corey Lowman <corey@neuralmagic.com>

* initial commit * PR edits * Delete recipe.yaml * fix onnx problem * Fixing torch import issue and numpy attr error * Another attempt at fixing get_numpy_dtype * Fix numpy.float usage Co-authored-by: bogunowicz@arrival.com <bogunowicz@arrival.com> Co-authored-by: Corey Lowman <corey@neuralmagic.com>

bfineran

🚀

KSGulin

Super clean implementation!

dbogunowicz · 2022-12-21T05:55:24Z

@corey-nm great job! I like the PR message.
Can we override the tests and land it now?

corey-nm · 2022-12-21T14:54:14Z

I think @bfineran would have to force merge it

bogunowicz@arrival.com and others added 13 commits December 1, 2022 11:54

initial commit

c8da811

Merge branch 'main' into feature/damian/export_pipeline_refactor

5c4eff1

Fixing match initializer logic

c71ec86

Adding ConstantsToInitializers pass (#1227)

ab0b64e

Adding UnwrapBatchNorms transform (#1230)

7066e9c

[Export Refactor] Adding InitializersToUint8 transform (#1228)

e0a3c28

* Adding InitializersToUint8 transform * Update src/sparseml/exporters/transforms/initializers_to_uint8.py

Merge branch 'main' into feature/damian/export_pipeline_refactor

be819ea

fix quality

bdfc800

corey-nm added the mle-team label Dec 12, 2022

corey-nm and others added 16 commits December 12, 2022 15:40

[Export Refactor] Adding ConvToQLinearConv transform (#1221)

1a61014

* Adding ConvToQLinearConv transform * Responding to review comments * Respond to reviews

[Export Refactor] Adding FlattenQParams transform (#1229)

f63110a

* Adding FlattenQParams transform * Respond to review

Adding GemmToMatMulIntegerAddCastMul trasnform (#1237)

0850edc

Adding MatMulToMatMulIntegerAddCastMul transform (#1238)

e263cfe

Adding FoldReLUQuants transform (#1240)

aa67daf

Adding PropagateEmbeddingQuantization transform (#1242)

206efcb

Adding RemoveDuplicateQuantizeOps transform (#1243)

2ce8bbf

[Export Refactor]Adding GemmToQLinearMatMul transform (#1225)

3218c4e

* Adding ConvToQLinearConv transform * Responding to review comments * Adding GemmToQLinearMatMul * Styling

[Exporter Refactor] Adding FoldConvDivBn (#1235)

612a749

* Adding FoldConvDivBn * Expanding docstring

Adding RemoveDuplicateQConvWeights transform (#1244)

8741b61

[Exporter Refactor] Adding QuantizeResiduals transform (#1245)

acfd115

* Adding QuantizeResiduals transform * Adding tests

Styling

c5c09d1

Fixing conv-integer transform and add sorting to core ops (#1252)

199f04a

Don't print out onnx model on validation error (#1253)

cfcddd9

corey-nm and others added 22 commits December 15, 2022 08:58

FoldReluQuants now modifies all children of relu node (#1254)

051c247

Adding shape check for weight comparison in duplicate-qconv-weights (#…

1c6722f

…1255)

[Exporter Refactor] Adding DeleteRepeatedQdq transform (#1257)

ff39921

* Adding DeleteRepeatedQdq transform * Adding unit test for delete repeated qdq * Using assert_node_type * Update src/sparseml/exporters/transforms/delete_repeated_qdq.py

[Exporter Refactor] Adding SkipInputQuantize transform (#1256)

89a5516

* Adding SkipInputQuantize transform * add tests Co-authored-by: bogunowicz@arrival.com <bogunowicz@arrival.com>

[Exporter Refactor] Fixing matching logic of qlinear transforms (#1251)

38c1531

* Fixing matching logic of qlinear transforms * Adding folding of input/output quants to qlinears

Merge remote-tracking branch 'origin/main' into feature/damian/export…

98cf64e

…_pipeline_refactor

Merge remote-tracking branch 'origin/main' into feature/damian/export…

cc02edb

…_pipeline_refactor

[Exporter Refactor] Adding any_of for get_structural_matches and Ma…

1e4b4d2

…tchResult to str (#1262) * Addin any_of and MatchResult to str * Fixing docstring of get_structural_matches

Adding add_node_deferred and delete_node_deffered to OnnxTransform (#…

ed35f07

…1263)

[Exporter Refactor] Standardize trivial transforms (#1264)

a8c40db

* Standardization of some transforms * Adding logging methods to OnnxTransform class

[Exporter Refactor] Standardize non core transforms (#1265)

452eb83

* Standardizing transforms with node removals * Using log_match

[Exporter Refactor] Standardizing MatMulToQLinearMatMul (#1266)

1a1a9e9

* Standardizing MatMulToQLinearMatMul * Using log_match

[Exporter Refactor] Standardizing ConvToConvIntegerAddCastMul (#1267)

205b4c4

* Standardizing ConvToConvIntegerAddCastMul * Using log_match

[Exporter Refactor] Standardize qlinears (#1268)

ab083ca

* Standardizing qlinear transforms * Using log_match

[ExporterRefactor] Standardizing XToMatMulIntegerAddCastMul transforms (

bf97b6c

#1269) * Standardizing MatMulIntegerAddCastMul transforms * Using log_match and any_of

[Exporter Refactor] Standardize qat embedding (#1270)

a762fdf

* Standardizing QuantizeQATEmbedding * Add log_match

Using renamed versions of transforms (#1271)

bf5cc22

Removing unused tests

7350498

Merge branch 'main' into feature/damian/export_pipeline_refactor

0ab1da9

bfineran changed the title ~~[Exporter Refactor] Feature Branch~~ Generic model export pipeline + torch to ONNX and ONNX to deepsparse refactor Dec 20, 2022

bfineran approved these changes Dec 20, 2022

View reviewed changes

KSGulin approved these changes Dec 20, 2022

View reviewed changes

Merge branch 'main' into feature/damian/export_pipeline_refactor

5de842a

bfineran merged commit ec37d3e into main Dec 29, 2022

bfineran deleted the feature/damian/export_pipeline_refactor branch December 29, 2022 15:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generic model export pipeline + torch to ONNX and ONNX to deepsparse refactor #1192

Generic model export pipeline + torch to ONNX and ONNX to deepsparse refactor #1192

dbogunowicz commented Dec 1, 2022 •

edited by corey-nm

Loading

bfineran left a comment

KSGulin left a comment

dbogunowicz commented Dec 21, 2022

corey-nm commented Dec 21, 2022

Generic model export pipeline + torch to ONNX and ONNX to deepsparse refactor #1192

Generic model export pipeline + torch to ONNX and ONNX to deepsparse refactor #1192

Conversation

dbogunowicz commented Dec 1, 2022 • edited by corey-nm Loading

Summary

Test Plan

bfineran left a comment

Choose a reason for hiding this comment

KSGulin left a comment

Choose a reason for hiding this comment

dbogunowicz commented Dec 21, 2022

corey-nm commented Dec 21, 2022

dbogunowicz commented Dec 1, 2022 •

edited by corey-nm

Loading