SparseML v0.6.0

New Features:

• YOLOv5 sparsification tutorials and recipes added.
• YOLOv3 sparse transfer learning tutorial added.
• PyTorch image classification using recipes and recipes for ResNet-50 and MobileNet tutorial added.
• BERT additional recipes added for FP32, 3 and 6 layer sparse models.
• Support for phased pruning added: https://arxiv.org/pdf/2106.12379.pdf
• Research folder created for sparsifying passage retrieval.

Changes:

• README updated for Hugging Face transformers integration based on the new implementation.
• ONNX export in PyTorch now supports dictionary inputs.
• Quantized graph export optimizations for YOLOv5.
• PyTorch image classification integration updated to use new manager.modify(...) apis and saves recipes to runs folder.
• DeepSparse YOLO links updated to point at new example location.
• kwargs support added for ONNX export in PyTorch to enable dyanmic_axes and named inputs.

Resolved Issues:

• torch 1.8 quantization export no longer folds incorrectly.
• ONNX toposort issue addressed for nodes with more than two outputs.
• Unused initializers removed in quantized ONNX graphs.

Known Issues:

• None

SparseML v0.5.1 Patch Release

This is a patch release for 0.5.0 that contains the following changes:

• version updated for consistency to reflect DeepSparse repo hotfix.

SparseML v0.5.0

New Features:

• research folder added to root directory intended for research contributions.
• First research contributions added for information retrieval.
• Tutorial for sparsifying BERT on the SQuAD dataset created.
• LayerPruningModifier and LearningRateFunctionModifier implementations added for PyTorch.

Changes:

• Hugging Face transformers integration reworked to match new integration standards.
• CIFAR data augmentations updated for PyTorch datasets.
• Pruning algorithms using a pruning scorer object for better extensibility refactored with new pruning methods.

Resolved Issues:

• If the source URL is down, tests no longer fail for VOC dataset.
• Because the DeepSparse API includes more information for kernel sparsify performance analysis, previously failing tests have been updated to correctly check and return the updated info.
• Models with more than 1 input can now complete the PyTorch ONNX export process.
• Edge cases and better defaults improved with the WoodFisher/M-FAC algorithm for better recovery.
• Deprecated use of torch.nonzero API call in the pruning modifiers to .nonzero(as_tuple=False).

Known Issues:

• None

SparseML v0.4.0

New Features:

• M-FAC/Woodfisher pruning algorithm alpha implemented.
• Movement pruning algorithm alpha implemented.
• Distillation code added for GLUE dataset in Hugging Face/transformers integration.
• BERT quantization pipeline enabled for training and export to ONNX.

Changes:

• Readme redesigned for better clarity on the repository's purpose.
• All examples, notebooks, and scripts moved under the integrations directory.
• Integrations for ultralytics/yolov3, ultralytics/yolov5, pytorch, keras, tensorflow reworked to match new integrations standards for better ease of use.

Resolved Issues:

• rwightman/timm integration bugs dealing with checkpoint paths and loading models addressed.
• tensorflow-gpu for tensorflow v1 now recognized correctly.
• Neural Magic dependencies upgrade to intended bug versions instead of minor versions.

Known Issues:

• Movement pruning is currently only working with FP16 training on GPUs, FP32 is diverging to NaN.

SparseML v0.3.1 Patch Release

This is a patch release for 0.3.0 that contains the following changes:

• DeepSparse APIs now properly referencing VNNI check
• Block sparse masks now applied for pruning modifiers
• Some tests marked as flaky to make tests more consistent
• Docs updated for new Discourse and Slack links
• Modifier code refactored to better support Automatic Mixed Precision Training (AMP) in PyTorch
• Emulated_step added to manager for inconsistent steps_per_epoch in PyTorch
• Serialization of block sparse-enabled pruning modifiers no longer fail on reload

SparseML v0.3.0

New Features:

• YOLO integration with Ultralytics deployed including DeepSparse examples for benchmarking and running detection over videos.
• Framework and Sparsification Info APIs now available for all supported ML frameworks.
• Properties added to the ScheduledManager class to allow for lookup of contained modifiers such as pruning and quantization.
• ALL_PRUNABLE token added for pruning modifiers.
• PyTorch global magnitude pruning support implemented.
• QAT support added for BERT.

Changes:

• Version changed to be loaded from version.py file, default build on branches is now nightly.
• Additional unit tests added in for Keras integration.
• PyTorch max supported version updated to 1.7.
• Improved performance for parsing and fixing QAT ONNX graphs from PyTorch.

Resolved Issues:

• Docs typos and broken links addressed.
• Pickling models with PyTorch pruning hooks work as expected.
• Incorrect loss scaling for DDP in PyTorch vision.py script addressed.

Known Issues:

• None

SparseML v0.2.0

New Features:

• Keras sparsification beta supporting pruning and examples with Keras Applications available.
• Training and sparsification integrated with the rwightman/pytorch-image-models repository.
• Training, sparsification, and deployment integrated with the ultralytics/yolov5 repository.
• Integrations with the SparseZoo to run PyTorch and Keras code implemented with recipes directly from the zoo.
• PyTorch sparse-quantized transfer learning notebook available.
• Keras pruned ResNet models implemented.
• Groups of modifiers enabled in SparseML recipes.

Changes:

• Examples directory renamed to integrations.

Resolved Issues:

• GroupedPruningMaskCreator now able to save to PyTorch state dicts.
• Quantization-aware training compatibility issues addressed with PyTorch.
• Docs and readme corrections made for minor issues and broken links.
• Makefile no longer deletes files for docs compilation and cleaning.

Known Issues:

• None

SparseML v0.1.1 Patch Release

This is a patch release for 0.1.0 that contains the following changes:

• Docs updates: tagline, overview, update to use sparsification for verbiage, fix broken links for recipes
• Flaky decorator added to some sparsity tests so if they fail due to random chance will immediately retest
• Modifier groups enabled for recipes
• DeepSparse nightly build dependencies now match on major.minor and not full version
• Serialization support for MaskedLayer in Keras added
• Support implemented for loading recipes from SparseZoo to common scripts and APIs
• Examples directory renamed to integrations to clarify function
• Rwightman integration added
• Ultralytics integration added
• PyTorch sparse-quantized transfer learning notebook added

SparseML v0.1.0 First GitHub Release

Welcome to our initial release on GitHub! Older release notes can be found here.

New Features:

• Keras Alpha for optimizing models using pruning added.
• PyTorch 1.7 is supported.
• PyTorch distributed supported for built-in training flows.
• Torchvision models added to PyTorch ModelRegistry class.
• MakeFile flows and utilities implemented for GitHub repo structure.

Changes:

• Software packaging updated to reflect new GitHub distribution channel, from file naming conventions to license enforcement removal.
• Migration made to use the SparseZoo package for loading pre-trained models and recipes.
• Migration made to use DeepSparse Engine for analyzing and benchmarking ONNX models.
• ONNX and ONNXRuntime dependency versions updated to include latest.

Resolved Issues:

• Infinite recursion resolved for the PyTorch ScheduledOptimizer in nested optimizer flows.
• tf2onnx folding nodes now working with Sparsify.

Known Issues:

• TensorFlow pre-trained models are not pushed currently in the SparseZoo and will fail to load.
• TensorFlow V1 is no longer being built for newer operating systems such as Ubuntu 20.04. Therefore, SparseML with TensorFlow V1 is unsupported on these operating systems as well.