Sparsify.alpha.auto

[rahul-tuli]

This PR stitches together sparsify.auto with sparsifyml

Consists of the following pre-approved pieces:

• Sync version with main #176
• Delete old sparsify.package #177

And following unapproved pieces:

• Run integration tests only if sparsifyml installed locally, else skip.
  Additionally remove sparsifyml from nm_dependencies
• Max steps propagation to teacher
• Fix failing GHA Fix: failing GHA #183

Usage:

(sparsify) ~ sparsify.auto --help
usage: sparsify.auto [-h] --task TASK --dataset DATASET
                     [--save_directory SAVE_DIRECTORY]
                     [--performance PERFORMANCE] [--base_model BASE_MODEL]
                     [--recipe RECIPE] [--recipe_args RECIPE_ARGS]
                     [--distill_teacher DISTILL_TEACHER]
                     [--num_trials NUM_TRIALS]
                     [--max_train_time MAX_TRAIN_TIME]
                     [--maximum_trial_saves MAXIMUM_TRIAL_SAVES]
                     [--no_stopping] [--resume RESUME]
                     [--optimizing_metric OPTIMIZING_METRIC]
                     [--kwargs KWARGS] [--teacher_kwargs TEACHER_KWARGS]
                     [--tuning_parameters TUNING_PARAMETERS]
                     [--teacher_tuning_parameters TEACHER_TUNING_PARAMETERS]
                     [--teacher_only]

optional arguments:
  -h, --help            show this help message and exit
  --task TASK           task to train the sparsified model on
  --dataset DATASET     path to dataset to train on
  --save_directory SAVE_DIRECTORY
                        Absolute path to save directory
  --performance PERFORMANCE
                        Preferred tradeoff between accuracy and performance.
                        Can be a string or a float value in the range [0, 1].
                        Currently supported strings (and their respective
                        float values are `accuracy` (0), `balanced` (0.5),
                        and `performant` (1.0)
  --base_model BASE_MODEL
                        path to base model to begin sparsification from
  --recipe RECIPE       file path to or zoo stub of sparsification recipe to
                        be applied
  --recipe_args RECIPE_ARGS
                        keyword args to override recipe variables with
  --distill_teacher DISTILL_TEACHER
                        teacher to use for distillation. Can be a path to a
                        model file or zoo stub, 'off' for no distillation,
                        and default value of 'auto' to auto-tune base model
                        as teacher
  --num_trials NUM_TRIALS
                        Number of tuning trials to be run before returning
                        best found model. Set to None to not impose a trial
                        limit. max_train_time may limit the actual num_trials
                        ran
  --max_train_time MAX_TRAIN_TIME
                        Maximum number of hours to train before returning
                        best trained model.
  --maximum_trial_saves MAXIMUM_TRIAL_SAVES
                        Number of best trials to save on the drive. Items
                        saved for a trial include the trained model and
                        associated artifacts. If this value is set to n, then
                        at most n+1 models will be saved at any given time on
                        the machine. Default value of None allows for
                        unlimited model saving
  --no_stopping         Set to True to turn off tuning stopping condition,
                        which may end tuning early if no improvement was made
  --resume RESUME       To continue a tuning run, provide path to the high
                        level directory of run you wish to resume
  --optimizing_metric OPTIMIZING_METRIC
                        The criterion to search model for, multiple metrics
                        can be specified, e.g. --optimizing_metric f1
                        --optimizing_metric latency. Supported metrics are
                        ['accuracy', 'f1', 'recall', 'mAP', 'latency',
                        'throughput', 'compression', 'file_size',
                        'memory_usage']
  --kwargs KWARGS       optional task specific arguments to add to config
  --teacher_kwargs TEACHER_KWARGS
                        optional task specific arguments to add to teacher
                        config
  --tuning_parameters TUNING_PARAMETERS
                        path to config file containing custom parameter
                        tuning settings. See example tuning config output for
                        expected format
  --teacher_tuning_parameters TEACHER_TUNING_PARAMETERS
                        path to config file containing custom teacher
                        parameter tuning settings. See example tuning config
                        output for expected format
  --teacher_only        set to True to only auto tune the teacher

Example Commands:

• Image Classification

sparsify.auto --task image_classification --dataset /home/ubuntu/datasets/imagenette/imagenette-160

NOTE: Using experimental fast data loading logic. To disable, pass
    "--load_fast=false" and report issues on GitHub. More details:
    https://github.com/tensorflow/tensorboard/issues/4784

TensorBoard listening on http://localhost:6006/

*************************SPARSIFY AUTO**********************
Starting hyperparameter tuning on student model
************************************************************

INFO:auto_banner:Starting hyperparameter tuning on student model

*************************SPARSIFY AUTO**********************
Starting tuning trial #0
************************************************************

INFO:auto_banner:Starting tuning trial #0
Model id is set to sparsify_auto_image_classification
2023-02-22 18:38:59 __main__     INFO     created model with key resnet50: ResNet(
  (input): _Input(
    (conv): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (act): ReLU(inplace=True)
    (pool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    
    .
    .
    .
    .
          (activation_post_process): Identity()
        )
      )
    )
  )
  (classifier): _Classifier(
    (avgpool): AdaptiveAvgPool2d(output_size=1)
    (fc): Linear(in_features=2048, out_features=10, bias=True)
    (softmax): Softmax(dim=1)
  )
)
2023-02-22 18:38:59 __main__     INFO     running on device cuda:0
INFO:__main__:running on device cuda:0
Test epoch -1/-1: 100%|██████████████████████████| 8/8 [00:01<00:00,  5.14it/s]
2023-02-22 18:39:01 __main__     INFO
Initial validation results: ModuleRunResults(__loss__=2.218540668487549, top1acc=12.199999809265137, top5acc=69.4000015258789)
INFO:__main__:
Initial validation results: ModuleRunResults(__loss__=2.218540668487549, top1acc=12.199999809265137, top5acc=69.4000015258789)
2023-02-22 18:39:01 __main__     INFO     Starting training from epoch 0
INFO:__main__:Starting training from epoch 0
Train epoch 0/206:  87%|██████████████████▎  | 176/202 [00:21<

• Transformers QA

(sparsify) ~ sparsify.auto --task question_answering --dataset squad
.
.
.
.

  warnings.warn(
[INFO|trainer.py:1607] 2023-02-22 21:54:15,467 >> ***** Running training *****
[INFO|trainer.py:1608] 2023-02-22 21:54:15,467 >>   Num examples = 88524
[INFO|trainer.py:1609] 2023-02-22 21:54:15,467 >>   Num Epochs = 3
[INFO|trainer.py:1610] 2023-02-22 21:54:15,467 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:1611] 2023-02-22 21:54:15,467 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:1612] 2023-02-22 21:54:15,467 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1613] 2023-02-22 21:54:15,467 >>   Total optimization steps = 33198
  0%|                                                | 0/33198 [00:00<?, ?it/s][W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
  0%|▏                                   | 157/33198 [00:31<1:49:56,  5.01it/s

Depends on: https://github.com/neuralmagic/sparsifyml/pull/25

[bfineran]

LGTM pending GHA issues, are these related?

[KSGulin]

LGTM, good stuff @rahul-tuli! We do need some verification that the end to end runs are producing the expected results. We can land without the full testing, but let's be ready to follow up with another PR in case there's updates to be made

[rahul-tuli]

LGTM, good stuff @rahul-tuli! We do need some verification that the end to end runs are producing the expected results. We can land without the full testing, but let's be ready to follow up with another PR in case there's updates to be made

I agree, I ran a few runs that ended fine, but will keep an eye out for solidifying the code

[rahul-tuli]

Note: CLOSED this PR by mistake, reopened now