Auto pruners #2490

suiguoxin · 2020-05-26T07:13:13Z

Add algo implementation / examples / test / doc for the following pruning algos:

NetAdapt
SimulatedAnnealing
ADMM
AutoCompress

QuanluZhang · 2020-06-30T03:31:12Z

docs/en_US/Compressor/Pruner.md

+- **trainer:** Function used for the first optimization subproblem.
+This function should include `model, optimizer, criterion, epoch, callback` as parameters, where callback should be inserted after loss.backward of the normal training process.
+- **optimize_iteration:** ADMM optimize iterations.
+- **training_epochs:** training epochs of the first optimization subproblem.


it is not clear what is "the first optimization subproblem". better to give a little more description in the introduction of this pruner

QuanluZhang · 2020-06-30T03:32:02Z

docs/en_US/Compressor/Pruner.md

+- **optimize_iteration:** ADMM optimize iterations.
+- **training_epochs:** training epochs of the first optimization subproblem.
+- **row:** penalty parameters for ADMM training.
+- **base_algo:** base pruning algorithm. 'level', 'l1' or 'l2', by default 'l1'.


why this one does not have experiment_data_dir?

ADMMPruner is not an auto pruner, there is thus no experiment data generated. More explanation on what is included as experiment data added for the auto pruners.

QuanluZhang · 2020-06-30T03:32:26Z

docs/en_US/Compressor/Pruner.md

+
+
+## AutoCompress Pruner
+For each round t, AutoCompressPruner prune the model for the same sparsity each round to achive the ovrall sparsity:


ovrall -> overall

QuanluZhang · 2020-06-30T05:07:57Z

docs/en_US/Compressor/Pruner.md

+- **sparsity:** How much percentage of convolutional filters are to be pruned.
+- **op_types:** "Conv2d" or "default".
+- **trainer:** Function used for the first optimization subproblem.


it is not clear how to write trainer. who should provide callback? what is the reason to provide callback? why it should be put behind loss.backward?

QuanluZhang · 2020-06-30T05:08:31Z

docs/en_US/Compressor/Pruner.md

+This function should include `model, optimizer, criterion, epoch, callback` as parameters, where callback should be inserted after loss.backward of the normal training process.
+- **evaluator:** Function to evaluate the masked model. This function should include `model` as the only parameter, and returns a scalar value.
+- **dummy_input:** The dummy input for model speed up, users should put it on right device before pass in.


why there is model speed up here?

speedup is called inside AutoCompress to keep the model un-masked and realize real pruning after each iteration.

QuanluZhang · 2020-06-30T05:09:24Z

docs/en_US/Compressor/Pruner.md

+- **dummy_input:** The dummy input for model speed up, users should put it on right device before pass in.
+- **iterations:** The number of overall iterations.
+- **optimize_mode:** Optimize mode, 'maximize' or 'minimize', by default 'maximize'.


only this auto pruner supports optimize_mode?

optimize_mode is supported in NetAdapt SimualatedAnnealing and AucoCompress. Sorry for have missed this arg for NetAdaptPruner.

QuanluZhang · 2020-06-30T05:10:15Z

docs/en_US/Compressor/Pruner.md

+- **cool_down_rate:** Simualated Annealing related parameter.
+- **perturbation_magnitude:** Initial perturbation magnitude to the sparsities. The magnitude decreases with current temperature.
+- **optimize_iteration:** ADMM optimize iterations.


what is the relation with ADMM?

AutoCompress Pruner call SimualtedAnnealing Pruner and ADMM Pruner iteratively.

QuanluZhang · 2020-06-30T05:10:48Z

docs/en_US/Compressor/Pruner.md

+- **perturbation_magnitude:** Initial perturbation magnitude to the sparsities. The magnitude decreases with current temperature.
+- **optimize_iteration:** ADMM optimize iterations.
+- **epochs:** training epochs of the first optimization subproblem.


this one also has two subproblems?

These are args for ADMM

colorjam · 2020-06-29T08:27:20Z

src/sdk/pynni/nni/compression/torch/pruning/auto_compress_pruner.py

+        """
+        _logger.info('Starting AutoCompress pruning...')
+
+        sparsity_each_round = 1 - pow(1-self._sparsity, 1/self._optimize_iterations)


Why use this sparsity strategy?

This strategy is used to ensure that same number of weights will be pruned in each iteration.

colorjam · 2020-06-30T03:24:49Z

src/sdk/pynni/nni/compression/torch/pruning/net_adapt_pruner.py

+        1. Con = Res_i - delta_Res
+        2. for every layer:
+            Choose Num Filters to prune
+            Choose which filter to prunee


prunee -> prune

colorjam · 2020-06-30T03:25:41Z

src/sdk/pynni/nni/compression/torch/pruning/net_adapt_pruner.py

+            and fine tune the model for a short term after each pruning iteration.
+        optimize_mode : str
+            optimize mode, 'maximize' or 'minimize', by default 'maximize'
+        base_algo : str


better to add a description that we use base_algo to choose which filter to prune

chicm-ms · 2020-06-30T08:01:13Z

docs/en_US/Compressor/Pruner.md

@@ -398,5 +402,176 @@ We try to reproduce the experiment result of the fully connected network on MNIS
 The above figure shows the result of the fully connected network. `round0-sparsity-0.0` is the performance without pruning. Consistent with the paper, pruning around 80% also obtain similar performance compared to non-pruning, and converges a little faster. If pruning too much, e.g., larger than 94%, the accuracy becomes lower and convergence becomes a little slower. A little different from the paper, the trend of the data in the paper is relatively more clear.


+## NetAdapt Pruner


The order of each section is better consistent with the content directory/list at beginning.

colorjam · 2020-06-30T08:44:57Z

src/sdk/pynni/nni/compression/torch/pruning/auto_compress_pruner.py

+
+            # use speed up to prune the model before next iteration, because SimulatedAnnealingPruner & ADMMPruner don't take masked models
+            self._model_to_prune.load_state_dict(torch.load(os.path.join(
+                self._experiment_data_dir, 'model_admm_masked.pth')))


Why reload the checkpoint?

The model weights have changed after admm pruner.

colorjam · 2020-06-30T08:46:52Z

src/sdk/pynni/nni/compression/torch/pruning/admm_pruner.py

+            Penalty parameters for ADMM training.
+        base_algo : str
+            Base pruning algorithm. `level`, `l1` or `l2`, by default `l1`.
+            Given the sparsity distrution among the ops, the assigned `base_algo` is used to decide which filters/channels/weights to prune.


distrution -> distribution

suiguoxin added 30 commits April 21, 2020 10:05

init sapruner

80165e9

seperate sapruners from other one-shot pruners

3fcebef

update

b4be2d0

fix model params issue

805c32c

make the process runnable

4c33432

show evaluation result in example

489f7b6

sort the sparsities and scale it

00dbddf

fix rescale issue

e1f9654

fix scale issue; add pruning history

4b5ea0d

record the actual total sparsity

6120b70

fix sparsity 0/1 problem

a6114f7

revert useless modif

2e928ae

revert useless modif

546ca73

fix 0 pruning weights problem

1dc4713

save pruning history in csv file

d1a5646

fix typo

75a53da

remove check perm in Makefile

e8900f8

use os path

9c5ba41

save config list in json format

9a60501

update analyze py; update docker

951c60f

update

c784790

update analyze

836e74b

update log info in compressor

70aca26

init NetAdapt Pruner

efa0637

refine examples

8695564

Merge remote-tracking branch 'msft/master' into sapruner

8fdad96

update

db3074e

fine tune

78ee01a

update

3e40c4a

fix quote issue

2560050

QuanluZhang reviewed Jun 30, 2020

View reviewed changes

refine doc

d86fad4

suiguoxin force-pushed the auto-pruners branch from af6926a to d86fad4 Compare June 30, 2020 05:57

suiguoxin and others added 4 commits June 30, 2020 06:19

resolve conflict

c29f758

refine doc

32d14d9

refince doc

5950bec

resolve conflict

8a11b45

colorjam reviewed Jun 30, 2020

View reviewed changes

chicm-ms reviewed Jun 30, 2020

View reviewed changes

suiguoxin force-pushed the auto-pruners branch from f1ca4a0 to 76b0393 Compare June 30, 2020 08:10

refine doc

be782bc

suiguoxin force-pushed the auto-pruners branch from 76b0393 to be782bc Compare June 30, 2020 08:12

colorjam reviewed Jun 30, 2020

View reviewed changes

chicm-ms approved these changes Jun 30, 2020

View reviewed changes

suiguoxin added 3 commits June 30, 2020 10:07

refine doc

d449e6a

refine doc

cb6376b

refine doc

cee3fdd

colorjam approved these changes Jun 30, 2020

View reviewed changes

QuanluZhang approved these changes Jun 30, 2020

View reviewed changes

chicm-ms merged commit f5caa19 into microsoft:master Jun 30, 2020

chicm-ms mentioned this pull request Jul 1, 2020

June release end game #2621

Closed

24 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto pruners #2490

Auto pruners #2490

suiguoxin commented May 26, 2020

QuanluZhang Jun 30, 2020

suiguoxin Jun 30, 2020

QuanluZhang Jun 30, 2020

suiguoxin Jun 30, 2020

QuanluZhang Jun 30, 2020

suiguoxin Jun 30, 2020

QuanluZhang Jun 30, 2020

suiguoxin Jun 30, 2020

QuanluZhang Jun 30, 2020

suiguoxin Jun 30, 2020

QuanluZhang Jun 30, 2020

suiguoxin Jun 30, 2020

QuanluZhang Jun 30, 2020

suiguoxin Jun 30, 2020

QuanluZhang Jun 30, 2020

suiguoxin Jun 30, 2020

colorjam Jun 29, 2020

suiguoxin Jun 30, 2020

colorjam Jun 30, 2020

suiguoxin Jun 30, 2020

colorjam Jun 30, 2020

suiguoxin Jun 30, 2020

chicm-ms Jun 30, 2020 •

edited

Loading

suiguoxin Jun 30, 2020

colorjam Jun 30, 2020

suiguoxin Jun 30, 2020

colorjam Jun 30, 2020

suiguoxin Jun 30, 2020



		## AutoCompress Pruner
		For each round t, AutoCompressPruner prune the model for the same sparsity each round to achive the ovrall sparsity:

		@@ -398,5 +402,176 @@ We try to reproduce the experiment result of the fully connected network on MNIS
		The above figure shows the result of the fully connected network. `round0-sparsity-0.0` is the performance without pruning. Consistent with the paper, pruning around 80% also obtain similar performance compared to non-pruning, and converges a little faster. If pruning too much, e.g., larger than 94%, the accuracy becomes lower and convergence becomes a little slower. A little different from the paper, the trend of the data in the paper is relatively more clear.


		## NetAdapt Pruner

Auto pruners #2490

Auto pruners #2490

Conversation

suiguoxin commented May 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chicm-ms Jun 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chicm-ms Jun 30, 2020 •

edited

Loading