[Feature]Support Relational Knowledge Distillation #127

HIT-cwh · 2022-03-28T12:20:14Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

Modification

Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.

Checklist

Before PR:

Pre-commit or other linting tools are used to fix the potential lint issues.
Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
CLA has been signed and all committers have signed the CLA in this PR.

codecov · 2022-03-28T12:34:38Z

Codecov Report

Merging #127 (0132b63) into dev_v0.3.0 (f5ee768) will increase coverage by 1.08%.
The diff coverage is 95.00%.

@@              Coverage Diff               @@
##           dev_v0.3.0     #127      +/-   ##
==============================================
+ Coverage       65.28%   66.36%   +1.08%     
==============================================
  Files              91       92       +1     
  Lines            3272     3366      +94     
  Branches          600      613      +13     
==============================================
+ Hits             2136     2234      +98     
+ Misses           1036     1030       -6     
- Partials          100      102       +2

Flag	Coverage Δ
unittests	`66.33% <95.00%> (+1.08%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmrazor/models/losses/relational_kd.py	`94.82% <94.82%> (ø)`
mmrazor/models/losses/__init__.py	`100.00% <100.00%> (ø)`
mmrazor/models/pruners/structure_pruning.py	`89.53% <0.00%> (+1.14%)`	⬆️
mmrazor/core/distributed_wrapper.py	`42.10% <0.00%> (+4.96%)`	⬆️
mmrazor/models/architectures/base.py	`100.00% <0.00%> (+13.33%)`	⬆️
mmrazor/models/architectures/mmdet.py	`100.00% <0.00%> (+36.36%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5ee768...0132b63. Read the comment docs.

mmrazor/models/losses/rkd.py

configs/distill/rkd/README.md

mmrazor/models/losses/rkd.py

configs/distill/rkd/rkd_neck_resnet34_resnet18_8xb32_in1k.py

mmrazor/models/losses/rkd.py

configs/distill/rkd/rkd_neck_resnet50_vgg11bn_8xb16_cifar100.py

pppppM · 2022-04-01T08:01:28Z

The RelationalKD should split to RelationalDistLoss and RelationalAngleLoss, and rename the rkd.py

* [Feature] Add function to meet mmdeploy support (#102) * add init_model function for mmdeploy * fix lint * add unittest for init_xxx_model * fix lint * mv test_inference.py to test_apis directory * [Feature] Add function to meet mmdeploy support (#102) * add init_model function for mmdeploy * fix lint * add unittest for init_xxx_model * fix lint * mv test_inference.py to test_apis directory * [Refactor] Delete redundant `set_random_seed` function (#104) * refactor set_random_seed * add unittests * fix unittests error * fix lint * avoid bc breaking * [Feature] Add diff seeds to diff ranks and set torch seed in worker_init_fn (#113) * add init_random_seed * Set diff seed to diff workers * [Feature] Add multi machine dist_train (#114) * support multi nodes * update training doc * fix lints * remove fixed seed * fix ddp wrapper registry (#128) * [Docs] Add brief installation steps in README(_zh-CN).md (#121) * Add brief installation * add brief installtion ref to mmediting pr#816 Co-authored-by: caoweihan <caoweihan@sensetime.com> * [BUG]Fix bugs in pruner (#126) * fix bugs in pruner when pruning models with shared modules * pruner can trace models with dilation conv2d * fix deploy_subnet * fix add_pruning_attrs * fix bugs in modify_forward * fix lint * fix StructurePruner * test tracing models with shared modules Co-authored-by: caoweihan <caoweihan@sensetime.com> * [Docs]Add some more details to docs (#133) * add docs for dataset * add cfg-options for distillation * fix docs Co-authored-by: caoweihan <caoweihan@sensetime.com> * reset norm running status after prepare_from_supernet (#81) * [Improvement]Sync train api (#115) Co-authored-by: caoweihan <caoweihan@sensetime.com> * [Feature]Support Relational Knowledge Distillation (#127) * add rkd * add rkd pytest * add rkd configs * fix readme * fix rkd * split rkd loss to distance-wise and angle-wise losses * rename rkd losses * add rkd metaflie * add rkd related links * rename rkd metafile and add to model index * delete cifar100 Co-authored-by: caoweihan <caoweihan@sensetime.com> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: qiufeng <44188071+wutongshenqiu@users.noreply.github.com> Co-authored-by: wutongshenqiu <690364065@qq.com> Co-authored-by: whcao <41630003+HIT-cwh@users.noreply.github.com> Co-authored-by: caoweihan <caoweihan@sensetime.com>

* [Feature] Add function to meet mmdeploy support (open-mmlab#102) * add init_model function for mmdeploy * fix lint * add unittest for init_xxx_model * fix lint * mv test_inference.py to test_apis directory * [Feature] Add function to meet mmdeploy support (open-mmlab#102) * add init_model function for mmdeploy * fix lint * add unittest for init_xxx_model * fix lint * mv test_inference.py to test_apis directory * [Refactor] Delete redundant `set_random_seed` function (open-mmlab#104) * refactor set_random_seed * add unittests * fix unittests error * fix lint * avoid bc breaking * [Feature] Add diff seeds to diff ranks and set torch seed in worker_init_fn (open-mmlab#113) * add init_random_seed * Set diff seed to diff workers * [Feature] Add multi machine dist_train (open-mmlab#114) * support multi nodes * update training doc * fix lints * remove fixed seed * fix ddp wrapper registry (open-mmlab#128) * [Docs] Add brief installation steps in README(_zh-CN).md (open-mmlab#121) * Add brief installation * add brief installtion ref to mmediting pr#816 Co-authored-by: caoweihan <caoweihan@sensetime.com> * [BUG]Fix bugs in pruner (open-mmlab#126) * fix bugs in pruner when pruning models with shared modules * pruner can trace models with dilation conv2d * fix deploy_subnet * fix add_pruning_attrs * fix bugs in modify_forward * fix lint * fix StructurePruner * test tracing models with shared modules Co-authored-by: caoweihan <caoweihan@sensetime.com> * [Docs]Add some more details to docs (open-mmlab#133) * add docs for dataset * add cfg-options for distillation * fix docs Co-authored-by: caoweihan <caoweihan@sensetime.com> * reset norm running status after prepare_from_supernet (open-mmlab#81) * [Improvement]Sync train api (open-mmlab#115) Co-authored-by: caoweihan <caoweihan@sensetime.com> * [Feature]Support Relational Knowledge Distillation (open-mmlab#127) * add rkd * add rkd pytest * add rkd configs * fix readme * fix rkd * split rkd loss to distance-wise and angle-wise losses * rename rkd losses * add rkd metaflie * add rkd related links * rename rkd metafile and add to model index * delete cifar100 Co-authored-by: caoweihan <caoweihan@sensetime.com> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: qiufeng <44188071+wutongshenqiu@users.noreply.github.com> Co-authored-by: wutongshenqiu <690364065@qq.com> Co-authored-by: whcao <41630003+HIT-cwh@users.noreply.github.com> Co-authored-by: caoweihan <caoweihan@sensetime.com>

caoweihan added 3 commits March 28, 2022 19:38

add rkd

8f570cd

add rkd pytest

bf61b6d

add rkd configs

d000fe0

humu789 self-assigned this Mar 30, 2022

humu789 requested changes Mar 30, 2022

View reviewed changes

fix readme

9677197

pppppM requested changes Apr 1, 2022

View reviewed changes

fix rkd

052ee3b

pppppM requested changes Apr 1, 2022

View reviewed changes

configs/distill/rkd/rkd_neck_resnet50_vgg11bn_8xb16_cifar100.py Outdated Show resolved Hide resolved

caoweihan and others added 6 commits April 1, 2022 16:34

split rkd loss to distance-wise and angle-wise losses

4e55516

rename rkd losses

c354869

add rkd metaflie

d30de6f

add rkd related links

ca306fa

rename rkd metafile and add to model index

c915e09

delete cifar100

0132b63

pppppM approved these changes Apr 2, 2022

View reviewed changes

humu789 approved these changes Apr 2, 2022

View reviewed changes

pppppM merged commit de4dd13 into open-mmlab:dev_v0.3.0 Apr 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature]Support Relational Knowledge Distillation #127

[Feature]Support Relational Knowledge Distillation #127

HIT-cwh commented Mar 28, 2022

codecov bot commented Mar 28, 2022 •

edited

Loading

pppppM commented Apr 1, 2022

[Feature]Support Relational Knowledge Distillation #127

[Feature]Support Relational Knowledge Distillation #127

Conversation

HIT-cwh commented Mar 28, 2022

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

codecov bot commented Mar 28, 2022 • edited Loading

Codecov Report

pppppM commented Apr 1, 2022

codecov bot commented Mar 28, 2022 •

edited

Loading