Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve] Update benchmark scripts #1028

Merged
merged 11 commits into from
Sep 20, 2022

Conversation

mzr1996
Copy link
Member

@mzr1996 mzr1996 commented Sep 13, 2022

This is a PR for mmcls 1.x

Motivation

Refactor and improve dev scripts

Modification

  1. Refactor to support MMEngine.
  2. Support --cfg-options to specify the same options
# AMP training benchmark
python .dev_scripts/benchmark_regression/3-benchmark_train.py mm_model \
    --job-name cls-amp --work-dir work_dirs/benchmark_train_amp \
    --cfg-options optim_wrapper.type='AmpOptimWrapper' optim_wrapper.loss_scale='dynamic'
  1. Use --range to specify the training benchmark range
# Do quarterly training benchmark
python .dev_scripts/benchmark_regression/3-benchmark_train.py mm_model \
    --range quarter
# Do half-yearly training benchmark, but not include quarterly models
python .dev_scripts/benchmark_regression/3-benchmark_train.py mm_model \
    --range "=half-year"
  1. Fix some training configs.
    • Use BCE in rsb-a1 training recipe
    • Fix the wrong ShuffleNet training schedule.

@mzr1996 mzr1996 force-pushed the 1x-update-train-benchmark branch 3 times, most recently from 7208286 to a58216f Compare September 14, 2022 02:06
@codecov
Copy link

codecov bot commented Sep 14, 2022

Codecov Report

Base: 0.02% // Head: 91.47% // Increases project coverage by +91.44% 🎉

Coverage data is based on head (dbcc7e8) compared to base (b8b31e9).
Patch coverage: 53.33% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff              @@
##           dev-1.x    #1028       +/-   ##
============================================
+ Coverage     0.02%   91.47%   +91.44%     
============================================
  Files          121      120        -1     
  Lines         8217     8230       +13     
  Branches      1368     1366        -2     
============================================
+ Hits             2     7528     +7526     
+ Misses        8215      571     -7644     
- Partials         0      131      +131     
Flag Coverage Δ
unittests 91.47% <53.33%> (+91.44%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmcls/apis/inference.py 0.00% <0.00%> (ø)
mmcls/datasets/multi_label.py 100.00% <ø> (+100.00%) ⬆️
mmcls/models/backbones/timm_backbone.py 28.26% <0.00%> (+28.26%) ⬆️
mmcls/models/utils/attention.py 72.72% <10.44%> (+72.72%) ⬆️
mmcls/models/backbones/vision_transformer.py 74.31% <22.22%> (+74.31%) ⬆️
mmcls/datasets/dataset_wrappers.py 25.97% <22.53%> (+25.97%) ⬆️
mmcls/models/losses/label_smooth_loss.py 98.11% <83.33%> (+98.11%) ⬆️
mmcls/datasets/transforms/formatting.py 88.09% <95.65%> (+88.09%) ⬆️
mmcls/__init__.py 100.00% <100.00%> (+84.61%) ⬆️
mmcls/datasets/transforms/__init__.py 100.00% <100.00%> (+100.00%) ⬆️
... and 155 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Collaborator

@Ezra-Yu Ezra-Yu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@mzr1996 mzr1996 added the 1.0rc Functionalities for MMClassification 1.0rc label Sep 20, 2022
@mzr1996 mzr1996 merged commit e9e2d48 into open-mmlab:dev-1.x Sep 20, 2022
mzr1996 added a commit to mzr1996/mmpretrain that referenced this pull request Nov 24, 2022
* Update train benchmark scripts

* Add `--cfg-options` for dev scripts and enhance `--range`.

* Fix bug of regex expression.

* Fix minor bugs

* Update ShuffleNet configs

* Update rsb-a1 configs and label smooth loss mode.

* Update inference dev scripts

* From `mmengine` instead of `mmcv` import fileio.

* Fix lint

* Update pre-commit hook

* Use `use_sigmoid` option instead of "bce" mode in label smooth loss.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1.0rc Functionalities for MMClassification 1.0rc
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants