-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add schedule&runtime tutorial doc #499
Conversation
Codecov Report
@@ Coverage Diff @@
## master #499 +/- ##
==========================================
+ Coverage 79.48% 79.87% +0.39%
==========================================
Files 106 107 +1
Lines 5975 6093 +118
Branches 968 987 +19
==========================================
+ Hits 4749 4867 +118
+ Misses 1095 1094 -1
- Partials 131 132 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
docs/tutorials/customize_runtime.md
Outdated
Create the `mmcls/core/optimizer` folder and the `mmcls/core/optimizer/__init__.py` file. | ||
The newly defined module should be imported in `mmcls/core/optimizer/__init__.py` so that the registry will | ||
find the new module and add it: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Users should also need to add from .optimizer import *
into mmcls/core/__init__.py
to register the optimzier.
docs/tutorials/customize_runtime.md
Outdated
``` | ||
|
||
The default optimizer constructor is implemented [here](https://github.com/open-mmlab/mmcv/blob/9ecd6b0d5ff9d2172c49a182eaa669e9f27bb8e7/mmcv/runner/optimizer/default_constructor.py#L11), | ||
which could also serve as a template for new optimizer constructor. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The DefaultOptimizerConstructor
supports paramwise_cfg
, please add an example about how to use it in config file.
docs/tutorials/customize_runtime.md
Outdated
|
||
## Customize Training Schedules | ||
|
||
we use step learning rate with default value in config files, this calls [`StepLRHook`](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/lr_updater.py#L153) in MMCV. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we
-> We
docs/tutorials/customize_runtime.md
Outdated
|
||
so that 1 epoch for training and 1 epoch for validation will be run iteratively. | ||
|
||
:::{note} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use
```{note}
docs/tutorials/customize_runtime.md
Outdated
] | ||
``` | ||
|
||
You can also set the priority of the hook by adding key `priority` to `'NORMAL'` or `'HIGHEST'` as below |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
'NORMAL'
or 'HIGHEST'
?
Here is the priority level table, and users can also use a specific value to modify priority finely.
Level | Value |
---|---|
HIGHEST | 0 |
VERY_HIGH | 10 |
HIGH | 30 |
ABOVE_NORMAL | 40 |
NORMAL | 50 |
BELOW_NORMAL | 60 |
LOW | 70 |
VERY_LOW | 90 |
LOWEST | 100 |
docs/tutorials/customize_runtime.md
Outdated
The above-mentioned tutorials already cover how to modify `optimizer_config`, `momentum_config`, and `lr_config`. | ||
Here we reveals how what we can do with `log_config`, `checkpoint_config`, and `evaluation`. | ||
|
||
#### Checkpoint config |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please rearrange these sections and rename titles according to the above list.
docs/tutorials/customize_runtime.md
Outdated
Some models need gradient clip to clip the gradients to stabilize the training process. An example is as below: | ||
|
||
```python | ||
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2)) | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The optimizer_config
will be passed to OptimizerHook
, users are easy to confuse it and the optimizer_cfg
in the optimizer constructor.
And users can specify different kinds of optimizer hooks, like GradientCumulativeOptimizerHook
here. Consider adding some introduction here.
docs/tutorials/customize_runtime.md
Outdated
#### Evaluation config | ||
|
||
The config of `evaluation` will be used to initialize the [`EvalHook`](https://github.com/open-mmlab/mmclassification/blob/master/mmcls/core/evaluation/eval_hooks.py). | ||
Except the key `interval`, other arguments such as `metrics` will be passed to the `dataset.evaluate()` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The EvaluationHook
supports save_best
now, many users may want this function, consider adding an introduction about it.
Co-authored-by: Ma Zerun <mzr1996@163.com>
Co-authored-by: Ma Zerun <mzr1996@163.com>
docs/tutorials/schedule.md
Outdated
|
||
### Warmup strategy | ||
|
||
在配置文件中预热(warmup)的逐步学习率调整,主要的参数有以下几个: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing translation
docs/tutorials/schedule.md
Outdated
|
||
In academic research and industrial practice, it may be necessary to use optimization methods not implemented by MMClassification, and users can add them through the following methods. | ||
|
||
```(note) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
```{note}
instead of ```(note)
docs/tutorials/runtime.md
Outdated
- [CheckpointSaverHook](#checkpointsaverhook) | ||
- [LoggerHooks](#loggerhooks) | ||
- [EvaluationHook](#evaluationhook) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The TOC's links don't match titles.
docs/tutorials/schedule.md
Outdated
@@ -0,0 +1,304 @@ | |||
# Tutorial 6: Customize Schedule | |||
|
|||
In this tutorial, we will introduce some methods about how to construct optimizers, customize learning rate and momentum schedules, use multiple learning rates and weight_decay, gradient clipping, gradient accumulation, and customize self-implemented methods for the project. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The underline is only used in variable names.
docs/tutorials/runtime.md
Outdated
checkpoint_config = dict(interval=1) | ||
``` | ||
|
||
The users could set `max_keep_ckpts` to only save only small number of checkpoints or decide whether to store state dict of optimizer by `save_optimizer`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The users
can be used in comments, which is for developers. In the documentation, just use we
or you
.
docs/tutorials/schedule.md
Outdated
|
||
**After completing your configuration file,you could use [learning rate visualization tool](https://mmclassification.readthedocs.io/zh_CN/latest/tools/visualization.html#id3) to draw the corresponding learning rate adjustment curve.** | ||
|
||
## Use multiple learning rates and weight_decays |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
## Use multiple learning rates and weight_decays | |
## Parameter-wise finely configuration |
Examples are as follows: | ||
|
||
```python | ||
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a comment about norm_type
, since it's not self-explanatory.
docs/tutorials/schedule.md
Outdated
When the optimizer hook type is not specified, `OptimizerHook` is used by default, and the above is equivalent to: | ||
|
||
```python | ||
optimizer_config = dict(type="OptimizerHook", grad_clip=dict(max_norm=35, norm_type=2)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This section is not relevant to gradient clipping, move it to where users need to change it.
docs/tutorials/schedule.md
Outdated
|
||
### Gradient accumulation | ||
|
||
When computing resources are lacking, BatchSize can only be set to a small value, which affects the effect of the resulting model. Gradient accumulation can be used to circumvent this problem. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BatchSize
? Strange uppercase.
docs/tutorials/schedule.md
Outdated
- ConsineAnnealing schedule: | ||
|
||
```python | ||
lr_config = dict( | ||
policy='CosineAnnealing', | ||
warmup='linear', | ||
warmup_iters=1000, | ||
warmup_ratio=1.0 / 10, | ||
min_lr_ratio=1e-5) | ||
``` | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Forget to replace the example?
docs/tutorials/runtime.md
Outdated
@@ -0,0 +1,265 @@ | |||
# Tutorial 7: Customize Runtime Settings | |||
|
|||
In this tutorial, we will introduce some methods about how to customize optimization methods, training schedules, workflow and hooks when running your own settings for the project. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How to customize optimization methods has been moved to tutorial 6
docs/tutorials/runtime.md
Outdated
|
||
```{note} | ||
1. The parameters of model will not be updated during val epoch. | ||
2. Keyword `total_epochs` in the config only controls the number of training epochs and will not affect the validation workflow. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
total_epochs
or max_epochs
?
docs/tutorials/runtime.md
Outdated
|
||
## Customize Workflow | ||
|
||
By default, we recommend users to use **`EvaluationHook`** to do evaluation after training epoch, but they can still use `val` workflow as an alternative. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line should be moved after the introduction.
And consider adding a note to remind users that modifying workflow is unnecessary in most situations.
docs/tutorials/runtime.md
Outdated
workflow = [('train', 1)] | ||
``` | ||
|
||
which means running 1 epoch for training. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Watch out for the uppercase.
docs/tutorials/runtime.md
Outdated
|
||
The hook mechanism is widely used in the OpenMMLab open source algorithm library. Combined with the `Runner`, the entire life cycle of the training process can be managed easily. You can learn more about the hook through [related article](https://www.calltutors.com/blog/what-is-hook/). | ||
|
||
Hooks only work when they are registered in the constructor. At present, hooks are mainly divided into two categories: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hooks only work when they are registered in the constructor. At present, hooks are mainly divided into two categories: | |
Hooks only work after being registered into the runner. At present, hooks are mainly divided into two categories: |
docs/tutorials/runtime.md
Outdated
```{note} | ||
1. In the default configuration files of MMClassification, the evaluation field is generally placed in the datasets configs. | ||
|
||
2. 'EvalHook' in 'MMClassification/mmcls/core/evaluation/eval_hooks.py' will be deprecated, recommend to use 'EvaluationHook' in MMCV as above. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This note can be removed because config file modification is not relevant to use which implementation.
docs/tutorials/runtime.md
Outdated
|
||
### Use implemented hooks | ||
|
||
Some hooks have been already implemented in MMCV 和 MMClassification: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and
docs/tutorials/runtime.md
Outdated
|
||
- load_from : only imports model weights, which is mainly used to load pre-trained or trained models; | ||
|
||
- resume_from : not only import model weights, but also optimizer information, current epoch information, mainly used to continue training from the breakpoint. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
breakpoint
-> checkpoint
docs/tutorials/runtime.md
Outdated
|
||
- resume_from : not only import model weights, but also optimizer information, current epoch information, mainly used to continue training from the breakpoint. | ||
|
||
- init_cfg.Pretrained : load the model weight, and you can specify a specific ‘key’ layer to load. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- init_cfg.Pretrained : load the model weight, and you can specify a specific ‘key’ layer to load. | |
- init_cfg.Pretrained : Load weights during weight initialization, and you can specify which module to load. This is usually used when fine-tuning a model. |
docs/tutorials/runtime.md
Outdated
- init_cfg.Pretrained : load the model weight, and you can specify a specific ‘key’ layer to load. | ||
|
||
```{note} | ||
It is recommended to specify pre-training weights using init_cfg.Pretrained when fine-tuning the model. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a link to the fine-tuning tutorial.
docs_zh-CN/tutorials/schedule.md
Outdated
@@ -200,6 +200,15 @@ momentum_config = dict( | |||
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2)) | |||
``` | |||
|
|||
当使用继承并修改基础配置方式时,如果基础配置中 `grad_clip=None`,需要添加 `_delete_=True`。有关 `_delete_` 可以参靠[教程 1:如何编写配置文件](https://mmclassification.readthedocs.io/zh_CN/latest/tutorials/config.html#id16)。案列如下: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
“参靠”,“案列”
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
* add cn tutorials/config.md * add heads api and doc title link * Update tutorials index * Update tutorials index * Update config.md * add english version * Update config.md * add custom_runtime * Update docs * modify title * modify en to zh_CN in chinses docs * Update Readme * fix punctuations * Update docs/tutorials/customize_runtime.md Co-authored-by: Ma Zerun <mzr1996@163.com> * Update docs/tutorials/customize_runtime.md Co-authored-by: Ma Zerun <mzr1996@163.com> * split to schedule and runtime * fix lint * improve docs after review * fix TOC * imporve expersion * fix an error * Imporve schedule.md * Improve runtime.md * Improve chinese docs. * Fix toc-tree * fix en link and add a case of gradient clipping * fix wrong word Co-authored-by: Ma Zerun <mzr1996@163.com>
Motivation
Add schedule and runtime tutorial documentation.
Modification
Checklist
Before PR:
After PR: