Add schedule&runtime tutorial doc #499
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #499 +/- ##
==========================================
+ Coverage 79.48% 79.87% +0.39%
==========================================
Files 106 107 +1
Lines 5975 6093 +118
Branches 968 987 +19
==========================================
+ Hits 4749 4867 +118
+ Misses 1095 1094 -1
- Partials 131 132 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
mzr1996
requested changes
Nov 3, 2021
docs/tutorials/customize_runtime.md
Outdated
Comment on lines
79
to
81
| Create the `mmcls/core/optimizer` folder and the `mmcls/core/optimizer/__init__.py` file. | ||
| The newly defined module should be imported in `mmcls/core/optimizer/__init__.py` so that the registry will | ||
| find the new module and add it: |
There was a problem hiding this comment.
Users should also need to add from .optimizer import * into mmcls/core/__init__.py to register the optimzier.
docs/tutorials/customize_runtime.md
Outdated
| ``` | ||
|
|
||
| The default optimizer constructor is implemented [here](https://github.com/open-mmlab/mmcv/blob/9ecd6b0d5ff9d2172c49a182eaa669e9f27bb8e7/mmcv/runner/optimizer/default_constructor.py#L11), | ||
| which could also serve as a template for new optimizer constructor. |
There was a problem hiding this comment.
The DefaultOptimizerConstructor supports paramwise_cfg, please add an example about how to use it in config file.
docs/tutorials/customize_runtime.md
Outdated
|
|
||
| ## Customize Training Schedules | ||
|
|
||
| we use step learning rate with default value in config files, this calls [`StepLRHook`](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/lr_updater.py#L153) in MMCV. |
docs/tutorials/customize_runtime.md
Outdated
|
|
||
| so that 1 epoch for training and 1 epoch for validation will be run iteratively. | ||
|
|
||
| :::{note} |
docs/tutorials/customize_runtime.md
Outdated
| ] | ||
| ``` | ||
|
|
||
| You can also set the priority of the hook by adding key `priority` to `'NORMAL'` or `'HIGHEST'` as below |
There was a problem hiding this comment.
'NORMAL' or 'HIGHEST'?
Here is the priority level table, and users can also use a specific value to modify priority finely.
| Level | Value |
|---|---|
| HIGHEST | 0 |
| VERY_HIGH | 10 |
| HIGH | 30 |
| ABOVE_NORMAL | 40 |
| NORMAL | 50 |
| BELOW_NORMAL | 60 |
| LOW | 70 |
| VERY_LOW | 90 |
| LOWEST | 100 |
docs/tutorials/customize_runtime.md
Outdated
| The above-mentioned tutorials already cover how to modify `optimizer_config`, `momentum_config`, and `lr_config`. | ||
| Here we reveals how what we can do with `log_config`, `checkpoint_config`, and `evaluation`. | ||
|
|
||
| #### Checkpoint config |
There was a problem hiding this comment.
Please rearrange these sections and rename titles according to the above list.
docs/tutorials/customize_runtime.md
Outdated
Comment on lines
140
to
144
| Some models need gradient clip to clip the gradients to stabilize the training process. An example is as below: | ||
|
|
||
| ```python | ||
| optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2)) | ||
| ``` |
There was a problem hiding this comment.
The optimizer_config will be passed to OptimizerHook, users are easy to confuse it and the optimizer_cfg in the optimizer constructor.
And users can specify different kinds of optimizer hooks, like GradientCumulativeOptimizerHook here. Consider adding some introduction here.
docs/tutorials/customize_runtime.md
Outdated
| #### Evaluation config | ||
|
|
||
| The config of `evaluation` will be used to initialize the [`EvalHook`](https://github.com/open-mmlab/mmclassification/blob/master/mmcls/core/evaluation/eval_hooks.py). | ||
| Except the key `interval`, other arguments such as `metrics` will be passed to the `dataset.evaluate()` |
There was a problem hiding this comment.
The EvaluationHook supports save_best now, many users may want this function, consider adding an introduction about it.
Co-authored-by: Ma Zerun <mzr1996@163.com>
Co-authored-by: Ma Zerun <mzr1996@163.com>
Ezra-Yu
changed the title
mzr1996
requested changes
Nov 8, 2021
docs/tutorials/schedule.md
Outdated
|
|
||
| ### Warmup strategy | ||
|
|
||
| 在配置文件中预热（warmup）的逐步学习率调整，主要的参数有以下几个： |
docs/tutorials/schedule.md
Outdated
|
|
||
| In academic research and industrial practice, it may be necessary to use optimization methods not implemented by MMClassification, and users can add them through the following methods. | ||
|
|
||
| ```(note) |
docs/tutorials/runtime.md
Outdated
Comment on lines
10
to
12
| - [CheckpointSaverHook](#checkpointsaverhook) | ||
| - [LoggerHooks](#loggerhooks) | ||
| - [EvaluationHook](#evaluationhook) |
There was a problem hiding this comment.
The TOC's links don't match titles.
docs/tutorials/schedule.md
Outdated
| @@ -0,0 +1,304 @@ | |||
| # Tutorial 6: Customize Schedule | |||
|
|
|||
| In this tutorial, we will introduce some methods about how to construct optimizers, customize learning rate and momentum schedules, use multiple learning rates and weight_decay, gradient clipping, gradient accumulation, and customize self-implemented methods for the project. | |||
There was a problem hiding this comment.
The underline is only used in variable names.
docs/tutorials/runtime.md
Outdated
| checkpoint_config = dict(interval=1) | ||
| ``` | ||
|
|
||
| The users could set `max_keep_ckpts` to only save only small number of checkpoints or decide whether to store state dict of optimizer by `save_optimizer`. |
There was a problem hiding this comment.
The users can be used in comments, which is for developers. In the documentation, just use we or you.
docs/tutorials/schedule.md
Outdated
|
|
||
| **After completing your configuration file，you could use [learning rate visualization tool](https://mmclassification.readthedocs.io/zh_CN/latest/tools/visualization.html#id3) to draw the corresponding learning rate adjustment curve.** | ||
|
|
||
| ## Use multiple learning rates and weight_decays |
There was a problem hiding this comment.
Suggested change
| ## Use multiple learning rates and weight_decays | |
| ## Parameter-wise finely configuration |
| Examples are as follows: | ||
|
|
||
| ```python | ||
| optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2)) |
There was a problem hiding this comment.
Add a comment about norm_type, since it's not self-explanatory.
docs/tutorials/schedule.md
Outdated
Comment on lines
186
to
189
| When the optimizer hook type is not specified, `OptimizerHook` is used by default, and the above is equivalent to: | ||
|
|
||
| ```python | ||
| optimizer_config = dict(type="OptimizerHook", grad_clip=dict(max_norm=35, norm_type=2)) |
There was a problem hiding this comment.
This section is not relevant to gradient clipping, move it to where users need to change it.
docs/tutorials/schedule.md
Outdated
|
|
||
| ### Gradient accumulation | ||
|
|
||
| When computing resources are lacking, BatchSize can only be set to a small value, which affects the effect of the resulting model. Gradient accumulation can be used to circumvent this problem. |
docs/tutorials/schedule.md
Outdated
Comment on lines
197
to
207
| - ConsineAnnealing schedule: | ||
|
|
||
| ```python | ||
| lr_config = dict( | ||
| policy='CosineAnnealing', | ||
| warmup='linear', | ||
| warmup_iters=1000, | ||
| warmup_ratio=1.0 / 10, | ||
| min_lr_ratio=1e-5) | ||
| ``` | ||
|
|
mzr1996
requested changes
Nov 8, 2021
docs/tutorials/runtime.md
Outdated
| @@ -0,0 +1,265 @@ | |||
| # Tutorial 7: Customize Runtime Settings | |||
|
|
|||
| In this tutorial, we will introduce some methods about how to customize optimization methods, training schedules, workflow and hooks when running your own settings for the project. | |||
There was a problem hiding this comment.
How to customize optimization methods has been moved to tutorial 6
docs/tutorials/runtime.md
Outdated
|
|
||
| ```{note} | ||
| 1. The parameters of model will not be updated during val epoch. | ||
| 2. Keyword `total_epochs` in the config only controls the number of training epochs and will not affect the validation workflow. |
docs/tutorials/runtime.md
Outdated
|
|
||
| ## Customize Workflow | ||
|
|
||
| By default, we recommend users to use **`EvaluationHook`** to do evaluation after training epoch, but they can still use `val` workflow as an alternative. |
There was a problem hiding this comment.
This line should be moved after the introduction.
And consider adding a note to remind users that modifying workflow is unnecessary in most situations.
docs/tutorials/runtime.md
Outdated
| workflow = [('train', 1)] | ||
| ``` | ||
|
|
||
| which means running 1 epoch for training. |
docs/tutorials/runtime.md
Outdated
|
|
||
| The hook mechanism is widely used in the OpenMMLab open source algorithm library. Combined with the `Runner`, the entire life cycle of the training process can be managed easily. You can learn more about the hook through [related article](https://www.calltutors.com/blog/what-is-hook/). | ||
|
|
||
| Hooks only work when they are registered in the constructor. At present, hooks are mainly divided into two categories: |
There was a problem hiding this comment.
Suggested change
| Hooks only work when they are registered in the constructor. At present, hooks are mainly divided into two categories: | |
| Hooks only work after being registered into the runner. At present, hooks are mainly divided into two categories: |
docs/tutorials/runtime.md
Outdated
| ```{note} | ||
| 1. In the default configuration files of MMClassification, the evaluation field is generally placed in the datasets configs. | ||
|
|
||
| 2. 'EvalHook' in 'MMClassification/mmcls/core/evaluation/eval_hooks.py' will be deprecated, recommend to use 'EvaluationHook' in MMCV as above. |
There was a problem hiding this comment.
This note can be removed because config file modification is not relevant to use which implementation.
docs/tutorials/runtime.md
Outdated
|
|
||
| ### Use implemented hooks | ||
|
|
||
| Some hooks have been already implemented in MMCV 和 MMClassification: |
docs/tutorials/runtime.md
Outdated
|
|
||
| - load_from : only imports model weights, which is mainly used to load pre-trained or trained models; | ||
|
|
||
| - resume_from : not only import model weights, but also optimizer information, current epoch information, mainly used to continue training from the breakpoint. |
docs/tutorials/runtime.md
Outdated
|
|
||
| - resume_from : not only import model weights, but also optimizer information, current epoch information, mainly used to continue training from the breakpoint. | ||
|
|
||
| - init_cfg.Pretrained : load the model weight, and you can specify a specific ‘key’ layer to load. |
There was a problem hiding this comment.
Suggested change
| - init_cfg.Pretrained : load the model weight, and you can specify a specific ‘key’ layer to load. | |
| - init_cfg.Pretrained : Load weights during weight initialization, and you can specify which module to load. This is usually used when fine-tuning a model. |
docs/tutorials/runtime.md
Outdated
| - init_cfg.Pretrained : load the model weight, and you can specify a specific ‘key’ layer to load. | ||
|
|
||
| ```{note} | ||
| It is recommended to specify pre-training weights using init_cfg.Pretrained when fine-tuning the model. |
There was a problem hiding this comment.
Add a link to the fine-tuning tutorial.
mzr1996
requested changes
Nov 17, 2021
docs_zh-CN/tutorials/schedule.md
Outdated
| @@ -200,6 +200,15 @@ momentum_config = dict( | |||
| optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2)) | |||
| ``` | |||
|
|
|||
| 当使用继承并修改基础配置方式时，如果基础配置中 `grad_clip=None`，需要添加 `_delete_=True`。有关 `_delete_` 可以参靠[教程 1：如何编写配置文件](https://mmclassification.readthedocs.io/zh_CN/latest/tutorials/config.html#id16)。案列如下： | |||
mzr1996
added a commit
to mzr1996/mmpretrain
that referenced
this pull request
Nov 24, 2022
* add cn tutorials/config.md * add heads api and doc title link * Update tutorials index * Update tutorials index * Update config.md * add english version * Update config.md * add custom_runtime * Update docs * modify title * modify en to zh_CN in chinses docs * Update Readme * fix punctuations * Update docs/tutorials/customize_runtime.md Co-authored-by: Ma Zerun <mzr1996@163.com> * Update docs/tutorials/customize_runtime.md Co-authored-by: Ma Zerun <mzr1996@163.com> * split to schedule and runtime * fix lint * improve docs after review * fix TOC * imporve expersion * fix an error * Imporve schedule.md * Improve runtime.md * Improve chinese docs. * Fix toc-tree * fix en link and add a case of gradient clipping * fix wrong word Co-authored-by: Ma Zerun <mzr1996@163.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Add schedule and runtime tutorial documentation.
Modification
Checklist
Before PR:
After PR: