[Refactor]Refactor exporting One-Stage model to ONNX #6003

jshilong · 2021-09-01T08:49:44Z

Motivation

The recent ONNX-related development is too quickly, which made the code hard to read, and there is a problem that the return type of the same function such as _get_bboxes in the DenseHead.,

Modification

This PR moves all ONNX related code of the One-Stage model to a new function onnx_export in the corresponding class.

BC-breaking (Optional)

None

* revert batch to single * update anchor_head * replace preds with bboxes * add point_bbox_coder * FCOS add get_selected_priori * unified anchor-free and anchor-based get_bbox_single * update code * update reppoints and sabl * add sparse priors * add mlvlpointsgenerator * revert __init__ of core * refactor reppoints * delete label channal * add docstr * fix typo * fix args * fix typo * fix doc * fix stride_h * add offset * Unified bbox coder * add offset * remove point_bbox_coder.py * fix docstr * new interface of single_proir * fix device * add unitest * add cuda unitest * add more cuda unintest * fix reppoints * fix device * update all prior * update vfnet * add unintest for ssd and yolo and rename prior_idxs * add docstr for MlvlPointGenerator * update reppoints and rpnhead * add space * add num_base_priors * update some model * update docstr * fixAugFPN test and lint. * Fix autoassign * add docs * Unified fcos decoding * update docstr * fix train error * Fix Vfnet * Fix some * update centernet * revert * add warnings * fix unittest error * delete duplicated * fix comment * fix docs * fix type Co-authored-by: zhangshilong <2392587229zsl@gmail.com>

mmdet/core/anchor/anchor_generator.py

mmdet/core/anchor/point_generator.py

mmdet/models/detectors/single_stage.py

ZwwWayne · 2021-09-04T01:23:16Z

some remaining issues, like num_base_anchors should also be fixed with comments

jshilong · 2021-09-04T13:41:42Z

some remaining issues, like num_base_anchors should also be fixed with comments

This pr is still working in progress, any suggestion is appreciated

mmdet/core/anchor/anchor_generator.py

mmdet/models/dense_heads/base_dense_head.py

mmdet/core/anchor/point_generator.py

jshilong · 2021-09-09T12:50:29Z

some remaining issues, like num_base_anchors should also be fixed with comments

Renaming the attribute num_anchors to num_base_priors will affect the training, I suggest doing it in the future when all models change to prior_generator

ZwwWayne · 2021-09-10T09:05:44Z

some remaining issues, like num_base_anchors should also be fixed with comments

Renaming the attribute num_anchors to num_base_priors will affect the training, I suggest doing it in the future when all models change to prior_generator

Sure, we can do that in the next PR.

mmdet/core/export/onnx_helper.py

mmdet/models/detectors/single_stage.py

mmdet/models/dense_heads/base_dense_head.py

mmdet/models/dense_heads/yolo_head.py

jshilong · 2021-09-22T03:08:37Z

@jshilong Seems somewhere in fcos has used torch.arange with non-int64 input, which makes onnx2tensorrt failed.
configs/fcos/fcos_r50_caffe_fpn_gn-head_4x4_1x_coco.py
checkpoints/fcos/fcos_r50_caffe_fpn_gn-head_4x4_1x_coco_batch.onnx
--trt-file
checkpoints/fcos/fcos_r50_caffe_fpn_gn-head_4x4_1x_coco_batch.trt
--input-img
data/blueangels.jpg
--show
--verbose
--verify
--workspace-size 1
--max-shape 1344
--shape 400 600
[TensorRT] VERBOSE: ModelImporter.cpp:125: Range_675 [Range] inputs: [2512 -> ()], [1180 -> ()], [2513 -> ()],
Traceback (most recent call last):
File "/home/PJLAB/maningsheng/projects/openmmlab/mmdetection/tools/deployment/onnx2tensorrt.py", line 254, in
verbose=args.verbose)
File "/home/PJLAB/maningsheng/projects/openmmlab/mmdetection/tools/deployment/onnx2tensorrt.py", line 45, in onnx2tensorrt
max_workspace_size=max_workspace_size)
File "/home/PJLAB/maningsheng/projects/openmmlab/mmcv-pt1.8/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt
raise RuntimeError(f'parse onnx failed:\n{error_msgs}')
RuntimeError: parse onnx failed:
In node -1 (importRange): UNSUPPORTED_NODE: Assertion failed: inputs.at(0).isInt32() && "For range operator with dynamic inputs, this version of TensorRT only supports INT32!"

I will check it

jshilong · 2021-10-11T13:45:08Z

@jshilong Seems somewhere in fcos has used torch.arange with non-int64 input, which makes onnx2tensorrt failed.
configs/fcos/fcos_r50_caffe_fpn_gn-head_4x4_1x_coco.py
checkpoints/fcos/fcos_r50_caffe_fpn_gn-head_4x4_1x_coco_batch.onnx
--trt-file
checkpoints/fcos/fcos_r50_caffe_fpn_gn-head_4x4_1x_coco_batch.trt
--input-img
data/blueangels.jpg
--show
--verbose
--verify
--workspace-size 1
--max-shape 1344
--shape 400 600
[TensorRT] VERBOSE: ModelImporter.cpp:125: Range_675 [Range] inputs: [2512 -> ()], [1180 -> ()], [2513 -> ()],
Traceback (most recent call last):
File "/home/PJLAB/maningsheng/projects/openmmlab/mmdetection/tools/deployment/onnx2tensorrt.py", line 254, in
verbose=args.verbose)
File "/home/PJLAB/maningsheng/projects/openmmlab/mmdetection/tools/deployment/onnx2tensorrt.py", line 45, in onnx2tensorrt
max_workspace_size=max_workspace_size)
File "/home/PJLAB/maningsheng/projects/openmmlab/mmcv-pt1.8/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt
raise RuntimeError(f'parse onnx failed:\n{error_msgs}')
RuntimeError: parse onnx failed:
In node -1 (importRange): UNSUPPORTED_NODE: Assertion failed: inputs.at(0).isInt32() && "For range operator with dynamic inputs, this version of TensorRT only supports INT32!"

Would you mind helping to retest it? I may have fixed it in point_generator

VVsssssk · 2021-10-12T03:52:20Z

Hello,When I have test this pr,I tranform python2onnx,all model is successed.But when I tranform onnx2trt,only fcos success.

ERR LOG:
(pt1.8) PJLAB\shenkun@shai14001070l:~/workspace/mmdetection$ python tools/deployment/onnx2tensorrt.py configs/fsaf/fsaf_r50_fpn_1x_coco.py tmp/fsaf.onnx --trt-file='fsaf.trt' --input-img='tests/data/color.jpg' --shape 800 1216tools/deployment/onnx2tensorrt.py:199: UserWarning: Arguments like --to-rgb, --mean, --std, --dataset would be parsed directly from config file and are deprecated and will be removed in future releases.
warnings.warn(
Traceback (most recent call last):
File "tools/deployment/onnx2tensorrt.py", line 247, in
onnx2tensorrt(
File "tools/deployment/onnx2tensorrt.py", line 40, in onnx2tensorrt
trt_engine = onnx2trt(
File "/home/PJLAB/shenkun/workspace/mmcv/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt
raise RuntimeError(f'parse onnx failed:\n{error_msgs}')
RuntimeError: parse onnx failed:
In node -1 (convertAxis): UNSUPPORTED_NODE: Assertion failed: axis >= 0 && axis < nbDims

(pt1.8) PJLAB\shenkun@shai14001070l:~/workspace/mmdetection$ python tools/deployment/onnx2tensorrt.py configs/retinanet/retinanet_r50_fpn_1x_coco.py tmp/retinanet.onnx --trt-file='retinanet.trt' --input-img='tests/data/color.jpg' --shape 800 1216
tools/deployment/onnx2tensorrt.py:199: UserWarning: Arguments like --to-rgb, --mean, --std, --dataset would be parsed directly from config file and are deprecated and will be removed in future releases.
warnings.warn(
Traceback (most recent call last):
File "tools/deployment/onnx2tensorrt.py", line 247, in
onnx2tensorrt(
File "tools/deployment/onnx2tensorrt.py", line 40, in onnx2tensorrt
trt_engine = onnx2trt(
File "/home/PJLAB/shenkun/workspace/mmcv/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt
raise RuntimeError(f'parse onnx failed:\n{error_msgs}')
RuntimeError: parse onnx failed:
In node -1 (convertAxis): UNSUPPORTED_NODE: Assertion failed: axis >= 0 && axis < nbDims

(pt1.8) PJLAB\shenkun@shai14001070l:~/workspace/mmdetection$ python tools/deployment/onnx2tensorrt.py configs/ssd/ssd300_coco.py tmp/ssd.onnx --trt-file='ssd.trt' --input-img='tests/data/color.jpg' --shape 800 1216tools/deployment/onnx2tensorrt.py:199: UserWarning: Arguments like --to-rgb, --mean, --std, --dataset would be parsed directly from config file and are deprecated and will be removed in future releases.
warnings.warn(
Traceback (most recent call last):
File "tools/deployment/onnx2tensorrt.py", line 247, in
onnx2tensorrt(
File "tools/deployment/onnx2tensorrt.py", line 40, in onnx2tensorrt
trt_engine = onnx2trt(
File "/home/PJLAB/shenkun/workspace/mmcv/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt
raise RuntimeError(f'parse onnx failed:\n{error_msgs}')
RuntimeError: parse onnx failed:
In node -1 (convertAxis): UNSUPPORTED_NODE: Assertion failed: axis >= 0 && axis < nbDims

(pt1.8) PJLAB\shenkun@shai14001070l:~/workspace/mmdetection$ python tools/deployment/onnx2tensorrt.py configs/yolo/yolov3_d53_320_273e_coco.py tmp/yolov3.onnx --trt-file='yolov3.trt' --input-img='tests/data/color.jpg' --shape 800 1216tools/deployment/onnx2tensorrt.py:199: UserWarning: Arguments like --to-rgb, --mean, --std, --dataset would be parsed directly from config file and are deprecated and will be removed in future releases.
warnings.warn(
Traceback (most recent call last):
File "tools/deployment/onnx2tensorrt.py", line 247, in
onnx2tensorrt(
File "tools/deployment/onnx2tensorrt.py", line 40, in onnx2tensorrt
trt_engine = onnx2trt(
File "/home/PJLAB/shenkun/workspace/mmcv/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt
raise RuntimeError(f'parse onnx failed:\n{error_msgs}')
RuntimeError: parse onnx failed:
In node -1 (convertAxis): UNSUPPORTED_NODE: Assertion failed: axis >= 0 && axis < nbDims

jshilong · 2021-10-13T03:24:54Z

Hello,When I have test this pr,I tranform python2onnx,all model is successed.But when I tranform onnx2trt,only fcos success.

ERR LOG: (pt1.8) PJLAB\shenkun@shai14001070l:~/workspace/mmdetection$ python tools/deployment/onnx2tensorrt.py configs/fsaf/fsaf_r50_fpn_1x_coco.py tmp/fsaf.onnx --trt-file='fsaf.trt' --input-img='tests/data/color.jpg' --shape 800 1216tools/deployment/onnx2tensorrt.py:199: UserWarning: Arguments like --to-rgb, --mean, --std, --dataset would be parsed directly from config file and are deprecated and will be removed in future releases. warnings.warn( Traceback (most recent call last): File "tools/deployment/onnx2tensorrt.py", line 247, in onnx2tensorrt( File "tools/deployment/onnx2tensorrt.py", line 40, in onnx2tensorrt trt_engine = onnx2trt( File "/home/PJLAB/shenkun/workspace/mmcv/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt raise RuntimeError(f'parse onnx failed:\n{error_msgs}') RuntimeError: parse onnx failed: In node -1 (convertAxis): UNSUPPORTED_NODE: Assertion failed: axis >= 0 && axis < nbDims

(pt1.8) PJLAB\shenkun@shai14001070l:~/workspace/mmdetection$ python tools/deployment/onnx2tensorrt.py configs/retinanet/retinanet_r50_fpn_1x_coco.py tmp/retinanet.onnx --trt-file='retinanet.trt' --input-img='tests/data/color.jpg' --shape 800 1216 tools/deployment/onnx2tensorrt.py:199: UserWarning: Arguments like --to-rgb, --mean, --std, --dataset would be parsed directly from config file and are deprecated and will be removed in future releases. warnings.warn( Traceback (most recent call last): File "tools/deployment/onnx2tensorrt.py", line 247, in onnx2tensorrt( File "tools/deployment/onnx2tensorrt.py", line 40, in onnx2tensorrt trt_engine = onnx2trt( File "/home/PJLAB/shenkun/workspace/mmcv/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt raise RuntimeError(f'parse onnx failed:\n{error_msgs}') RuntimeError: parse onnx failed: In node -1 (convertAxis): UNSUPPORTED_NODE: Assertion failed: axis >= 0 && axis < nbDims

(pt1.8) PJLAB\shenkun@shai14001070l:~/workspace/mmdetection$ python tools/deployment/onnx2tensorrt.py configs/ssd/ssd300_coco.py tmp/ssd.onnx --trt-file='ssd.trt' --input-img='tests/data/color.jpg' --shape 800 1216tools/deployment/onnx2tensorrt.py:199: UserWarning: Arguments like --to-rgb, --mean, --std, --dataset would be parsed directly from config file and are deprecated and will be removed in future releases. warnings.warn( Traceback (most recent call last): File "tools/deployment/onnx2tensorrt.py", line 247, in onnx2tensorrt( File "tools/deployment/onnx2tensorrt.py", line 40, in onnx2tensorrt trt_engine = onnx2trt( File "/home/PJLAB/shenkun/workspace/mmcv/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt raise RuntimeError(f'parse onnx failed:\n{error_msgs}') RuntimeError: parse onnx failed: In node -1 (convertAxis): UNSUPPORTED_NODE: Assertion failed: axis >= 0 && axis < nbDims

(pt1.8) PJLAB\shenkun@shai14001070l:~/workspace/mmdetection$ python tools/deployment/onnx2tensorrt.py configs/yolo/yolov3_d53_320_273e_coco.py tmp/yolov3.onnx --trt-file='yolov3.trt' --input-img='tests/data/color.jpg' --shape 800 1216tools/deployment/onnx2tensorrt.py:199: UserWarning: Arguments like --to-rgb, --mean, --std, --dataset would be parsed directly from config file and are deprecated and will be removed in future releases. warnings.warn( Traceback (most recent call last): File "tools/deployment/onnx2tensorrt.py", line 247, in onnx2tensorrt( File "tools/deployment/onnx2tensorrt.py", line 40, in onnx2tensorrt trt_engine = onnx2trt( File "/home/PJLAB/shenkun/workspace/mmcv/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt raise RuntimeError(f'parse onnx failed:\n{error_msgs}') RuntimeError: parse onnx failed: In node -1 (convertAxis): UNSUPPORTED_NODE: Assertion failed: axis >= 0 && axis < nbDims

This PR works fine under PyTorch 1.6, This problem only appears in the higher version PyTorch.

* Refactor one-stage get_bboxes logic (#5317) * revert batch to single * update anchor_head * replace preds with bboxes * add point_bbox_coder * FCOS add get_selected_priori * unified anchor-free and anchor-based get_bbox_single * update code * update reppoints and sabl * add sparse priors * add mlvlpointsgenerator * revert __init__ of core * refactor reppoints * delete label channal * add docstr * fix typo * fix args * fix typo * fix doc * fix stride_h * add offset * Unified bbox coder * add offset * remove point_bbox_coder.py * fix docstr * new interface of single_proir * fix device * add unitest * add cuda unitest * add more cuda unintest * fix reppoints * fix device * update all prior * update vfnet * add unintest for ssd and yolo and rename prior_idxs * add docstr for MlvlPointGenerator * update reppoints and rpnhead * add space * add num_base_priors * update some model * update docstr * fixAugFPN test and lint. * Fix autoassign * add docs * Unified fcos decoding * update docstr * fix train error * Fix Vfnet * Fix some * update centernet * revert * add warnings * fix unittest error * delete duplicated * fix comment * fix docs * fix type Co-authored-by: zhangshilong <2392587229zsl@gmail.com> * support onnx export for fcos * support onnx export for fcos fsaf retina and ssd * resolve comments * resolve comments * add with nms * support cornernet * resolve comments * add default with nms * fix trt arrange should be int Co-authored-by: Haian Huang(深度眸) <1286304229@qq.com>

* Refactor one-stage get_bboxes logic (open-mmlab#5317) * revert batch to single * update anchor_head * replace preds with bboxes * add point_bbox_coder * FCOS add get_selected_priori * unified anchor-free and anchor-based get_bbox_single * update code * update reppoints and sabl * add sparse priors * add mlvlpointsgenerator * revert __init__ of core * refactor reppoints * delete label channal * add docstr * fix typo * fix args * fix typo * fix doc * fix stride_h * add offset * Unified bbox coder * add offset * remove point_bbox_coder.py * fix docstr * new interface of single_proir * fix device * add unitest * add cuda unitest * add more cuda unintest * fix reppoints * fix device * update all prior * update vfnet * add unintest for ssd and yolo and rename prior_idxs * add docstr for MlvlPointGenerator * update reppoints and rpnhead * add space * add num_base_priors * update some model * update docstr * fixAugFPN test and lint. * Fix autoassign * add docs * Unified fcos decoding * update docstr * fix train error * Fix Vfnet * Fix some * update centernet * revert * add warnings * fix unittest error * delete duplicated * fix comment * fix docs * fix type Co-authored-by: zhangshilong <2392587229zsl@gmail.com> * support onnx export for fcos * support onnx export for fcos fsaf retina and ssd * resolve comments * resolve comments * add with nms * support cornernet * resolve comments * add default with nms * fix trt arrange should be int Co-authored-by: Haian Huang(深度眸) <1286304229@qq.com>

jshilong changed the base branch from master to refactor_dense September 1, 2021 08:50

pull master

38c0ecf

jshilong added the refactor label Sep 1, 2021

jshilong changed the title ~~Onestage onnx~~ [Refactor]Refactor exporting One-Stage model to ONNX Sep 1, 2021

jshilong added the WIP Working in progress label Sep 1, 2021

ZwwWayne reviewed Sep 4, 2021

View reviewed changes

RunningLeon self-requested a review September 6, 2021 03:15

RunningLeon reviewed Sep 6, 2021

View reviewed changes

mmdet/core/anchor/anchor_generator.py Show resolved Hide resolved

ZwwWayne requested a review from RunningLeon September 7, 2021 09:43

RunningLeon reviewed Sep 7, 2021

View reviewed changes

mmdet/models/dense_heads/base_dense_head.py Outdated Show resolved Hide resolved

RunningLeon reviewed Sep 7, 2021

View reviewed changes

mmdet/models/dense_heads/base_dense_head.py Outdated Show resolved Hide resolved

RunningLeon reviewed Sep 7, 2021

View reviewed changes

mmdet/models/dense_heads/base_dense_head.py Outdated Show resolved Hide resolved

RunningLeon reviewed Sep 7, 2021

View reviewed changes

mmdet/models/dense_heads/base_dense_head.py Outdated Show resolved Hide resolved

RunningLeon reviewed Sep 8, 2021

View reviewed changes

mmdet/core/anchor/point_generator.py Outdated Show resolved Hide resolved

jshilong requested review from RunningLeon and ZwwWayne September 9, 2021 12:45

ZwwWayne reviewed Sep 10, 2021

View reviewed changes

mmdet/core/export/onnx_helper.py Show resolved Hide resolved

ZwwWayne reviewed Sep 10, 2021

View reviewed changes

mmdet/models/detectors/single_stage.py Outdated Show resolved Hide resolved

jshilong requested review from ZwwWayne, hhaAndroid and RangiLyu September 13, 2021 07:23

ZwwWayne approved these changes Sep 13, 2021

View reviewed changes

RangiLyu mentioned this pull request Sep 13, 2021

fix yolox to onnx #6042

Closed

RunningLeon reviewed Sep 14, 2021

View reviewed changes

mmdet/models/dense_heads/base_dense_head.py Show resolved Hide resolved

RangiLyu reviewed Sep 14, 2021

View reviewed changes

mmdet/models/dense_heads/yolo_head.py Outdated Show resolved Hide resolved

jshilong force-pushed the refactor_dense branch from 38c0ecf to 304150d Compare September 26, 2021 09:39

jshilong added 8 commits October 8, 2021 14:51

support onnx export for fcos

b6f5700

support onnx export for fcos fsaf retina and ssd

032ae46

resolve comments

21e5b34

resolve comments

9903e59

add with nms

94ee765

support cornernet

8570f4c

resolve comments

b42f395

add default with nms

a377b13

jshilong force-pushed the refactor_dense branch from 304150d to e114eac Compare October 11, 2021 06:39

jshilong force-pushed the onestage_onnx branch 2 times, most recently from 5c778b7 to 0829b33 Compare October 11, 2021 07:40

pull refactor dense

59a4fb6

jshilong force-pushed the onestage_onnx branch from 0829b33 to 59a4fb6 Compare October 11, 2021 07:48

fix trt arrange should be int

7e56e59

jshilong requested review from RangiLyu, RunningLeon and ZwwWayne October 11, 2021 13:43

ZwwWayne approved these changes Oct 13, 2021

View reviewed changes

ZwwWayne merged commit c5a7b08 into open-mmlab:refactor_dense Oct 13, 2021

RangiLyu mentioned this pull request Oct 14, 2021

Iteration Plan of v2.18.0 - October 2021 #6281

Closed

16 tasks

ZwwWayne mentioned this pull request Oct 27, 2021

Bump version to v2.18.0 #6365

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Refactor]Refactor exporting One-Stage model to ONNX #6003

[Refactor]Refactor exporting One-Stage model to ONNX #6003

jshilong commented Sep 1, 2021 •

edited

Loading

ZwwWayne commented Sep 4, 2021

jshilong commented Sep 4, 2021 •

edited

Loading

jshilong commented Sep 9, 2021

ZwwWayne commented Sep 10, 2021

jshilong commented Sep 22, 2021

jshilong commented Oct 11, 2021

VVsssssk commented Oct 12, 2021 •

edited

Loading

jshilong commented Oct 13, 2021

[Refactor]Refactor exporting One-Stage model to ONNX #6003

[Refactor]Refactor exporting One-Stage model to ONNX #6003

Conversation

jshilong commented Sep 1, 2021 • edited Loading

Motivation

Modification

BC-breaking (Optional)

ZwwWayne commented Sep 4, 2021

jshilong commented Sep 4, 2021 • edited Loading

jshilong commented Sep 9, 2021

ZwwWayne commented Sep 10, 2021

jshilong commented Sep 22, 2021

jshilong commented Oct 11, 2021

VVsssssk commented Oct 12, 2021 • edited Loading

jshilong commented Oct 13, 2021

jshilong commented Sep 1, 2021 •

edited

Loading

jshilong commented Sep 4, 2021 •

edited

Loading

VVsssssk commented Oct 12, 2021 •

edited

Loading