Refactor one-stage get_bboxes logic #5317

hhaAndroid · 2021-06-08T08:07:29Z

Motivation

In order to facilitate understanding and reconstruction, it is necessary to unify the one-stage get_bboxes process as much as possible.

Modification

Unified bbox decode process. Please refer to Unified FCOS and VFNet bbox coder #5374
Unify anchor and point generator. Please refer to Refactor anchor_generator and point_generator #5349
Use the sparse prior index to speed up inference.
Batch inference is only supported in ONNX. Use get_bbox_single under non-ONNX logic.

BC-Breaking

None

speed benchmark

envs: python mmdet/utils/collect_env.py

sys.platform: linux
Python: 3.8.10 (default, Jun  4 2021, 15:09:15) [GCC 7.5.0]
CUDA available: True
GPU 0: Tesla V100-PCIE-16GB
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.2.r11.2/compiler.29373293_0
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.8.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.2-Product Build 20210312 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=c
ompute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fop
enmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unk
nown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-s
tringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trappin
g-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL
=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.9.0
OpenCV: 4.5.3
MMCV: 1.3.9
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMDetection: 2.14.0+5f61347

before modification
python -m torch.distributed.launch --nproc_per_node=1 --master_port=29500 .dev_scripts/benchmark_inference_fps.py .dev_scripts/batch_test_list.py checkpoint --out fps_test --launcher pytorch

model	fps	times_pre_image(ms)
atss/atss_r50_fpn_1x_coco.py	13.3	75.3
autoassign/autoassign_r50_fpn_8x2_1x_coco.py	6.8	146.5
carafe/faster_rcnn_r50_fpn_carafe_1x_coco.py	12.9	77.6
cascade_rcnn/cascade_rcnn_r50_fpn_1x_coco.py	11.0	90.8
cascade_rcnn/cascade_mask_rcnn_r50_fpn_1x_coco.py	8.3	120.6
cascade_rpn/crpn_faster_rcnn_r50_caffe_fpn_1x_coco.py	11.8	84.5
centripetalnet/centripetalnet_hourglass104_mstest_16x6_210e_coco.py	3.8	264.5
dcn/faster_rcnn_r50_fpn_dconv_c3-c5_1x_coco.py	12.8	78.4
deformable_detr/deformable_detr_r50_16x2_50e_coco.py	10.2	98.4
detectors/detectors_htc_r50_1x_coco.py	3.1	323.7
detr/detr_r50_8x2_150e_coco.py	14.7	68.1
double_heads/dh_faster_rcnn_r50_fpn_1x_coco.py	8.0	125.7
dynamic_rcnn/dynamic_rcnn_r50_fpn_1x_coco.py	14.2	70.2
empirical_attention/faster_rcnn_r50_fpn_attention_1111_1x_coco.py	9.1	109.3
faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py	13.9	71.8
fcos/fcos_center-normbbox-centeronreg-giou_r50_caffe_fpn_gn-head_1x_coco.py	15.7	63.8
foveabox/fovea_align_r50_fpn_gn-head_4x4_2x_coco.py	13.7	72.8
free_anchor/retinanet_free_anchor_r50_fpn_1x_coco.py	11.1	90.2
fsaf/fsaf_r50_fpn_1x_coco.py	13.6	73.3
gcnet/mask_rcnn_r50_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco.py	9.9	100.6
gfl/gfl_r50_fpn_1x_coco.py	14.8	67.6
gn/mask_rcnn_r50_fpn_gn-all_2x_coco.py	10.8	92.9
gn+ws/faster_rcnn_r50_fpn_gn_ws-all_1x_coco.py	10.9	91.4
grid_rcnn/grid_rcnn_r50_fpn_gn-head_2x_coco.py	10.2	98.3
groie/faster_rcnn_r50_fpn_groie_1x_coco.py	10.0	100.3
guided_anchoring/ga_retinanet_r50_caffe_fpn_1x_coco.py	11.0	91.2
guided_anchoring/ga_faster_r50_caffe_fpn_1x_coco.py	10.5	94.9
hrnet/faster_rcnn_hrnetv2p_w18_1x_coco.py	7.0	143.2
htc/htc_r50_fpn_1x_coco.py	5.0	202.0
libra_rcnn/libra_faster_rcnn_r50_fpn_1x_coco.py	13.4	74.8
mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py	11.8	84.9
ms_rcnn/ms_rcnn_r50_caffe_fpn_1x_coco.py	11.8	85.0
nas_fcos/nas_fcos_nashead_r50_caffe_fpn_gn-head_4x4_1x_coco.py	9.2	108.5
nas_fpn/retinanet_r50_nasfpn_crop640_50e_coco.py	10.0	99.8
paa/paa_r50_fpn_1x_coco.py	2.8	356.2
pafpn/faster_rcnn_r50_pafpn_1x_coco.py	12.5	80.2
pisa/pisa_faster_rcnn_r50_fpn_1x_coco.py	12.9	77.3
point_rend/point_rend_r50_caffe_fpn_mstrain_1x_coco.py	8.6	116.1
regnet/mask_rcnn_regnetx-3.2GF_fpn_1x_coco.py	10.0	99.6
reppoints/reppoints_moment_r50_fpn_1x_coco.py	12.3	81.2
res2net/faster_rcnn_r2_101_fpn_2x_coco.py	8.6	116.6
resnest/faster_rcnn_s50_fpn_syncbn-backbone+head_mstrain-range_1x_coco.py	8.9	112.9
retinanet/retinanet_r50_fpn_1x_coco.py	12.8	78.0
rpn/rpn_r50_fpn_1x_coco.py	19.2	52.1
sabl/sabl_retinanet_r50_fpn_1x_coco.py	13.1	76.1
sabl/sabl_faster_rcnn_r50_fpn_1x_coco.py	12.7	78.8
scnet/scnet_r50_fpn_1x_coco.py	6.5	154.6
sparse_rcnn/sparse_rcnn_r50_fpn_1x_coco.py	12.5	79.8
tridentnet/tridentnet_r50_caffe_1x_coco.py	3.3	300.9
vfnet/vfnet_r50_fpn_1x_coco.py	12.2	81.9
yolact/yolact_r50_1x8_coco.py	19.5	51.2
yolo/yolov3_d53_320_273e_coco.py	28.7	34.9
yolof/yolof_r50_c5_8x8_1x_coco.py	16.5	60.8
cornernet/cornernet_hourglass104_mstest_8x6_210e_coco.py	4.5	223.9
ssd/ssd300_coco.py	34.4	29.1
centernet/centernet_resnet18_dcnv2_140e_coco.py	35	28.5

jshilong · 2021-07-15T09:29:51Z

I believe num_anchors as an attribute of anchor_heads to represent the number of base_anchors is not reasonable, because in gt_bboxes of most heads, It represents the number of all anchors instead of a number of base anchors, We need discuss the renaming of this attribute. @hhaAndroid @RangiLyu @ZwwWayne

jshilong · 2021-07-15T09:30:29Z

@jshilong Can we remove this after refactor the atss？ I don't think it can. The FCOS strategy is used in inference.

you are right

RangiLyu

Still many typos in docstring. There must be some omissions in this review. We need to check more carefully.

mmdet/models/dense_heads/atss_head.py

mmdet/models/dense_heads/base_dense_head.py

mmdet/models/dense_heads/paa_head.py

mmdet/models/dense_heads/reppoints_head.py

mmdet/models/dense_heads/rpn_head.py

hhaAndroid · 2021-07-22T02:00:05Z

Still many typos in docstring. There must be some omissions in this review. We need to check more carefully.

Thank you for your detailed review.

* revert batch to single * update anchor_head * replace preds with bboxes * add point_bbox_coder * FCOS add get_selected_priori * unified anchor-free and anchor-based get_bbox_single * update code * update reppoints and sabl * add sparse priors * add mlvlpointsgenerator * revert __init__ of core * refactor reppoints * delete label channal * add docstr * fix typo * fix args * fix typo * fix doc * fix stride_h * add offset * Unified bbox coder * add offset * remove point_bbox_coder.py * fix docstr * new interface of single_proir * fix device * add unitest * add cuda unitest * add more cuda unintest * fix reppoints * fix device * update all prior * update vfnet * add unintest for ssd and yolo and rename prior_idxs * add docstr for MlvlPointGenerator * update reppoints and rpnhead * add space * add num_base_priors * update some model * update docstr * fixAugFPN test and lint. * Fix autoassign * add docs * Unified fcos decoding * update docstr * fix train error * Fix Vfnet * Fix some * update centernet * revert * add warnings * fix unittest error * delete duplicated * fix comment * fix docs * fix type Co-authored-by: zhangshilong <2392587229zsl@gmail.com>

* Refactor one-stage get_bboxes logic (#5317) * revert batch to single * update anchor_head * replace preds with bboxes * add point_bbox_coder * FCOS add get_selected_priori * unified anchor-free and anchor-based get_bbox_single * update code * update reppoints and sabl * add sparse priors * add mlvlpointsgenerator * revert __init__ of core * refactor reppoints * delete label channal * add docstr * fix typo * fix args * fix typo * fix doc * fix stride_h * add offset * Unified bbox coder * add offset * remove point_bbox_coder.py * fix docstr * new interface of single_proir * fix device * add unitest * add cuda unitest * add more cuda unintest * fix reppoints * fix device * update all prior * update vfnet * add unintest for ssd and yolo and rename prior_idxs * add docstr for MlvlPointGenerator * update reppoints and rpnhead * add space * add num_base_priors * update some model * update docstr * fixAugFPN test and lint. * Fix autoassign * add docs * Unified fcos decoding * update docstr * fix train error * Fix Vfnet * Fix some * update centernet * revert * add warnings * fix unittest error * delete duplicated * fix comment * fix docs * fix type Co-authored-by: zhangshilong <2392587229zsl@gmail.com> * support onnx export for fcos * support onnx export for fcos fsaf retina and ssd * resolve comments * resolve comments * add with nms * support cornernet * resolve comments * add default with nms * fix trt arrange should be int Co-authored-by: Haian Huang(深度眸) <1286304229@qq.com>

* Refactor one-stage get_bboxes logic (#5317) * revert batch to single * update anchor_head * replace preds with bboxes * add point_bbox_coder * FCOS add get_selected_priori * unified anchor-free and anchor-based get_bbox_single * update code * update reppoints and sabl * add sparse priors * add mlvlpointsgenerator * revert __init__ of core * refactor reppoints * delete label channal * add docstr * fix typo * fix args * fix typo * fix doc * fix stride_h * add offset * Unified bbox coder * add offset * remove point_bbox_coder.py * fix docstr * new interface of single_proir * fix device * add unitest * add cuda unitest * add more cuda unintest * fix reppoints * fix device * update all prior * update vfnet * add unintest for ssd and yolo and rename prior_idxs * add docstr for MlvlPointGenerator * update reppoints and rpnhead * add space * add num_base_priors * update some model * update docstr * fixAugFPN test and lint. * Fix autoassign * add docs * Unified fcos decoding * update docstr * fix train error * Fix Vfnet * Fix some * update centernet * revert * add warnings * fix unittest error * delete duplicated * fix comment * fix docs * fix type Co-authored-by: zhangshilong <2392587229zsl@gmail.com> * support onnx export for fcos * support onnx export for fcos fsaf retina and ssd * resolve comments * resolve comments * add with nms * support cornernet * resolve comments * add default with nms * fix trt arrange should be int * refactor anchor head anchor free head * add dtype to single_level_grid_priors * atss fcos autoassign * fovea * fsaf free anchor * suport more * suport more * support all * resolve conversation * fix point generator * fix device * change to distancecoder * resolve conversation * fix grid prior * fix typos in autoassgin * fix typos * fix doc Co-authored-by: Haian Huang(深度眸) <1286304229@qq.com>