AP未达到论文中报告的结果

[Cathy1900]

你好，我在AI-TOD的test set上运行了detectors_cascade_rcnn_r50_aitod_rpn_nwd.py，但性能未达到论文中报告的精度（20.8 AP）。
我对detectors_cascade_rcnn_r50_aitod_rpn_nwd.py进行了以下修改：
1）在训练时，将detectors_cascade_rcnn_r50_aitod_rpn_nwd修改为在两个GPU上运行，每个GPU上运行4张图片，保持batch size为8。

2）在推理时，修改用于推理的图像和label路径：
ann_file='data/AI-TOD/annotations/aitod_test_v1_1.0.json',
img_prefix='data/AI-TOD/test/',

使用以下指令得到test set的性能：
python tools/test.py work_dirs/nwd/detectors_cascade_rcnn_r50_aitod_rpn_nwd/detectors_cascade_rcnn_r50_aitod_rpn_nwd.py work_dirs/nwd/detectors_cascade_rcnn_r50_aitod_rpn_nwd/epoch_12.pth --eval bbox

3）具体的AP性能如下：
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 14018/14018, 2.3 task/s, elapsed: 6169s, ETA: 0s
Evaluating bbox...
Loading and preparing results...
DONE (t=6.76s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=2665.67s).
Accumulating evaluation results...
DONE (t=29.74s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=1500 ] = 0.187
Average Precision (AP) @[ IoU=0.25 | area= all | maxDets=1500 ] = -1.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1500 ] = 0.451
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1500 ] = 0.126
Average Precision (AP) @[ IoU=0.50:0.95 | area=verytiny | maxDets=1500 ] = 0.041
Average Precision (AP) @[ IoU=0.50:0.95 | area= tiny | maxDets=1500 ] = 0.175
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1500 ] = 0.266
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1500 ] = 0.352
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.283
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.300
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1500 ] = 0.303
Average Recall (AR) @[ IoU=0.50:0.95 | area=verytiny | maxDets=1500 ] = 0.054
Average Recall (AR) @[ IoU=0.50:0.95 | area= tiny | maxDets=1500 ] = 0.317
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1500 ] = 0.383
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1500 ] = 0.424
Optimal LRP @[ IoU=0.50 | area= all | maxDets=1500 ] = -1.000
Optimal LRP Loc @[ IoU=0.50 | area= all | maxDets=1500 ] = -1.000
Optimal LRP FP @[ IoU=0.50 | area= all | maxDets=1500 ] = -1.000
Optimal LRP FN @[ IoU=0.50 | area= all | maxDets=1500 ] = -1.000
#Class-specific LRP-Optimal Thresholds #
[-1. -1. -1. -1. -1. -1. -1. -1.]

4）配置环境如下：
Python: 3.7.15 (default, Nov 24 2022, 21:12:53) [GCC 11.2.0]
CUDA available: True
GPU 0,1: NVIDIA A100 80GB PCIe
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.1.TC455_06.29069683_0
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PyTorch: 1.10.0+cu111
PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• LAPACK is enabled (usually provided by MKL)
• NNPACK is enabled
• CPU capability usage: AVX512
• CUDA Runtime 11.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
• CuDNN 8.0.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5,
  TorchVision: 0.11.0+cu111
  OpenCV: 4.6.0
  MMCV: 1.3.5
  MMCV Compiler: GCC 9.4
  MMCV CUDA Compiler: 11.1
  MMDetection: 2.13.0+

请问产生这种情况的原因是什么？期待您的回复，谢谢！

[Chasel-Tsui]

能否提供完整的config呢？训练集使用的是train set还是trainval set？

[Cathy1900]

能否提供完整的config呢？训练集使用的是train set还是trainval set？

这是data和优化器的设置，除了改samples_per_gpu，其余没动过了。
data = dict(
samples_per_gpu=4,
workers_per_gpu=2,
train=dict(
type='AITODDataset',
ann_file='data/AI-TOD/annotations/aitod_training_v1.0.json',
img_prefix='data/AI-TOD/trainval/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='Resize', img_scale=(800, 800), keep_ratio=True),
dict(type='RandomFlip', flip_ratio=0.5),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
]),
val=dict(
type='AITODDataset',
ann_file='data/AI-TOD/annotations/aitod_validation_v1.0.json',
img_prefix='data/AI-TOD/trainval/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(800, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]),
test=dict(
type='AITODDataset',
ann_file='data/AI-TOD/annotations/aitod_test_v1_1.0.json',
img_prefix='data/AI-TOD/test/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(800, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]))
evaluation = dict(interval=12, metric='bbox')
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=2000,
warmup_ratio=0.001,
step=[8, 11])
runner = dict(type='EpochBasedRunner', max_epochs=12)

[Chasel-Tsui]

似乎是因为训练只用了train set的标签，论文里的结果是用trainval set训练得到的，config中虽然图片路径是trainval，但是标签文件可能只是train set，onedrive公开的标签中包含了trainval set的json文件

[Cathy1900]

似乎是因为训练只用了train set的标签，论文里的结果是用trainval set训练得到的，config中虽然图片路径是trainval，但是标签文件可能只是train set，onedrive公开的标签中包含了trainval set的json文件

好的，谢谢您的回复，我再调整下