Skip to content
This repository has been archived by the owner on Oct 31, 2023. It is now read-only.

Lower accuracy of pretrained model in model zoo.... #1051

Open
Wangt-CN opened this issue Aug 19, 2019 · 9 comments
Open

Lower accuracy of pretrained model in model zoo.... #1051

Wangt-CN opened this issue Aug 19, 2019 · 9 comments
Assignees
@LeviViana

Comments

@Wangt-CN
Copy link

Wangt-CN commented Aug 19, 2019 •
edited
Loading

❓ Questions and Help

When I use the /configs/2e_mask_rcnn_R_101_FPN_1x.yaml and /configs/caffe2/2e_mask_rcnn_R_101_FPN_1x.yaml to evaluate the model (I didn't change any code in the mask-rcnn benchmark), the results accuracy is 2~3% lower than your report in model zoo.

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.385

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.597
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.418
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.208
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.418
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.514
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.316
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.493
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.516
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.313
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.556
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.659
Loading and preparing results...
DONE (t=1.69s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type segm
DONE (t=27.56s).
Accumulating evaluation results...
DONE (t=3.49s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.346
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.560
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.367
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.147
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.371
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.520
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.294
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.447
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.466
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.255
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.505
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.632
2019-08-19 08:43:41,982 maskrcnn_benchmark.inference INFO:
Task: bbox
AP, AP50, AP75, APs, APm, APl
0.3846, 0.5967, 0.4184, 0.2077, 0.4183, 0.5139
Task: segm
AP, AP50, AP75, APs, APm, APl
0.3464, 0.5595, 0.3668, 0.1475, 0.3708, 0.5204

The accuracy you report in model zoo is about 40.1 and 36.1.
I'm very confused, and I just run
python -m torch.distributed.launch --nproc_per_node=4 tools/test_net.py --config-file "/data1/48data/maskrcnn-benchmark/configs/e2e_mask_rcnn_R_101_FPN_1x.yaml" TEST.IMS_PER_BATCH 8
(I just have 4 GPUs so I change TEST.IMS_PER_BATCH = 8)
@Wangt-CN
Copy link
Author

And when use the pretrained mask-rcnn R-101 model in /configs/caffe2, the result is quite same and lower than report

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.386
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.599
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.422
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.206
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.421
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.519
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.315
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.492
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.515
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.305
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.557
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.670
Loading and preparing results...
DONE (t=1.73s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type segm
DONE (t=28.32s).
Accumulating evaluation results...
DONE (t=3.54s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.345
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.564
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.365
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.142
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.372
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.527
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.292
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.444
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.462
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.242
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.506
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.639
2019-08-19 09:23:39,267 maskrcnn_benchmark.inference INFO:
Task: bbox
AP, AP50, AP75, APs, APm, APl
0.3861, 0.5995, 0.4221, 0.2055, 0.4214, 0.5187
Task: segm
AP, AP50, AP75, APs, APm, APl
0.3449, 0.5636, 0.3649, 0.1417, 0.3724, 0.5268

Thanks for your attention
@fmassa

@LeviViana
Copy link
Contributor

It is maybe related to #672. Maybe this comment can help you get the reported performance.

@Wangt-CN
Copy link
Author

It is maybe related to #672. Maybe this comment can help you get the reported performance.

Thank you very much for your reply. However, I have noticed that comment and it's used for training when not using 8 GPUs.

However, in my problem I just use the pretrain model in the model zoo to do the inference (evaluation). Moreover, the image pre GPU is still 2 during inference, so maybe I needn't to change the _C.MODEL.RPN.FPN_POST_NMS_TOP_N_TEST = 2000 in config defaults profile I think?

@Wangt-CN
Copy link
Author

Wangt-CN commented Aug 19, 2019 •
edited
Loading

And here is the yaml file in the /configs/caffe2 and /configs
And you can see that FPN_POST_NMS_TOP_N_TEST: 2000 bacause there are 2 images per GPU

MODEL:
META_ARCHITECTURE: "GeneralizedRCNN"
WEIGHT: "catalog://Caffe2Detectron/COCO/35861795/e2e_mask_rcnn_R-101-FPN_1x"
BACKBONE:
CONV_BODY: "R-101-FPN"
RESNETS:
BACKBONE_OUT_CHANNELS: 256
RPN:
USE_FPN: True
ANCHOR_STRIDE: (4, 8, 16, 32, 64)
PRE_NMS_TOP_N_TRAIN: 2000
PRE_NMS_TOP_N_TEST: 1000
POST_NMS_TOP_N_TEST: 1000
FPN_POST_NMS_TOP_N_TEST: 2000
ROI_HEADS:
USE_FPN: True
ROI_BOX_HEAD:
POOLER_RESOLUTION: 7
POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)
POOLER_SAMPLING_RATIO: 2
FEATURE_EXTRACTOR: "FPN2MLPFeatureExtractor"
PREDICTOR: "FPNPredictor"
ROI_MASK_HEAD:
POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)
FEATURE_EXTRACTOR: "MaskRCNNFPNFeatureExtractor"
PREDICTOR: "MaskRCNNC4Predictor"
POOLER_RESOLUTION: 14
POOLER_SAMPLING_RATIO: 2
RESOLUTION: 28
SHARE_BOX_FEATURE_EXTRACTOR: False
MASK_ON: True
DATASETS:
TEST: ("coco_2014_minival",)
DATALOADER:
SIZE_DIVISIBILITY: 32

MODEL:
META_ARCHITECTURE: "GeneralizedRCNN"
WEIGHT: "/data1/48data/wt/vc-rcnn-py1.0/maskrcnn-benchmark/pretrained/e2e_mask_rcnn_R_101_FPN_1x.pth"
BACKBONE:
CONV_BODY: "R-101-FPN"
RESNETS:
BACKBONE_OUT_CHANNELS: 256
RPN:
USE_FPN: True
ANCHOR_STRIDE: (4, 8, 16, 32, 64)
PRE_NMS_TOP_N_TRAIN: 2000
PRE_NMS_TOP_N_TEST: 1000
POST_NMS_TOP_N_TEST: 1000
FPN_POST_NMS_TOP_N_TEST: 2000
ROI_HEADS:
USE_FPN: True
ROI_BOX_HEAD:
POOLER_RESOLUTION: 7
POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)
POOLER_SAMPLING_RATIO: 2
FEATURE_EXTRACTOR: "FPN2MLPFeatureExtractor"
PREDICTOR: "FPNPredictor"
ROI_MASK_HEAD:
POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)
FEATURE_EXTRACTOR: "MaskRCNNFPNFeatureExtractor"
PREDICTOR: "MaskRCNNC4Predictor"
POOLER_RESOLUTION: 14
POOLER_SAMPLING_RATIO: 2
RESOLUTION: 28
SHARE_BOX_FEATURE_EXTRACTOR: False
MASK_ON: True
DATASETS:
TRAIN: ("coco_2014_train", "coco_2014_valminusminival")
TEST: ("coco_2014_minival",)
DATALOADER:
SIZE_DIVISIBILITY: 32
SOLVER:
BASE_LR: 0.02
WEIGHT_DECAY: 0.0001
STEPS: (60000, 80000)
MAX_ITER: 90000`

I am quitely confused and I don't know where is the problem

@Wangt-CN
Copy link
Author

-----Updated-----

The result of another pretrained model : configs/caffe2/e2e_mask_rcnn_R_50_FPN_1x_caffe2 :

AP, AP50, AP75, APs, APm, APl
0.3614, 0.5716, 0.3910, 0.1921, 0.3921, 0.4858
Task: segm
AP, AP50, AP75, APs, APm, APl
0.3237, 0.5377, 0.3408, 0.1323, 0.3482, 0.4956

is still lower than the report in model zoo....

@LeviViana
Copy link
Contributor

Sorry, I read the issue too quickly. Indeed, this is weird I'm gonna check it in some environments later, I'll keep you informed.

@LeviViana LeviViana self-assigned this Aug 19, 2019
@Wangt-CN
Copy link
Author

Sorry, I read the issue too quickly. Indeed, this is weird I'm gonna check it in some environments later, I'll keep you informed.

Very appreciate for informing~ Thanks

@chenjoya
Copy link
Contributor

chenjoya commented Oct 5, 2019

Same problem here.
Mask R-CNN R-50-FPN:
Task: bbox
AP, AP50, AP75, APs, APm, APl
0.3677, 0.5784, 0.4006, 0.2126, 0.3963, 0.4763
Task: segm
AP, AP50, AP75, APs, APm, APl
0.3384, 0.5498, 0.3628, 0.1533, 0.3632, 0.4965

much lower than 37.8 box AP in model_zoo reported.

@ScXfjiang
Copy link

ScXfjiang commented Jan 9, 2020 •
edited
Loading

Same problem. The model is MaskRCNN R-50-FPN. I got 35.1% mAP for object detection mission, and 32.3% mAP for segmentation mission which should be 37.8% and 34.2%.

@kunwar31 kunwar31 mentioned this issue Jun 3, 2023
Merged
14 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@LeviViana LeviViana

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ScXfjiang @LeviViana @chenjoya @Wangt-CN