Change COCOEvaluator from AP50 to custom AP Score (like 30)?

[dinahrabe]

Hello, I would like to evaluate my object detection model on a lower AP score than the default values of the coco metrics, which give AP50, AP75 and the AP0.5:0.95. Specifically I would like to set the AP score to 30. Can someone help me how to do this? I can't find the spot for modification in the detectron2 documentation. Thanks a lot!

[bmaxdk]

Yes the default AP calculations include AP50, AP75 which are set within the COCO. You want AP30 then you may want to evaluate your model at an IoU of 0.3. You probably want to change the IoU thresholds that the COCO evaluator in Detection2 uses. Have you checked COCO Evaluator? You may want to modify this IoU thresholds to AP30 (0.3)

[dinahrabe]

Thanks, Of course I have checked COCO Evaluator. But it doesn't seem to be that easy to change the IoU threshold. Therefore my question here. Any hints on how deep into the model setup I would need to dig into?

[bmaxdk]

In the COCO Evaluator, this section of lines of code appeared in line 549 to line 541

if thresholds is None:
    step = 0.05
    thresholds = torch.arange(0.5, 0.95 + 1e-5, step, dtype=torch.float32)

to

if thresholds is None:
    thresholds = torch.tensor([0.3], dtype=torch.float32)  # AP30

If you are modifying this function directly in the Detectron2 source code, you will also need to ensure that the modified version of the function is actually being used in your main code.

If this function (_evaluate_box_proposals) is being called from another part of the code, make sure to either: Pass the custom thresholds directly when calling this function. Or ensure that the modified function is being utilized.

My understanding of your questions is simply changing default AP50 to custom AP Score like AP30. Typically, changing the thresholds would be enough if this function is directly responsible for the evaluation metrics you are concerned about. But you should verify that the function is being used as expected in the evaluation pipeline.
Will this answered your questions?? If not sorry that I couldn't help

[samaymaini]

Did this work for you? I am facing the same issue.