Pretrained Weight yolov3.pt and Optimizing Parameters for Inference

[takehiro-code]

❔Question

I have a question regarding the information of yolov3.pt and how I can obtain the optimal parameters (--conf-thres, --iou-thres, --img-size).

First of all, for the pre-trained weight of yolov3.pt, I believe it is trained in 2017 COCO dataset with 640x640 image size if I am correct (although I am not sure why it scaled to different width-heigh ratio from the input image e.g. 1920x1080 to 384x640)?

I am trying to optimize 3 of the parameters in detect.py on my custom sequence of data based on maximizing mAP. First of all, I believe for the pre-trained weight, --img-size of 640 is optimal since I believe pre-trained yolov3.pt model was trained with the dimension of 640x640 and hence the detection performance is optimized for that resolution if I am correct? For confidence threshold and iou threshold (--conf-thres, --iou-thres), I am not sure which parameters to choose, and the default parameters are 0.25, 0.45 respectively but I was unsure if those values are already optimal values for the pretrained weight, otherwise I am thinking to find values by for example for loop (iteratively) to find the best values. I hope that my question is logical and I really appreciate any suggestion or clarification. Thank you!

Additional context

I am making inference on my own custom sequence of images using detect.py with the pretrained weight of yolov3.pt

[glenn-jocher]

@takehiro-code you should train and detect at similar --img-size for best results.

Exact values for these settings, including --iou-thresh and --conf will vary based upon your dataset, domain, requirements, deployment conditions etc, and are up to you to tune.

To learn about confidence and iou thresholds in general you might try our iDetection app, which allows you to vary these in realtime and observe the effects instantly :)
https://apps.apple.com/app/id1452689527

[takehiro-code]

@glenn-jocher Thank you for the answer. I understand that those 3 parameters vary depending on different datasets.

One quick clarification is that yolov3.pt has been trained with the default value of img-size 640x640 and hence img-size of 640x640 is the most optimized in that training sample (not testing sample, so img-size may vary when making inference). However, for iou-thres and conf-thres, these values are not used in training the pre-trained network yolov3.pt as I don't see these 2 parameters in train.py and hence these are not coming from training, but I believe these two values are specific to running the tracker if I am correct?

[glenn-jocher]

@takehiro-code yes that's correct, confidence and IoU thresholds are not used during training for gradient computation, but they are used during inference (in test.py called directly and test.py called during training, and detect.py).

A good strategy for determining a good deployment confidence threshold is our new F1 curve. The max point will be a good balance between P and R. See ultralytics/yolov5#2048

[takehiro-code]

@glenn-jocher Thank you very much for the clarification. I am not sure if this has to be in new issue but just to clarify further, the pretrained yolov3.pt (118 MB) from the yolov3/releases has been trained with:
2017 train images from COCO with 118287 images
2017 validation images from COCO with 5000 images
2017 test images from COCO with 20288 out of 40670 images
which I see from coco.yaml if I am correct? I couldn't find the information of how the yolov3.pt has been trained from README or other issues, so would like to confirm.

Secondly, I would like to hear any other parameters how yolov3.pt has been trained such as epoch, batch size etc. From train.py I see that default value is 300 epochs, batch size of 16, 640x640 image size, but wasn't sure if these values are used for training yolov3.pt with coco.yaml (I see default of coco128.yaml for only 128 images of training). Note that my goal is to use the existing best pre-trained model.

[glenn-jocher]

@takehiro-code training is only done on train set. Exact commands to reproduce our training are shown here:
https://github.com/ultralytics/yolov3#training

[takehiro-code]

@glenn-jocher Thanks that clarifies for me regarding the pre-trained network info. One thing that I am unsure regarding finding the optimal confidence threshold and iou threshold is that I was originally computing mAP with a different confidence threshold and iou threshold. Is there a reason we use F1 over mAP to find optimal values? And is it logical to use mAP to find optimal values?

[glenn-jocher]

@takehiro-code mAP is the area under the PR curve. In theory it should be computed at --conf 0.0, we compute at --conf 0.001 in practice for speed.

--conf at max F1 may produce a better real-world balance between P and R.

[takehiro-code]

Thank you. I tried both F1 and mAP and I got F1 score result similar to what you have obtained above i.e. there is local maximum.

For mAP vs confidence threshold, I got a curve monotonically decreasing i.e.mAP is the highest at the lowest confidence threshold I tried, so I think it makes sense to me that in theory, mAP should be computed at 0.0.

[glenn-jocher]

@takehiro-code yes that is correct. Note that in YOLOv5 we produce all four plots vs confidence: P, R, mAP@0.5, F1: