Why is there a difference between TP,FP calculated by val.py and the saved results?

[Eri2726]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hi.
I am having trouble matching the sum of TP,FP calculated by val.py and the sum of the results saved by --save-txt argument.
I've been trying below:

---

This time, I tried only one image and label file.
With reference to #5727 , I changed the val.py as follows:

　#Compute metrics
    stats = [torch.cat(x, 0).cpu().numpy() for x in zip(*stats)]  # to numpy
    if len(stats) and stats[0].any():
        tp, fp, p, r, f1, ap, ap_class = ap_per_class(*stats, plot=plots, save_dir=save_dir, names=names)
        print("TP")
        print(tp)
        print("FP")
        print(fp)
        ap50, ap = ap[:, 0], ap.mean(1)  # AP@0.5, AP@0.5:0.95
        mp, mr, map50, map = p.mean(), r.mean(), ap50.mean(), ap.mean()
    nt = np.bincount(stats[3].astype(int), minlength=nc)  # number of targets per class

And executed val.py.

python val.py --weights C:\Users\~~~~\models\best.pt --data C:\Users\~~~~\data\valid3\data.yaml --img 640 --save-txt --save-conf --conf-thres 0.2 --iou-thres 0.6

As a result, tp and fp came out in the terminal below:

I think it shows that TP,FP of the one image I used.
But, the result of the txt file saved in runs/val/exp10/labels is as follows:

0 0.2742 0.624213 0.0461547 0.033026 0.847769
0 0.332273 0.707128 0.0376017 0.0249223 0.771621
0 0.395059 0.906789 0.0372657 0.0298265 0.741917
0 0.582561 0.406594 0.0304165 0.0242927 0.701719
0 0.226321 0.558442 0.0326438 0.02553 0.625718
0 0.640264 0.23205 0.0264291 0.0210307 0.595323
0 0.477026 0.730236 0.0265545 0.0219655 0.587266
0 0.317117 0.834086 0.0306103 0.0242775 0.586945
0 0.222225 0.097663 0.0422521 0.0322202 0.521401
0 0.84692 0.772109 0.0358007 0.0285754 0.513393
0 0.535209 0.91223 0.0319012 0.0249561 0.430841
0 0.804827 0.324432 0.0229721 0.0194096 0.373771
0 0.585995 0.402967 0.0236618 0.0208841 0.267536
0 0.654169 0.0538474 0.0232822 0.0184519 0.245073

I think that this is the result of detection, but there are 14 records in total.
In this case, TP+FP=12, so I guess the correct number of records is 12.
Why are there differences?

Could you please let me know?

Additional

No response

[glenn-jocher]

@Eri2726 the val.py script should calculate the overall TP and FP for all validation images. If you are only comparing a single image, then the results may not be the same as the overall TP and FP calculated by val.py. Additionally, the sum of TP and FP in the txt files saved by --save-txt argument should equal the overall TP and FP calculated by val.py. It's possible that the txt file only represents a subset of the validation set, but overall TP and FP across all validation images should still match. Please double check that you are comparing the correct values and if the issue persists, provide more information such as the exact steps you are taking and any error messages you are encountering.

[Eri2726]

@glenn-jocher
Thank you for your response.I executed and cheked val.py again, the problem could not be solved.I'll add the detail steps as follows:

1. Clone latest version

I've reclones the latest YOLOv5 version.

2. Edit val.py to confirm TP,FP

In order to outpu TP,FP to the terminal, the following changes were added after line 273 of the val.py.

    # Compute metrics
    stats = [torch.cat(x, 0).cpu().numpy() for x in zip(*stats)]  # to numpy
    if len(stats) and stats[0].any():
        tp, fp, p, r, f1, ap, ap_class = ap_per_class(*stats, plot=plots, save_dir=save_dir, names=names)
        ap50, ap = ap[:, 0], ap.mean(1)  # AP@0.5, AP@0.5:0.95
        mp, mr, map50, map = p.mean(), r.mean(), ap50.mean(), ap.mean()
    nt = np.bincount(stats[3].astype(int), minlength=nc)  # number of targets per class
    print("TP") #Added
    print(tp) #Added
    print("FP") #Added
    print(fp) #Added

3. Prepare validation data

I wanted to know TP,FP of one image, so I prepared only one image.
The folder structure is as bellows:

valid
│  data.yaml
│  labels.cache
│  labels.cache.npy
│
├─images
│      IMG_0121.jpg
│
└─labels
        IMG_0121.txt

4. Executed val.py

I typed bellow into the terminal and executed val.py.

python val.py --weights C:\Users\~~~~\yolov5-master\models\best.pt --data C:\Users\~~~~\yolov5-master\data\valid\data.yaml --img 640 --save-txt --save-conf --conf-thres 0.2 --iou-thres 0.6

5. TP,FP results

The terminal displayed the following:

val: data=C:\Users\~~~~\yolov5-master\data\valid\data.yaml, weights=['C:\\Users\~~~~\yolov5-master\\models\\best.pt'], batch_size=32, imgsz=640, conf_thres=0.2, iou_thres=0.6, max_det=300, task=val, device=, workers=8, single_cls=False, augment=False, verbose=False, save_txt=True, save_hybrid=False, save_conf=True, save_json=False, project=runs\val, name=exp, exist_ok=False, half=False, dnn=False
requirements: C:\Users\~~~~\requirements.txt not found, check failed.
WARNING  confidence threshold 0.2 > 0.001 produces invalid results
YOLOv5  2023-4-20 Python-3.9.16 torch-1.13.1+cpu CPU

Fusing layers... 
YOLOv5x_broccoli summary: 362 layers, 88390614 parameters, 0 gradients
val: Scanning C:\Users\~~~~\yolov5-master\data\valid\labels... 1 images, 0 backgrounds, 0 corrupt: 100%|██████████| 1/1 [00:03<00:00,  3.46s/it]
val: WARNING  Cache directory C:\Users\~~~~\yolov5-master\data\valid is not writeable: [WinError 183] : 'C:\\Users\~~~~\yolov5-master\\data\\valid\\labels.cache.npy' -> 'C:\\Users\~~~~\yolov5-master\\data\\valid\\labels.cache'
                 Class     Images  Instances          P          R      mAP50   mAP50-95: 100%|██████████| 1/1 [00:01<00:00,  1.16s/it]
TP
[         10]
FP
[          2]
                   all          1         18      0.815      0.556      0.704      0.401
Speed: 0.9ms pre-process, 1026.2ms inference, 0.0ms NMS per image at shape (32, 3, 640, 640)
Results saved to runs\val\exp3
1 labels saved to runs\val\exp3\labels

I got the result "TP=10","FP=2", right?

6. --save-txt result

The saved result in runs\val\exp3\labels is bellow:

0 0.2742 0.624213 0.0461547 0.033026 0.847769
0 0.332273 0.707128 0.0376017 0.0249223 0.771621
0 0.395059 0.906789 0.0372657 0.0298265 0.741917
0 0.582561 0.406594 0.0304165 0.0242927 0.701719
0 0.226321 0.558442 0.0326438 0.02553 0.625718
0 0.640264 0.23205 0.0264291 0.0210307 0.595323
0 0.477026 0.730236 0.0265545 0.0219655 0.587266
0 0.317117 0.834086 0.0306103 0.0242775 0.586945
0 0.222225 0.097663 0.0422521 0.0322202 0.521401
0 0.84692 0.772109 0.0358007 0.0285754 0.513393
0 0.535209 0.91223 0.0319012 0.0249561 0.430841
0 0.804827 0.324432 0.0229721 0.0194096 0.373771
0 0.585995 0.402967 0.0236618 0.0208841 0.267536
0 0.654169 0.0538474 0.0232822 0.0184519 0.245073

As you can see, there are 14 records in this text file.
I guess it should be 12 records.....

7. Other information

• OS：Windows11
• Anaconda：conda 23.1.0
• Python：Python 3.9.16
  Please let me know if you need any other information.

Regards,

[Eri2726]

8.Additional information

The images saved under runs/val/val_batch0_pred.jpg are as follows.

It looks like 13 bounding boxs are drawn.
*One of the thirteen overlaps.

When running val.py with a single image, the number of detected objects shown in val_batch0_pred.jpg and the number of detected objects written in the text file under the label folder should match, right???

Best, regards

[glenn-jocher]

@Eri2726 Yes, when running val.py on a single image the number of detected objects shown in val_batch0_pred.jpg and the number of detected objects written in the text file under the label folder should match. However, keep in mind that the number of detected objects may not be the same as the overall number of true positives and false positives calculated by val.py over the entire validation set. Also, note that overlapping bounding boxes on a single image may be counted as separate objects in the text file even if they overlap in reality. You can try setting the max_overlaps parameter in val.py to a higher value to avoid this.

[Eri2726]

@glenn-jocher Thank you for letting me know!
I’d also like to know about folloeing 2 things:

1. Why does the following happen?

keep in mind that the number of detected objects may not be the same as the overall number of true positives and false positives calculated by val.py over the entire validation set.

2. I'll try setting the max_overlaps parameter, but the parameter couldn't be found.How can I set that?

Regards,

[glenn-jocher]

@Eri2726 Regarding your questions,

1. The number of detected objects may not be the same as the overall number of true positives and false positives calculated by val.py over the entire validation set because when there are multiple bounding boxes around a single object, they may count as separate detections even though they correspond to the same object. Therefore, the number of detected objects may be higher than the overall TP and FP calculated across all validation images. Additionally, the algorithm used to calculate TP and FP in val.py may include some false positives that are not counted as detections and therefore not shown in the text files saved by --save-txt argument.

2. The max_overlaps parameter is not a standard parameter in val.py. You may need to modify the script itself to implement this parameter. For example, you can add a new parameter to the function that calculates the TP and FP and use this value to filter out overlapping bounding boxes before counting them as separate detections.

[Eri2726]

@glenn-jocher
Thank you for your kind answer.I understanded about No.2!On the other hand, there is additional question about No.1.I would be happy if you could answer following as well.

Therefore, the number of detected objects may be higher than the overall TP and FP calculated across all validation images. Additionally, the algorithm used to calculate TP and FP in val.py may include some false positives that are not counted as detections and therefore not shown in the text files saved by --save-txt argument.

I understanded the above specifications.
Is there a way to draw objects that the model detects as TP or FP in val.py?
In order to improve the accuracy of the model, I'd like to see with pictures what the cirrent model is detecting correctly and what is making false positives.

Best regards,

[glenn-jocher]

@Eri2726 dear user,

Thank you for your follow-up question. In val.py, you can set the --save-crops argument to save crops around the detected objects. This can help you visualize what the model is detecting and what is a false positive. Additionally, you can modify the script to draw bounding boxes around true positives and false positives by using the coordinates of the boxes in the saved text files. To do this, you can use a library such as OpenCV or PIL to draw rectangles on the images.

Please note that drawing bounding boxes on the images will not modify the saved text files and will only be for visualization purposes. Also, keep in mind that the number of detected objects shown in the saved images may not match the overall TP and FP calculated by val.py across all validation images due to the factors mentioned earlier.

I hope this helps. Let me know if you have any further questions.

Sincerely,

[Eri2726]

@glenn-jocher
Hi, thank you for your reply.
I developed my own algorithm and succeeded in visualizing TP, FP, and FN!
It's seems to be useful to improving the current model.

Thank you again.

[Sarthak2426]

@Eri2726

@glenn-jocher Hi, thank you for your reply. I developed my own algorithm and succeeded in visualizing TP, FP, and FN! It's seems to be useful to improving the current model.

Thank you again.

hello sir. its a humble request can u pls share the algo which u have devloped for tp and fp it will be to helpful

[glenn-jocher]

@Sarthak2426 hello,

Thank you for reaching out and expressing your interest in the algorithm that was developed to visualize TP, FP, and FN. While I appreciate your request, I'm afraid I am unable to share the specific algorithm as it was not provided in this conversation.

If you have any other questions or need assistance with any other topic, please feel free to ask. I'll be more than happy to help.

Regards,