-
-
Notifications
You must be signed in to change notification settings - Fork 16.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Impressive Results when changing conf-thres
and iou-thres
#8669
Comments
If you mean the conf-thres in val.py, yes it's wrong to change that. The reason is exactly what was said by the warning. |
In my specific application, I only care about precision/recall. Also, see this comment which says that changing mAP would produce inaccurate mAP but nothing about other metrics. |
As I can see, this is the only place where Line 217 in 4c1784b
So, this is basically filtering out the results. Results which have lower This is where AP for each class is calculated. Lines 263 to 271 in 4c1784b
And this is the actual code for calculating average precision. Lines 29 to 93 in 4c1784b
which calls this function: Lines 96 to 121 in 4c1784b
Just the first stage is affected by
Am I missing something? |
i have the same question |
I think this may be a bug. I checked that results for Recall should decrease as we increase |
conf-thres
conf-thres
(Most probably a bug)
I think the bug is here. In line 217, we are filtering out based on non_max_suppression. Then we enumerate on filtered result and use filtered targets to calculate mAP. This should not be the case. We should always return unfiltered targets. Lines 214 to 247 in 1c5e92a
I've also printed def ap_per_class(tp, conf, pred_cls, target_cls, plot=False, save_dir='.', names=(), eps=1e-16):
""" Compute the average precision, given the recall and precision curves.
Source: https://github.com/rafaelpadilla/Object-Detection-Metrics.
# Arguments
tp: True positives (nparray, nx1 or nx10).
conf: Objectness value from 0-1 (nparray).
pred_cls: Predicted object classes (nparray).
target_cls: True object classes (nparray).
plot: Plot precision-recall curve at mAP@0.5
save_dir: Plot save directory
# Returns
The average precision as computed in py-faster-rcnn.
"""
print(f"TP Shape: {tp.shape}")
print(f"Conf Shape : {conf.shape}")
print(f"predicted_cls: {pred_cls.shape}")
print(f"target_cls: {target_cls.shape}") ## using conf-thres 0.85
TP Shape: (33, 10)
Conf Shape : (33,)
predicted_cls: (33,)
target_cls: (30,)
# -------------
# using conf-thres 0.001
TP Shape: (25239, 10)
Conf Shape : (25239,)
predicted_cls: (25239,)
target_cls: (281,)
# -------------
# using conf-thres 0.001
TP Shape: (24777, 10)
Conf Shape : (24777,)
predicted_cls: (24777,)
target_cls: (266,)
# -------------- Even when having same conf-thres as recommended, we end up calculating the result wrong! Target_cls should always be the same, no matter what conf-thres is, this is because number of target labels per class is always the same. This way, even setting conf-thres to 0.001 gives wrong results. If you look at labels count column in every threshold above, you understand that this is wrong. Labels count shouldn't change for each run! I will do a PR soon. |
conf-thres
(Most probably a bug)conf-thres
(most probably a bug)
conf-thres
(most probably a bug)conf-thres
and iou-thres
(most probably a bug)
Now with PR #8686, my results are consistent and everything is fine. You can also see that label's count is persistent in each run and it is equal to actual labels that I have. However, my model is not doing any good, so I can not publish the results! Before that, I was actually happy about the result. Here's the results after fixing the bug: Conf-thres = 0.001Conf-thres = 0.2Conf-thres = 0.5Conf-thres = 0.7Conf-thres = 0.85Conf-thres = 0.9 |
We can not just say that changing See this paper which has demonstrated the importance of confidence threshold. It is interesting that it has actually done some experiments using Yolov5. Confidence Score: The Forgotten Dimension of Object Detection Performance Evaluation |
conf-thres
and iou-thres
(most probably a bug)conf-thres
and iou-thres
This PR is not required actually. If you have this problem, you should just update your forked repo to latest version. This has been fixed a little earlier. |
I was having the same issues before updating my repo. Thank you for your analysis! |
@kaminetzky you're welcome! It's great to hear that your issues have been addressed. Feel free to reach out if you need further assistance. Good luck with your work! |
Search before asking
Question
I have actually searched yolov5 and read theses related issues:
Still, with my dataset (which is medical image) I get impressive results when increasing
conf-thres
. I am not sure if my model is doing good or actually changingconf-thres
is basically wrong?Also, PR curves for each
conf-thres
is different. Is this normal?I am worried about the results because of warning that you issue when
conf-thres
is more than 0.001.[Default] Conf-thres = 0.001
Conf-thres = 0.5
Conf-thres = 0.7
Conf-thres = 0.8
Conf-thres = 0.85
Conf-thres = 0.86
Conf-thres = 0.88
Conf-thres = 0.9
Additional
No response
The text was updated successfully, but these errors were encountered: