Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix tick labels for background FN/FP #9414

Merged

Conversation

hotohoto
Copy link
Contributor

@hotohoto hotohoto commented Sep 14, 2022

Fix tick labels for background FN/FP in the confusion matrix.

They needs to be switched to each other.

This issue has been raised within #8000.

🛠️ PR Summary

Made with ❤️ by Ultralytics Actions

🌟 Summary

Improved clarity in confusion matrix terminology.

📊 Key Changes

  • Changed the annotation of false positives (FP) and false negatives (FN) to "true background" and "predicted background" for better clarity.
  • Simplified confusion matrix plotting by creating a single ticklabels variable.
  • Made sure that the ticklabels reflect the updated terminology and apply to both x and y axes.

🎯 Purpose & Impact

  • The renaming of confusion matrix terms aims to reduce confusion about the background class in object detection, making it clearer what constitutes a background prediction versus a foreground object.
  • Streamlining the plot code enhances maintainability and readability.
  • Users of the YOLOv5 library can now interpret their models' performance with a more intuitive understanding of where the background class fits into their predictions, potentially improving the model refinement process. 📈🧠

In the confusion matrix.
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👋 Hello @hotohoto, thank you for submitting a YOLOv5 🚀 PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

  • ✅ Verify your PR is up-to-date with ultralytics/yolov5 master branch. If your PR is behind you can update your code by clicking the 'Update branch' button or by running git pull and git merge master locally.
  • ✅ Verify all YOLOv5 Continuous Integration (CI) checks are passing.
  • ✅ Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." — Bruce Lee

@glenn-jocher
Copy link
Member

@hotohoto confusion matrix labels are correct. Backgrounds that are predicted as a class are a False Negative (FN) and vice versa.

@glenn-jocher glenn-jocher added the invalid This doesn't seem right label Sep 15, 2022
@hotohoto
Copy link
Contributor Author

hotohoto commented Sep 16, 2022

"Backgrounds that are predicted as a class are a False Negative (FN) and vice versa."

I mean it. That's the reason why I think the labels are incorrect. 🤔

Concretely, the last column in a confusion matrix is the case when it's predicted as a class but the ground truth is background. So it's a False Negative like you said. But it's labeled as Fasle Positive currently which is incorrect.

And vice versa.

(You may refer to the image at #8000 (comment))

@glenn-jocher

@glenn-jocher
Copy link
Member

@hotohoto the last column indicates cells that were identified as a class but in reality contains nothing. This is the definition of a FP detection, which is why the label shows 'background FP'

But probably to simplify things I'll just remove the FP and TP from the matrix and just leave 'background' as all the other classes.

@glenn-jocher glenn-jocher linked an issue Sep 16, 2022 that may be closed by this pull request
1 task
@hotohoto
Copy link
Contributor Author

hotohoto commented Sep 16, 2022

@hotohoto the last column indicates cells that were identified as a class but in reality contains nothing. This is the definition of a FP detection, which is why the label shows 'background FP'
But probably to simplify things I'll just remove the FP and TP from the matrix and just leave 'background' as all the other classes.

  • I think that's a possible interpretation, but then I think it should be FP rather than background FP.
  • I like your suggestion removing FP and TP from the matrix and leaving just background.

Thanks. @glenn-jocher

hotohoto and others added 2 commits September 16, 2022 18:33
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@glenn-jocher glenn-jocher removed the invalid This doesn't seem right label Sep 16, 2022
@glenn-jocher glenn-jocher merged commit c7a2d6b into ultralytics:master Sep 16, 2022
@glenn-jocher
Copy link
Member

@hotohoto PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐

@leo-q8
Copy link

leo-q8 commented Nov 1, 2022

@hotohoto confusion matrix labels are correct. Backgrounds that are predicted as a class are a False Negative (FN) and vice versa.

@hotohoto the last column indicates cells that were identified as a class but in reality contains nothing. This is the definition of a FP detection, which is why the label shows 'background FP'

But probably to simplify things I'll just remove the FP and TP from the matrix and just leave 'background' as all the other classes.

@glenn-jocher @hotohoto Hi,I'm confused by these two reply. Are they all talking about the same situation as shown in the follow figure, that is, backgrounds(in reality contains nothing) are predicted as a class?
fp
So, the red prediction should be a False Positive detection?

@leo-q8
Copy link

leo-q8 commented Nov 1, 2022

I also have a question about the following predicted background processing code.

yolov5/utils/metrics.py

Lines 175 to 178 in a83d2a5

if n:
for i, dc in enumerate(detection_classes):
if not any(m1 == i):
self.matrix[dc, self.nc] += 1 # predicted background

When all detections are False Positive background, which means all values of the iou will be less than iou_thres.
And n must be False in this case,causing none of the backbound FP is counted into self.matrix.
I think it should be modified as follows:

for i, dc in enumerate(detection_classes):
    if not n or not any(m1 == i): # when n == False,all detections should be counted into predicted background
        self.matrix[dc, self.nc] += 1  # predicted background

@glenn-jocher
Copy link
Member

@avril-wang1214 Thank you for the feedback. I'll review the provided information and make the necessary adjustments to the code. Your insights are greatly appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Background fn
3 participants