How can I get AUC-ROC curve?Or how can I get specificity?

[hse801]

❔Question

I want to calculate AUC for the test result. How can I get auc-roc curve in yolo?
or since we already have recall, how can I calculate specificity? I saw the metrics.py but couldn't figure it out.
Can someone give me a guide?

Additional context

[github-actions]

👋 Hello @hse801, thank you for your interest in 🚀 YOLOv5! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://www.ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@hse801 you can add any custom metrics you want to metrics.py and output/display it with test.py.

For details on your custom metrics I'd say Google is your friend:
https://en.wikipedia.org/wiki/Receiver_operating_characteristic#:~:text=A%20receiver%20operating%20characteristic%20curve,why%20it%20is%20so%20named.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[luoyizhi516]

❔Question

I want to calculate AUC for the test result. How can I get auc-roc curve in yolo? or since we already have recall, how can I calculate specificity? I saw the metrics.py but couldn't figure it out. Can someone give me a guide?

Additional context

Brother, did you solve this problem? I am also a little confused here, how to get TN, this is a big challenge for me

[Deloustal83]

nobody knows?

[ohjunee]

@hse801 you can add any custom metrics you want to metrics.py and output/display it with test.py.

For details on your custom metrics I'd say Google is your friend: https://en.wikipedia.org/wiki/Receiver_operating_characteristic#:~:text=A%20receiver%20operating%20characteristic%20curve,why%20it%20is%20so%20named.

@glenn-jocher I think I'm asking the same question.. You mean that the code for evaluating ROC curves or AUC is not currently implemented, but it should be implemented by the user himself, right?

[glenn-jocher]

@ohjunee, that's correct. To evaluate ROC curves or AUC, you will need to add the code yourself as a custom metric to metrics.py. Once this metric is added, it can be output and displayed using test.py.

I recommend using websites like https://en.wikipedia.org/wiki/Receiver_operating_characteristic#:~:text=A%20receiver%20operating%20characteristic%20curve,why%20it%20is%20so%20named to learn more about ROC curves and AUC, and how to calculate these metrics yourself.

[ohjunee]

nobody knows?

stats = [torch.cat(x, 0).cpu().numpy() for x in zip(*stats)]  # to numpy
if len(stats) and stats[0].any():
    if task == 'test' and single_cls:
        import sklearn.metrics
        from utils.metrics import plot_roc_curve
        y_true, y_score = stats[0][:,0].astype(int), stats[1]
        fpr, tpr, thresholds = sklearn.metrics.roc_curve(y_true, y_score, pos_label=1)
        auc = sklearn.metrics.auc(fpr, tpr)
        plot_roc_curve(fpr, tpr, auc, save_dir=save_dir)

Share the green rock curve code through the sckit-learn package. The code works for models learned by single class (person).

[glenn-jocher]

Dear @ohjunee,

Thank you for your interest in 🚀 YOLOv5! Based on your previous question, I understand you are asking about calculating AUC for the test result or alternatively calculating specificity.

To address your question, the code you have posted is an implementation to display ROC curve by using Scikit-learn package. It works for models learned by a single class. If you would like to calculate AUC or specificity, you will need to add the code yourself as a custom metric to metrics.py. Once this metric is added, it can be output and displayed using test.py.

Please refer to the documentation on metrics.py for more information on how to add custom metrics.

Thank you for your patience and please let us know if you have any other questions or concerns.

Best regards.

[SBhat2615]

❔Question

I want to calculate AUC for the test result. How can I get auc-roc curve in yolo? or since we already have recall, how can I calculate specificity? I saw the metrics.py but couldn't figure it out. Can someone give me a guide?

Additional context

Was it possible to calculate AUC & specificity ??
could u please tell on how to proceed ?? And how to access values such as tp, fp, from metrics.py ???

Is it possible to do using CLI command for yolov8 or should i dowload the ultralytics repo & then write a python script ??

[glenn-jocher]

@SBhat2615 i understand your interest in calculating AUC and specificity using YOLOv5. The ability to calculate these metrics is not directly available via CLI commands in YOLOv5, but rather would require modifying the source code, such as metrics.py to include the computation of these metrics. Once added, you can access the true positive (tp) and false positive (fp) values via the modified metrics.py file.

To proceed, you'll need to download the Ultralytics YOLOv5 repository and make the necessary customizations to metrics.py to calculate AUC and specificity. Please note that this will require writing a Python script and incorporating the necessary code adjustments.

Should you have any further questions, please feel free to ask!

[Ahmad-Sheeraz]

hey @glenn-jocher to calculate the specificity, TN (True Negatives) are required. I am having a hard time calculating those. Any possibility of the implementation of Specificity in YOLOv5 or v8 any time soon?

[glenn-jocher]

@Ahmad-Sheeraz, specific metrics such as specificity (True Negatives) are not currently included in YOLOv5. However, you can calculate specificity by modifying the source code, specifically the metrics.py file, to include these metrics. While there are no immediate plans for the implementation of specificity in YOLOv5 or v8, you can add it as a custom metric for your own use.

Please feel free to reach out if you have any more questions or need further assistance!

[SBhat2615]

@Ahmad-Sheeraz, specific metrics such as specificity (True Negatives) are not currently included in YOLOv5. However, you can calculate specificity by modifying the source code, specifically the metrics.py file, to include these metrics. While there are no immediate plans for the implementation of specificity in YOLOv5 or v8, you can add it as a custom metric for your own use.

Please feel free to reach out if you have any more questions or need further assistance!

Pip install method (recommended)

!pip install ultralytics==8.0.20

from IPython import display
display.clear_output()

import ultralytics
ultralytics.checks()

Git clone method (for development)

%cd {HOME}
!git clone github.com/ultralytics/ultralytics
%cd {HOME}/ultralytics
%pip install -e .

from IPython import display
display.clear_output()

import ultralytics
ultralytics.checks()

Should we use the second method to change metrics.py?
I'm also working on modifying the model architecture (backbone & head) for specific task. What is the best way to proceed?

[tomhoq]

@SBhat2615 Have you managed to do it?