Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add plots and additional metrics #420

Closed
ORippler opened this issue Jul 8, 2022 · 4 comments · Fixed by #429
Closed

Add plots and additional metrics #420

ORippler opened this issue Jul 8, 2022 · 4 comments · Fixed by #429

Comments

@ORippler
Copy link
Contributor

ORippler commented Jul 8, 2022

Is your feature request related to a problem? Please describe.
Imo, it would be nice to have:

  • Pixel-wise AUPR + PRO until 30% FPR as metrics
  • Plots with baselines where applicable for all metrics (ROC-Curve, PR-Curve, PRO-Curve)

Describe the solution you'd like

  1. Add/implement appropriate torchmetric classes, and add them to the AnomlibMetricCollection
  2. Add callbacks which access the underlying scores for metric calculation (i.e. tuple of tpr/fpr for auroc) and which subsequently use the scores to generate and save the plots.
    • Alternatively, one could think of expanding/inheriting the VisualizerCallback

Thoughts?

@ashwinvaidya17
Copy link
Collaborator

@ORippler I think segregating ImageVisualizerCallback and MetricVisualizerCallback might be a cleaner approach. Let me know what are your thoughts on that.

Also, just to break down your issue here is a list to capture it. Feel free to suggest any modifications.

  • Add AUPR metrics
  • Add PRO metrics
  • Add visualizer for plotting the underlying scores in each metrics

@ashwinvaidya17
Copy link
Collaborator

Related issues

@ORippler
Copy link
Contributor Author

@ORippler I think segregating ImageVisualizerCallback and MetricVisualizerCallback might be a cleaner approach. Let me know what are your thoughts on that.

Also, just to break down your issue here is a list to capture it. Feel free to suggest any modifications.

* [ ]  Add AUPR metrics

* [ ]  Add PRO metrics

* [ ]  Add visualizer for plotting the underlying scores in each metrics

Sure we can split up the Callbacks, but then I suggest inheriting from a VisualizerCallbackBase.

@samet-akcay
Copy link
Contributor

Sure we can split up the Callbacks, but then I suggest inheriting from a VisualizerCallbackBase

Sure, this separation should cause minimal duplication.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants