Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metric visualizations #429

Merged
merged 23 commits into from
Jul 13, 2022
Merged

Add metric visualizations #429

merged 23 commits into from
Jul 13, 2022

Commits on Jul 11, 2022

  1. Refactor VisualizerCallback, fix Wandb bug

    * `VisualizerCallback` is refactored into a base `VisualizerCallbackBase`
    class which is used to construct `VisualizerCallbackImage` via inheritance.
    * Fix bug in `VisualizerCallback.on_test_end`, where `pl_module.logger`
    was accessed instead of iterating over `trainer.loggers`
    (wandb errort silently before iirc)
    * Skeleton for `VisualizerCallbackMetric` is added
    ORippler committed Jul 11, 2022
    Configuration menu
    Copy the full SHA
    bc793aa View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    cfd92b1 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2022

  1. Configuration menu
    Copy the full SHA
    302a728 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    fe19315 View commit details
    Browse the repository at this point in the history
  3. Explicitly close figure in ImageGrid.generate

    Otherwise, matplotlib starts to complain and objects aren't gced
    properly
    ORippler committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    b4bbde7 View commit details
    Browse the repository at this point in the history
  4. Shift Visualizer to VisualizerCallbackBase

    `Visualizer` is needed by all Callbacks for writing images to disk
    ORippler committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    c0d7f59 View commit details
    Browse the repository at this point in the history
  5. Add first metric visualization

    Every metric that should be visualized needs to implement its own
    `generate_figure` function, and the resulting plot is then saved by
    `VisualizerCallbackMetric`
    ORippler committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    6fe5999 View commit details
    Browse the repository at this point in the history
  6. Merge branch 'feature/metrics_visualizer' of github.com:ORippler/anom…

    …alib-1 into feature/metrics_visualizer
    ORippler committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    cee564f View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    7b20171 View commit details
    Browse the repository at this point in the history
  8. Add AUPR metric

    ORippler committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    01286f4 View commit details
    Browse the repository at this point in the history
  9. Make AUPR accessible

    ORippler committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    f9522a0 View commit details
    Browse the repository at this point in the history
  10. Add AUPRO metric and its vizualiation

    * Needed to change signature of `AnomalyModule._collect_outputs` for
    pixel-wise metrics as we need spatial dimensions for
    connected-componenta-analysis in AUPRO
    * Add AUPRO, which uses kornia and the fact that
    per-region overlap == per-region tpr for fast AUPRO computation
    * Updated docstrings for AUPR/AUROC
    ORippler committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    629056b View commit details
    Browse the repository at this point in the history
  11. Adjust tests, CLI and notebooks

    Due to bugs in CLI, feature could not be tested well
    ORippler committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    bbbb188 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2022

  1. Configuration menu
    Copy the full SHA
    2b6a391 View commit details
    Browse the repository at this point in the history
  2. Bugfix of AUPRO implementation

    Since unique fpr/tpr curves are generated for each label,
    we cannot use fpr_index across calls ro `roc`. Instead, we now
    bilinearly resample generated pro curves at `fpr <= self.fpr_limit`
    to fixed sampling points, and then aggregate over the resampled curve.
    ORippler committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    f69b719 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0da986f View commit details
    Browse the repository at this point in the history
  4. Improve variable naming/wording in AUPRO

    We follow a scheme similar to the ROC curve, where the PRO curve is
    composed of per-region TPR plotted against global FPR.
    ORippler committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    30b1c17 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    4c5a795 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    87b3a2a View commit details
    Browse the repository at this point in the history
  7. Rename VisualizerCallbacks

    We now prepend the last part, i.e. `VisualizerCallbackBase` -> `BaseVisualizerCallback`
    ORippler committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    469f4ef View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    e2cf6c2 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    1dacd8a View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    43d8e4a View commit details
    Browse the repository at this point in the history