🚀 Customisable Image Visualizer

[samet-akcay]

📝 Description

• This PR introduces the ImageVisualizer class to Anomalib, providing a flexible and customizable way to visualize anomaly detection results. The ImageVisualizer can be used as a callback in the Anomalib Engine. The subsequent PRs will add the visualizer to the base model, so users will be able to modify it from there as well.
• Addresses Make visualization configurable #1917

Features

• Visualize multiple fields from ImageItem
• Fully customizable field selection and layout
• Adjustable visualization parameters (gap, font size, background color)
• Overlay option to any field (ie. heatmap on image, masks on image etc.)

Examples

1: Engine-specific visualizer

When not set, Anomalib will use the default visualizer settings, which would be equivalent to the following

from anomalib.data import MVTec
from anomalib.engine import Engine
from anomalib.models import Padim

data = MVTec()
model = Padim()
engine = Engine()
engine.train(datamodule=data, model=model)

The code above will produce the following output

2: Default visualizer

from anomalib.data import MVTec
from anomalib.engine import Engine
from anomalib.models import Padim
from anomalib.visualization.image import ImageVisualizer

data = MVTec()
model = Padim()
visualizer = ImageVisualizer()
engine = Engine(callbacks=[visualizer])
engine.train(datamodule=data, model=model)

3: Customized visualizer

It is possible to overlay any custom field on top of each other. This includes multiple fields, which overall makes this design fully customizable.

visualizer = ImageVisualizer(
    fields=["image", "gt_mask", "anomaly_map"], 
    overlay_fields=[("image", ["anomaly_map"])]
)

4: Modifying overlay appearance

visualizer = ImageVisualizer(
    overlay_fields_config={
        "pred_mask": {"alpha": 0.7, "color": (255, 0, 0), "mode": "fill"},
    },
)

5: Advanced configuration combining multiple customizations

fields_config={
    "anomaly_map": {"colormap": True, "normalize": True}, 
    "pred_mask": {"color": (0, 0, 255)}
}
overlay_fields_config={
    "anomaly_map": {"colormap": True, "normalize": True},
    "pred_mask": {"alpha": 0.7, "mode": "contour"},
}
text_config={"font": "times.ttf", "size": 24, "color": "white", "background": (0, 0, 0, 180)}

visualizer = ImageVisualizer(
    fields=["image", "gt_mask", "anomaly_map", "pred_mask"],
    overlay_fields=[("image", ["anomaly_map"]), ("image", ["pred_mask"])],
    field_size=(384, 384),
    fields_config=fields_config,
    overlay_fields_config=overlay_fields_config,
    text_config=text_config,
    output_dir="./custom_visualizations",
)

✨ Changes

Select what type of change your PR is:

• 🐞 Bug fix (non-breaking change which fixes an issue)
• 🔨 Refactor (non-breaking change which refactors the code base)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update
• 🔒 Security update

✅ Checklist

Before you submit your pull request, please make sure you have completed the following steps:

• 📋 I have summarized my changes in the CHANGELOG and followed the guidelines for my type of change (skip for minor changes, documentation updates, and test enhancements).
• 📚 I have made the necessary updates to the documentation (if applicable).
• 🧪 I have written tests that support my changes and prove that my fix is effective or my feature works (if applicable).

For more information about code review checklists, see the Code Review Checklist.

[jpcbertoldo]

Just a personal preference but since this PR is open: I think it helps when the contours of the mask are overlaid like this for exemple

[ashwinvaidya17]

The new visualizer looks amazing. I have a few minor comments. Also, how do we plan to tackle text-only output of VLM?

[ashwinvaidya17]

Where is this used?

[samet-akcay]

I want user to use keywords for the rest of the arguments after *. For example, following line follows this keyword-only format

https://github.com/samet-akcay/anomalib/blob/c17c77ce655cc146e065699bd10f69a9cb9b3ac1/src/anomalib/visualization/image/functional.py#L376

[samet-akcay]

image = visualize_field(field, value, colormap=colormap, normalize=normalize)

[ashwinvaidya17]

So, basically we want to force the users to pass parameters using keywords

[samet-akcay]

Yes, because such args are hard to read/follow if a keyword is not provided

[samet-akcay]

The new visualizer looks amazing. I have a few minor comments. Also, how do we plan to tackle text-only output of VLM?

This is for image-based models. I'll need to implement VideoVisualizer, and possibly something like VLMVisulizer

[samet-akcay]

@jpcbertoldo, how does it look like now? Both is possible with the updated PR

[ashwinvaidya17]

Looks fine to me. But not sure if I am completely sold on the field config, and overlay field config part. I usually rely on my IDE to suggest customisable parameters. With the nested dict, it is harder to figure how, and to what I can I change the parameters to.
How often would we see users change these parameters? If it is not very frequent, then I am in favour of being a little more opinionated and keep these parameters as constants. If the users want something else, they can modify the source.

[samet-akcay]

Looks fine to me. But not sure if I am completely sold on the field config, and overlay field config part. I usually rely on my IDE to suggest customisable parameters. With the nested dict, it is harder to figure how, and to what I can I change the parameters to. How often would we see users change these parameters? If it is not very frequent, then I am in favour of being a little more opinionated and keep these parameters as constants. If the users want something else, they can modify the source.

that'd make my life easier :)

[jpcbertoldo]

@jpcbertoldo, how does it look like now? Both is possible with the updated PR

Looks really cool. The design is somehow complete but not complicated as expected haha.

I wanted to try something like this

visualizer = ImageVisualizer(
    fields=["image",  "anomaly_map"],
    overlay_fields=[("image", ["anomaly_map", "gt_mask"]), ("anomaly_map", ["gt_mask"])],
    fields_config={"pred_mask": {"color": (255, 0, 0)}},
    overlay_fields_config={
        "anomaly_map": {"alpha": 0.6, "mode": "fill"},
        "pred_mask": {"alpha": 1., "mode": "contour"},
    },
)

Will try tomorrow, my environment has some unsaved stuff right now.

By the way the output of example 5 is outdated, right? Because the pred mask config is set to contour but we see a fill in the image.

[samet-akcay]

@jpcbertoldo, how does it look like now? Both is possible with the updated PR

Looks really cool. The design is somehow complete but not complicated as expected haha.

I wanted to try something like this

visualizer = ImageVisualizer(
    fields=["image",  "anomaly_map"],
    overlay_fields=[("image", ["anomaly_map", "gt_mask"]), ("anomaly_map", ["gt_mask"])],
    fields_config={"pred_mask": {"color": (255, 0, 0)}},
    overlay_fields_config={
        "anomaly_map": {"alpha": 0.6, "mode": "fill"},
        "pred_mask": {"alpha": 1., "mode": "contour"},
    },
)

Will try tomorrow, my environment has some unsaved stuff right now.

By the way the output of example 5 is outdated, right? Because the pred mask config is set to contour but we see a fill in the image.

For some reason, fields_config is overwriting the pred_mask configs in overlay_fields_config. I'll investigate that

[jpcbertoldo]

Really nice feature!

from anomalib.visualization.image import ImageVisualizer
visualizer = ImageVisualizer(
    overlay_fields=[("image", ["gt_mask"]), ("image", ["anomaly_map", "gt_mask"]), ("anomaly_map", ["gt_mask"])],
    overlay_fields_config={
        "anomaly_map": {"alpha": 0.5, "mode": "fill"},
        "gt_mask": {"alpha": 1., "mode": "contour", "color": (255, 0, 255)},
    },
)
engine = Engine(callbacks=[visualizer])
engine.test(datamodule=data, model=model)

gives

i would suggest gt_mask overlayed as a default, it really helps to compare the images because it gives a reference shape to locate things

[codecov]

Codecov Report

Attention: Patch coverage is 83.80567% with 40 lines in your changes missing coverage. Please review.

Please upload report for BASE (feature/design-simplifications@99b4e9d). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/anomalib/visualization/image/functional.py	73.84%	34 Missing ⚠️
src/anomalib/utils/path.py	82.75%	5 Missing ⚠️
...rc/anomalib/visualization/image/item_visualizer.py	98.03%	1 Missing ⚠️

Additional details and impacted files

@@                        Coverage Diff                        @@
##             feature/design-simplifications    #2334   +/-   ##
=================================================================
  Coverage                                  ?   78.44%           
=================================================================
  Files                                     ?      288           
  Lines                                     ?    12367           
  Branches                                  ?        0           
=================================================================
  Hits                                      ?     9701           
  Misses                                    ?     2666           
  Partials                                  ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.