v0.4.0: Evaluation metrics and PyTorch 2.0

This minor release adds evaluation metrics to the package and bumps PyTorch to version 2.0

Note: TorchCAM 0.4.0 requires PyTorch 2.0.0 or higher.

Highlights

Evaluation metrics

This release comes with a standard way to evaluate interpretability methods. This allows users to better evaluate models' robustness:

from functools import partial
from torchcam.metrics import ClassificationMetric
metric = ClassificationMetric(cam_extractor, partial(torch.softmax, dim=-1))
metric.update(input_tensor)
metric.summary()

What's Changed

New Features 🚀

• feat: Added CAM evaluation metric by @frgfm in #172
• ci: Added FUNDING button by @frgfm in #182
• feat: Added new TV models to demo by @frgfm in #184
• ci: Added precommits, bandit & autoflake by @frgfm in #192
• feat: Removes model hooks when the context manager exits by @frgfm in #198

Bug Fixes 🐛

• chore: Applied post-release modifications by @frgfm in #180
• fix: Fixed division by zero during normalization by @frgfm in #185
• fix: Fixed zero division for weight computation in gradient based methods by @frgfm in #187
• docs: Fixes README badges by @frgfm in #194
• ci: Fixes issue templates by @frgfm in #196
• fix: Fixes SmoothGradCAMpp by @frgfm in #204

Improvements

• docs: Improved documentation build script by @frgfm in #183
• docs: Updates README & CONTRIBUTING by @frgfm in #193
• feat: Removed param grad computation in scripts by @frgfm in #201
• style: Updates precommit hooks and mypy config by @frgfm in #203
• refactor: Replaces flake8 by ruff and updates python version by @frgfm in #211
• style: Bumps ruff & black, removes isort & pydocstyle by @frgfm in #216
• style: Bumps ruff and updates torch & torchvision version specifiers by @frgfm in #219
• docs: Updates CONTRIBUTING & README by @frgfm in #220
• test: Speeds up test suite using plugins by @frgfm in #222
• ci: Adds multiple build CI jobs by @frgfm in #223
• feat: Removes warnings for torchvision and matplotlib by @frgfm in #224

Full Changelog: v0.3.2...v0.4.0

v0.3.2: Some CAM fixes and support of batch processing

This patch release fixes the Score-CAM methods and improves the base API for CAM computation.

Note: TorchCAM 0.3.2 requires PyTorch 1.7.0 or higher.

Highlights

😯 Batch processing

CAM computation now supports batch sizes larger than 1 (#143) ! Practically, this means that you can compute CAMs for multiple samples at the same time, which will let you make the most of your GPU as well ⚡

The following snippet:

import torch
from torchcam.methods import LayerCAM
from torchvision.models import resnet18

# A preprocessed (resized & normalized) tensor
img_tensor = torch.rand((2, 3, 224, 224))
model = resnet18(pretrained=True).eval()
# Hook your model before inference
cam_extractor = LayerCAM(model)
out = model(img_tensor)
# Compute the CAM
activation_map = cam_extractor(out[0].argmax().item(), out)
print(activation_map[0].ndim)

will yield 3 as the batch dimension is now also used.

🖌️ Documentation theme

New year, new documentation theme!
For clarity and improved interface, the documentation theme was changed from Read the Docs to Furo (#162)

This comes with nice features like dark mode and edit button!

🖱️ Contribution process

Contributions are important to OSS projects, and for this reason, a few improvements were made to the contribution process:

• added a Makefile for easier development (#109)
• added a dedicated README for the documentation (#109)
• updated CONTRIBUTING (#109, #166)

Breaking changes

CAM signature

CAM extractors now outputs a list of tensors. The size of the list is equal to the number of target layers and ordered the same way.
Each of these elements used to be a 2D spatial tensor, and is now a 3D tensor to include the batch dimension:

# Model was hooked and a tensor of shape (2, 3, 224, 224) was forwarded to it
amaps = cam_extractor(0, out)
for elt in amaps: print(elt.shape)

will, from now on, yield

torch.Size([2, 7, 7])

What's Changed

Breaking Changes 🛠

• feat: Adds support for batch processing and fixes ScoreCAMs by @frgfm in #143

New Features 🚀

• ci: Added release note template and a job to check PR labels by @frgfm in #138
• docs: Added CITATION file by @frgfm in #144

Bug Fixes 🐛

• fix: Updated headers and added pydocstyle by @frgfm in #137
• chore: Updated PyTorch version specifier by @frgfm in #149
• docs: Fixed deprecated method call by @frgfm in #158
• chore: Fixed jinja2 deps (subdep of sphinx) by @frgfm in #159
• docs: Fixed docstring of ISCAM by @frgfm in #160
• docs: Fixed multi-version build by @frgfm in #163
• docs: Fixed codacy badge by @frgfm in #164
• docs: Fixed typo in CONTRIBUTING by @frgfm in #166
• docs: Fixed author entry in pyproject by @frgfm in #168
• style: Fixed import order by @frgfm in #175

Improvements

• docs: Added PR template and tools for contributing by @frgfm in #109
• refactor: Removed unused import by @frgfm in #110
• feat: Added text strip for multiple target selection in demo by @frgfm in #111
• refactor: Updated environment collection script by @frgfm in #112
• style: Updated flake8 config by @frgfm in #115
• ci: Updated isort config and related CI job by @frgfm in #118
• ci: Speeded up the example script CI check by @frgfm in #130
• refactor: Updated the timing function for latency eval by @frgfm in #129
• docs: Updated TOC of documentation by @frgfm in #161
• refactor: Updated build config and documentation theme by @frgfm in #162
• style: Updated mypy and isort configs by @frgfm in #167
• chore: Improved version specifiers and fixed conda recipe by @frgfm in #169
• docs: Fixed README badge and updated documentation by @frgfm in #170
• ci: Updated release job by @frgfm in #173
• refactor: Improved target resolution by @frgfm in #174
• ci: Updated the trigger for the release job by @frgfm in #176
• docs: Updated landing page screenshot by @frgfm in #179

Full Changelog: v0.3.1...v0.3.2

v0.3.1: Improved demo & reorganized package

This patch release adds new features to the demo and reorganizes the package for a clearer hierarchy.

Note: TorchCAM 0.3.1 requires PyTorch 1.5.1 or higher.

Highlights

CAM fusion is coming to the demo 🚀

With release 0.3.0, the support of multiple target layers was added as well as CAM fusion. The demo was updated to automatically fuse CAMs when you hooked multiple layers (add a "+" separator between each layer name):

Breaking changes

Submodule renaming

To anticipate further developments of the library, modules were renamed:

• torchcam.cams was renamed into torchcam.methods
• torchcam.cams.utils was renamed and made private (torchcam.methods._utils) since it's API may evolve quickly
• activation-based CAM methods are now implemented in torchcam.methods.activation rather than torchcam.cams.cam
• gradient-based CAM methods are now implemented in torchcam.methods.gradient rather than torchcam.cams.gradcam

0.3.0	0.3.1
>>> from torchcam.cams import LayerCAM	>>> from torchcam.methods import LayerCAM

What's Changed

• chore: Made post release modifications by @frgfm in #103
• docs: Updated changelog by @frgfm in #104
• feat: Added possibility to retrieve multiple CAMs in demo by @frgfm in #105
• refactor: Reorganized package hierarchy by @frgfm in #106
• docs: Fixed LaTeX syntax in docstrings by @frgfm in #107

Full Changelog: v0.3.0...v0.3.1

v0.3.0: Support of Layer-CAM & multi-layer CAM computation

This release extends CAM methods with Layer-CAM, greatly improves the core features (CAM computation for multiple layers at once, CAM fusion, support of torch.nn.Module), while improving accessibility for entry users.

Note: TorchCAM 0.3.0 requires PyTorch 1.5.1 or higher.

Highlights

Enters Layer-CAM

The previous release saw the introduction of Score-CAM variants, and this one introduces you to Layer-CAM, which is meant to be considerably faster, while offering very competitive localization cues!

Just like any other CAM methods, you can now use it as follows:

from torchcam.cams import LayerCAM
# model = ....
# Hook the model
cam_extractor = LayerCAM(model)

Consequently, the illustration of visual outputs for all CAM methods has been updated so that you can better choose the option that suits you:

Computing CAMs for multiple layers & CAM fusion

A class activation map is specific to a given layer in a model. To fully capture the influence of visual traits on your classification output, you might want to explore the CAMs for multiple layers.

For instance, here are the CAMs on the layers "layer2", "layer3" and "layer4" of a resnet18:

from torchvision.io.image import read_image
from torchvision.models import resnet18
from torchvision.transforms.functional import normalize, resize, to_pil_image
import matplotlib.pyplot as plt

from torchcam.cams import LayerCAM
from torchcam.utils import overlay_mask

# Download an image
!wget https://www.woopets.fr/assets/races/000/066/big-portrait/border-collie.jpg
# Set this to your image path if you wish to run it on your own data
img_path = "border-collie.jpg"

# Get your input
img = read_image(img_path)
# Preprocess it for your chosen model
input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
# Get your model
model = resnet18(pretrained=True).eval()
# Hook the model
cam_extractor = LayerCAM(model, ["layer2", "layer3", "layer4"])

out = model(input_tensor.unsqueeze(0))
cams = cam_extractor(out.squeeze(0).argmax().item(), out)
# Plot the CAMs
_, axes = plt.subplots(1, len(cam_extractor.target_names))
for idx, name, cam in zip(range(len(cam_extractor.target_names)), cam_extractor.target_names, cams):
  axes[idx].imshow(cam.numpy()); axes[idx].axis('off'); axes[idx].set_title(name);
plt.show()

Now, the way you would combine those together is up to you. By default, most approaches use an element-wise maximum. But, LayerCAM has its own fusion method:

# Let's fuse them
fused_cam = cam_extractor.fuse_cams(cams)
# Plot the raw version
plt.imshow(fused_cam.numpy()); plt.axis('off'); plt.title(" + ".join(cam_extractor.target_names)); plt.show()

# Overlay it on the image
result = overlay_mask(to_pil_image(img), to_pil_image(fused_cam, mode='F'), alpha=0.5)
# Plot the result
plt.imshow(result); plt.axis('off'); plt.title(" + ".join(cam_extractor.target_names)); plt.show()

Support of torch.nn.Module as target_layer

While making the API more robust, CAM constructors now also accept torch.nn.Module as target_layer. Previously, you had to pass the name of the layer as string, but you can now pass the object reference directly if you prefer:

from torchcam.cams import LayerCAM
# model = ....
# Hook the model
cam_extractor = LayerCAM(model, model.layer4)

⚡ Latency benchmark ⚡

Since CAMs can be used from localization or production pipelines, it is important to consider latency along with pure visual output quality. For this reason, a latency evaluation script has been included in this release along with a full benchmark table.

Should you wish to have latency metrics on your dedicated hardware, you can run the script on your own:

python scripts/eval_latency.py SmoothGradCAMpp --size 224

Notebooks ⏯️

Do you prefer to only run code rather than write it? Perhaps you only want to tweak a few things?
Then enjoy the brand new Jupyter notebooks than you can either run locally or on Google Colab!

🤗 Live demo 🤗

The ML community was recently blessed by HuggingFace with their beta of Spaces, which let you host free-of-charge your ML demos!

Previously, you were able to run the demo locally on deploy it on your own, but now, you can enjoy the live demo of TorchCAM 🎨

Breaking changes

Multiple CAM output

Since CAM extractor can now compute the resulting maps for multiple layer at a time, the return type of all CAMs has been changed from torch.Tensor to List[torch.Tensor] with N elements, where N is the number of target layers.

0.2.0	0.3.0
>>> from torchcam.cams import SmoothGradCAMpp >>> extractor = SmoothGradCAMpp(model) >>> out = model(input_tensor.unsqueeze(0)) >>> print(type(cam_extractor(out.squeeze(0).argmax().item(), out))) <class 'torch.Tensor'>	>>> from torchcam.cams import SmoothGradCAMpp >>> extractor = SmoothGradCAMpp(model) >>> out = model(input_tensor.unsqueeze(0)) >>> print(type(cam_extractor(out.squeeze(0).argmax().item(), out))) <class 'list'>

New features

CAMs

Implementations of CAM method

• Added support of conv1x1 as FC candidate in base CAM #69 (@frgfm)
• Added support of LayerCAM #77 (@frgfm)
• Added support of torch.nn.Module as target_layer or fc_layer #83 (@frgfm)
• Added support of multiple target layers for all CAM methods #89 #92 (@frgfm)
• Added layer-specific CAM fusion method #93 (@frgfm)

Scripts

Side scripts to make the most out of TorchCAM

• Added latency evaluation script #95 (@frgfm)

Test

Verifications of the package well-being before release

• Added unittests to verify that conv1x1 can be used as FC in base CAM #69 (@frgfm)
• Added unittest for LayerCAM #77 (@frgfm)
• Added unittest for gradient-based CAM method for models with in-place ops #80 (@frgfm)
• Added unittest to check support of torch.nn.Module as target_layer in CAM constructor #83 #88 (@frgfm)
• Added unittest for CAM fusion #93 (@frgfm)

Documentation

Online resources for potential users

• Added LayerCAM ref in the README and in the documentation #77 (@frgfm)
• Added CODE_OF_CONDUCT #86 (@frgfm)
• Added changelog to the documentation #91 (@frgfm)
• Added latency benchmark & GIF illustration of CAM on a video in README #95 (@frgfm)
• Added documentation of .fuse_cams method #93 (@frgfm)
• Added ref to HF Space demo in README and documentation #96 (@frgfm)
• Added tutorial notebooks and reference page in the documentation #99 #100 #101 #102 (@frgfm)

Others

Other tools and implementations

• Added class_idx & target_layer selection in the demo #67 (@frgfm)
• Added CI jobs to build on different OS & Python versions, to validate the demo, and the example script #73 #74 (@frgfm)
• Added LayerCAM to the demo #77 (@frgfm)
• Added an environment collection script #78 (@frgfm)
• Added CI check for the latency evaluation script #95 (@frgfm)

Bug fixes

CAMs

• Fixes backward hook mechanism for in-place operations #80 (@frgfm)

Documentation

• Fixed docutils version constraint for documentation building #98 (@frgfm)

Others

• Fixed CI job to build documentation #64 (@frgfm)
• Fixed Pillow version constraint #73 #84 (@frgfm)

Improvements

CAMs

• Improved weight broadcasting for all CAMs #77 (@frgfm)
• Refactored hook enabling #80 (@frgfm)
• Improved the warning message for target automatic resolution #87 #92 (@frgfm)
• Improved arg type checking for CAM constructor #88 (@frgfm)

Scripts

• Improved the layout option of the example script #66 (@frgfm)
• Refactored example script #80 #94 (@frgfm)
• Updated all scripts for support of multiple target layers #89 (@frgfm)

Test

• Updated unittests for multiple target layer support #89 (@frgfm)

Documentation

• Added latest release doc version & updated README badge #63 (@frgfm)
• Added demo screenshot in the README #67 (@frgfm)
• Updated instructions in README #89 (@frgfm)
• Improved documentation landing page #91 (@frgfm)
• Updated contribution guidelines #94 (@frgfm)
• Updated documentation requirements #99 (@frgfm)

Others

• Updated package version and fixed CI jobs to validate release publish #63 (@frgfm)
• Updated license from MIT to Apache 2.0 #70 (@frgfm)
• Refactored CI jobs #73 (@frgfm)
• Improved bug report template #78 (@frgfm)
• Updated streamlit syntax in demo #94 (@frgfm)
• Added isort config and CI job #97 (@frgfm)
• Added CI job for sanity check of the documentation build #98 (@frgfm)

Compatibility with 3D inputs and improved documentation

This release extends TorchCAM compatibility to 3D inputs, and improves documentation.

Note: TorchCAM 0.2.0 requires PyTorch 1.5.1 or higher.

Highlights

Compatibility for inputs with more than 2 spatial dimensions

The first papers about CAM methods were built for classification models using 2D (spatially) inputs. However, the latest methods can be extrapolated to higher dimension inputs and it's now live:

import torch
from torchcam.cams import SmoothGradCAMpp
# Define your model compatible with 3D inputs
video_model = ...
extractor = SmoothGradCAMpp(video_model)
# Forward your input
scores = model(torch.rand((1, 3, 32, 224, 224)))
# Retrieve the CAM
cam = extractor(scores[0].argmax().item(), scores)

Multi-version documentation

While documentation was up-to-date with the latest commit on the main branch, previously if you were running an older release of the library, you had no corresponding documentation.

As of now, you can select the version of the documentation you wish to access (stable releases or latest commit):

Demo app

Since spatial information is at the very core of TorchCAM, a minimal Streamlit demo app was added to explore the activation of your favorite models. You can run the demo with the following commands:

streamlit run demo/app.py

Here is how it renders retrieving the heatmap using SmoothGradCAMpp on a pretrained resnet18:

New features

CAMs

Implementations of CAM method

• Enabled CAM compatibility for inputs with more than 2 spatial dimensions #45 (@frgfm)
• Added support of XGradCAM #47 (@frgfm)

Test

Verifications of the package well-being before release

• Added unittests for XGradCAM #47 (@frgfm)

Documentation

Online resources for potential users

• Added references to XGradCAM in README and documentation #47 (@frgfm)
• Added multi-version documentation & added github star button #53, #54, #55, #56 (@frgfm)
• Revamped README #59 (@frgfm) focusing on short easy code snippets
• Improved documentation #60 (@frgfm)

Others

Other tools and implementations

• Added issue templates for bug report and feature request #49 (@frgfm)
• Added option to specify a single CAM method in example script #52 (@frgfm)
• Added minimal demo app #59 (@frgfm)

Bug fixes

CAMs

• Fixed automatic layer resolution on GPU #41 (@frgfm)
• Fixed backward hook warnings for Pytorch >= 1.8.0 #58 (@frgfm)

Utils

• Fixed RGBA -> RGB conversion in overlay_mask #38 (@alexandrosstergiou)

Test

• Fixed overlay_mask unittest #38 (@alexandrosstergiou)

Documentation

• Fixed codacy badge in README #46 (@frgfm)
• Fixed typo in documentation #62 (@frgfm)

Others

• Fixed CI job for conda build #34 (@frgfm)
• Fixed model mode in example script #37 (@frgfm)
• Fixed sphinx version #40 (@frgfm)
• Fixed usage instructions in README #43 (@frgfm)
• Fixed example script for local image input #51 (@frgfm)

Improvements

CAMs

• Added NaN check in gradcams #37 (@frgfm)

Test

• Added NaN check unittest for gradcam #37 (@frgfm)
• Switched from unittest to pytest #45 (@frgfm) and split test files by module

Documentation

• Updated README badges #34, illustration #39 and usage instructions #41 (@frgfm)
• Added instructions to run all CI checks locally in CONTRIBUTING #34, #45 (@frgfm)
• Updated project hierarchy description in CONTRIBUTING #43 (@frgfm)
• Added minimal code snippet in documentation #41 (@frgfm)

Others

• Updated version in setup #34 and requirements #61 (@frgfm)
• Leveraged automatic layer resolution in example script #41 (@frgfm)
• Updated CI job to run unittests #45 (@frgfm)

Automatic target layer resolution and support of IS-CAM

This release adds an implementation of IS-CAM and greatly improves interface.

Note: torchcam 0.1.2 requires PyTorch 1.1 or newer.

Highlights

CAMs

Implementation of CAM extractor
New

• Add an IS-CAM implementation #13 (@frgfm)
• Added automatic target layer resolution #32 (@frgfm)

Improvements

• Added support for submodule hooking #21 (@pkmandke)

Fixes

• Fixed hooking mechanism edge case #23 (@frgfm)

Test

Verifications of the package well-being before release
New

• Updated test for torchcam.cams #13, #30 (@frgfm)

Improvements

• Removed pretrained model loading in unittests #25 (@frgfm)
• Switched all models to eval, removed gradient when not required, and changed to simpler models #33 (@frgfm)

Documentation

Online resources for potential users
New

• Added entry for IS-CAM #13, #30 (@frgfm)

Fixes

• Fixed examples in docstrings of gradient-based CAMs #28, #33 (@frgfm)

Others

Other tools and implementations
New

• Added annotation typing to the codebase & mypy verification CI job #19 (@frgfm)
• Added package publishing verification jobs #12 (@frgfm)

Improvements

• Improved example script #15 (@frgfm)
• Optimized CI cache #20 (@frgfm)

Fixes

• Fixed coverage upload job #16 (@frgfm)
• Fixed doc deployment job #24 (@frgfm)
• Fixed conda recipe #29 (@frgfm)

Support of SmoothGradCAM++, Score-CAM and SS-CAM

This release adds implementations of SmoothGradCAM++, Score-CAM and SS-CAM.

Note: torchcam 0.1.1 requires PyTorch 1.1 or newer.

brought to you by @frgfm

Highlights

CAMs

Implementation of CAM extractor
New

• Add a SmoothGradCAM++ implementation (#4)
• Add a Score-CAM implementation (#5)
• Add a SS-CAM implementation (#11).

Improvements

• Refactor CAM extractor for better code reusability (#6)

Test

Verifications of the package well-being before release
New

• Updated test for torchcam.cams (#4, #5, #11)

Documentation

Online resources for potential users
Improvements

• Add detailed explanation of CAM computation (#8, #11)
• Add websearch referencing of documentation (#7)

Others

Other tools and implementations

• Fixed conda upload job (#3)

Class activation maps for CNN in PyTorch

This release adds implementations of CAM, GradCAM and GradCAM++.

Note: torchcam 0.1.0 requires PyTorch 1.1 or newer.

brought to you by @frgfm

Highlights

GradCAM

Implementation of gradient-based CAM extractor
New

• Add a CAM implementation (#2)
• Add Grad-CAM and Grad-CAM++ implementations (#1, #2).

Test

Verifications of the package well-being before release
New

• Add test for torchcam.cams (#1, #2)
• Add test for torschscan.utils (#1)

Documentation

Online resources for potential users
New

• Add sphinx automatic documentation build for existing features (#1, #2)
• Add contribution guidelines (#1)
• Add installation, usage, and benchmark in readme (#1, #2)

Others

Other tools and implementations

• Add ̀overlay_mask` to easily overlay mask on images (#1).