A question about the captum.metrics.infidelity

[bachml]

As shown in the example of API reference:

net = ImageClassifier()
saliency = Saliency(net)
input = torch.randn(2, 3, 32, 32, requires_grad=True)
Computes saliency maps for class 3.
attribution = saliency.attribute(input, target=3)
define a perturbation function for the input
def perturb_fn(inputs):
noise = torch.tensor(np.random.normal(0, 0.003, inputs.shape)).float()
return noise, inputs - noise
Computes infidelity score for saliency maps
infid = infidelity(net, perturb_fn, input, attribution)

As you can see, if we multiply attribution with a constant, such like attribution = saliency.attribute(input, target=3)*100, then we will have larger infidelity results.

This means the result of infidelity is sensitive to the norm of the attribution map.

But I retry the experiment by the author's implementation(the author who proposed this metric in NIPS19, https://github.com/chihkuanyeh/saliency_evaluation), seems like the author's infidelity implementation is robust to the norm of the attribution map.

So captum's implementation is different from the author's implementation?

[NarineK]

Thank you for the question, @bachml! I think the reason is because in their implementation the authors of the paper scale the explanation sum by a factor as you can see here:
https://github.com/chihkuanyeh/saliency_evaluation/blob/44a66e2531f30b803be3bf5b0786971b7e7f72a1/infid_sen_utils.py#L296
That scaling factor isn't included in the original paper that's why we haven't included it in our implementation: https://arxiv.org/pdf/1901.09392.pdf
We could potentially perform that type of scaling based on an input flag as well.

[NarineK]

Closing, since it got addressed.