Lime class always returns empty attributions

[caesar-one]

🐛 Bug

I have tried to use the Lime class, without success, maybe I'm doing something wrong. It always returns empty attributions. I have tried with BERT, but the same happens with a very simple example. I attach the example that basically I copy-pasted from the documentation, to reproduce the behaviour.

To Reproduce

Code to reproduce the behavior:

import torch
from captum.attr import Lime

def SimpleClassifier(t):
    return torch.randn(t.shape[0], 3)

# SimpleClassifier takes a single input tensor of size Nx4x4,
# and returns an Nx3 tensor of class probabilities.
net = SimpleClassifier

# Generating random input with size 1 x 4 x 4
input = torch.randn(1, 4, 4)

# Defining Lime interpreter
lime = Lime(net)
# Computes attribution, with each of the 4 x 4 = 16
# features as a separate interpretable feature
attr = lime.attribute(input, target=1, n_samples=20000)

# attr =
# tensor([[[0., 0., 0., 0.],
#         [0., 0., 0., 0.],
#         [0., 0., 0., 0.],
#         [0., 0., 0., 0.]]])

Expected behavior

It should ideally return a tensor with attributions, at least with some non-zero values.

Environment

Describe the environment used for Captum

 - Captum / PyTorch Version (e.g., 1.0 / 0.4.0): 0.3.1 / 1.8.1
 - How you installed Captum / PyTorch (`conda`, `pip`, source): pip install captum
 - Python version: 3.9

Additional context

Other methods are working (as of now I tried Layer Integrated Gradients). Scikit-Learn is installed and working properly (version 0.24.2)

[vivekmig]

Hi @caesar-one , the issue is likely related to the default alpha of 1.0 used in Lime, the regularization used when training the Lasso surrogate interpretable model may be too high and causing all weights to go to 0. Can you try setting the interpretable model with a smaller alpha like this:

from captum._utils.models.linear_model import SkLearnLasso
lime = Lime(net, interpretable_model=SkLearnLasso(alpha=0.0))

[caesar-one]

Thank you @vivekmig ! It works :)

[aobo-y]

@vivekmig then this is a common issue. I also struggled with this for a while to figure out the reason when making the tutorial

Maybe use a simple linear as default can be less discouraging to new users. But I understand the original paper adopts linear lasso.... and we want to keep sklearn's vanilla flavor, but unfortunately its 1.0 alpha may indeed be too large most of the times.

[vivekmig]

@vivekmig then this is a common issue. I also struggled with this for a while to figure out the reason when making the tutorial

Maybe use a simple linear as default can be less discouraging to new users. But I understand the original paper adopts linear lasso.... and we want to keep sklearn's vanilla flavor, but unfortunately its 1.0 alpha may indeed be too large most of the times.

Yes, this is a good point, we've noticed similar issues in other use-cases as well, I think we should either lower the default alpha (keeping Lime) or alternatively switch the default o ordinary least squares regression.

[aobo-y]

This has been addressed in the release of version 0.4.