Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TCAV score computation doesn't preserve experimental set order #1285

Open
pris0 opened this issue May 24, 2024 · 0 comments
Open

TCAV score computation doesn't preserve experimental set order #1285

pris0 opened this issue May 24, 2024 · 0 comments

Comments

@pris0
Copy link

pris0 commented May 24, 2024

🐛 Bug

After creating a TCAV instance, when one calls the .interpret() method, the yielded dictionary of scores don't follow the order of the experimental sets given in input. This creates confusion.

To Reproduce

Steps to reproduce the behavior:

  1. Define some experimental sets. In my case I had 25 pairs of "base concept - random concept j" for j = 0, .. 24 for every base concept. The order is: base concept 0 - random concept 0, base concept 0 - random concept 1, ... base concept 0 - random concept 24, base concept 1 - random concept 0, base concept 1 - random concept 1, .. etc
  2. Create a TCAV instnce mytcav = TCAV( model=model, layers=["conv1", "conv2"], layer_attr_method=LayerIntegratedGradients(model, None, multiply_by_inputs=False),)
  3. Compute scores using the interpret function:
    scores = mytcav.interpret(inputs=input_class, experimental_sets=experimental_set_rand, target=target_id,)
  4. Compare the order of the experimental sets and of the produced scores.

Expected behavior

The keys of the scores' dictionary should follow the order of the experimental sets given in input.

  • Captum / PyTorch Version (e.g., 1.0 / 0.4.0): 0.7.0
  • OS (e.g., Linux): Linux
  • How you installed Captum / PyTorch (conda, pip, source): conda
  • Python version: 3.12.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant