Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistency in TCAV: Order of experimental sets #909

Closed
carlomarxdk opened this issue Mar 22, 2022 · 2 comments
Closed

Inconsistency in TCAV: Order of experimental sets #909

carlomarxdk opened this issue Mar 22, 2022 · 2 comments
Assignees
@NarineK

Comments

@carlomarxdk
Copy link

carlomarxdk commented Mar 22, 2022 •
edited
Loading

🐛 Bug

Ordering of concepts in the experemental_sets drastically changes the output of interpret (e.g. it is not consistent)

To reproduce:

Steps to reproduce:

  1. Run the "Show cases Testing with Concept Activation Vectors (TCAV) on Imagenet Dataset and GoogleNet model" official ttutorial
  2. Change the ordering of sets
  3. Run interpret

If you change the order of experemental_sets:
experimental_set_zig_dot = [[zigzagged_concept, dotted_concept, stripes_concept]] or
experimental_set_zig_dot = [[dotted_concept, stripes_concept, zigzagged_concept]] you get absolutely different results. In first case, indeed zigzagged concept has the largest magnitude score; however, in the second case, the dotted concept is going to have the highest magnitude score.

Expected behaviour

I would expect that the change of the ordering would give same results.
@carlomarxdk carlomarxdk changed the title TCAV: Order of experimental sets Inconsistency in TCAV: Order of experimental sets Mar 22, 2022
@NarineK
Copy link
Contributor

NarineK commented Apr 1, 2022

@carlomarxdk , thank you for pointing out to this issue. There was a bug related to the ordering of returned tcav scores. I'll send out the fix soon.

@NarineK NarineK self-assigned this Apr 1, 2022
NarineK added a commit to NarineK/captum-1 that referenced this issue Apr 1, 2022
@NarineK @facebook-github-bot
Fixes TCAV concept ordering bug reported in issue 909 on GH
d7c82fe 
Summary:
Fixes TCAV concept ordering bug reported in this issue: pytorch#909

The original implementation wasn't sorting the concepts correctly in the output. I changed so that the ordering is based on the input concept ordering in the experimental set.

Differential Revision: D35302278

fbshipit-source-id: 61fab8e24db586af514356d83843ab1bd4d026e3
@NarineK NarineK mentioned this issue Apr 1, 2022
Closed
facebook-github-bot pushed a commit that referenced this issue Apr 19, 2022
@NarineK @facebook-github-bot
Fixes TCAV concept ordering bug reported in issue 909 on GH (#915)
1c5bd8a 
Summary:
Pull Request resolved: #915

Fixes TCAV concept ordering bug reported in this issue: #909

The original implementation wasn't sorting the concepts correctly in the output. I changed so that the ordering is based on the input concept ordering in the experimental set.

Reviewed By: vivekmig

Differential Revision: D35302278

fbshipit-source-id: 82657184f42d9841a5adf56ef103150be5ef6146
@NarineK
Copy link
Contributor

NarineK commented Apr 20, 2022

@carlomarxdk, we just merged, #915. Let us know if this helps to address the issue

@NarineK NarineK closed this as completed Dec 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@NarineK NarineK

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@NarineK @carlomarxdk