Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Toy Example breaks with CUDA on compute_convergence_delta for Integrated Gradients #163

Closed
suragnair opened this issue Nov 7, 2019 · 1 comment
Closed

Toy Example breaks with CUDA on compute_convergence_delta for Integrated Gradients #163

suragnair opened this issue Nov 7, 2019 · 1 comment
Assignees
@vivekmig
Labels
bug Something isn't working triaged

Comments

@suragnair
Copy link

suragnair commented Nov 7, 2019 •
edited
Loading

For the toy example with cuda

model = ToyModel()
model = model.cuda()
model.eval()

input = torch.rand(2, 3).cuda()
baseline = torch.zeros(2, 3).cuda()

ig = IntegratedGradients(model)
attributions, delta = ig.attribute(input, baseline, target=0, return_convergence_delta=True)

fails with the error

~/anaconda3/envs/heterokaryon/lib/python3.7/site-packages/captum/attr/_utils/attribution.py in compute_convergence_delta(self, attributions, start_point, end_point, target, additional_forward_args)
    232         row_sums = [_sum_rows(attribution) for attribution in attributions]
    233         attr_sum = torch.tensor([sum(row_sum) for row_sum in zip(*row_sums)])
--> 234         return attr_sum - (end_point - start_point)
    235 
    236 

RuntimeError: expected device cpu and dtype Float but got device cuda:0 and dtype Float

presumably since attr_sum is not on GPU. Turning return_convergence_delta to False results in no error.

Similar issues may arise in other places, though I haven't checked.
@suragnair suragnair changed the title Toy Example (and others) break with CUDA on compute_convergence_delta Toy Example breaks with CUDA on compute_convergence_delta Nov 7, 2019
@suragnair suragnair changed the title Toy Example breaks with CUDA on compute_convergence_delta Toy Example breaks with CUDA on compute_convergence_delta Nov 7, 2019
@suragnair suragnair changed the title Toy Example breaks with CUDA on compute_convergence_delta Toy Example breaks with CUDA on compute_convergence_delta for Integrated Gradients Nov 7, 2019
@vivekmig
Copy link
Contributor

vivekmig commented Nov 7, 2019

Hi @suragnair, thanks for pointing out this bug! We will push a fix for this soon.

@vivekmig vivekmig self-assigned this Nov 7, 2019
@vivekmig vivekmig added bug Something isn't working triaged labels Nov 7, 2019
@vivekmig vivekmig mentioned this issue Nov 11, 2019
Closed
facebook-github-bot pushed a commit that referenced this issue Nov 12, 2019
@vivekmig @facebook-github-bot
Fixing CUDA delta issue (#169)
5737de1 
Summary:
Convergence delta was failing on GPUs since the total attribution tensor was not being created on the same device, this fixes that bug. Also, adds data parallel / CUDA tests for computing deltas whenever available.

Issue: #163
Pull Request resolved: #169

Differential Revision: D18440711

Pulled By: vivekmig

fbshipit-source-id: 5b67ba75492eb3c6933d13e1e67914bf1d3e5241
miguelmartin75 pushed a commit to miguelmartin75/captum that referenced this issue Nov 20, 2019
@vivekmig @miguelmartin75
Fixing CUDA delta issue (pytorch#169)
9adde47 
Summary:
Convergence delta was failing on GPUs since the total attribution tensor was not being created on the same device, this fixes that bug. Also, adds data parallel / CUDA tests for computing deltas whenever available.

Issue: pytorch#163
Pull Request resolved: pytorch#169

Differential Revision: D18440711

Pulled By: vivekmig

fbshipit-source-id: 5b67ba75492eb3c6933d13e1e67914bf1d3e5241
miguelmartin75 pushed a commit to miguelmartin75/captum that referenced this issue Dec 20, 2019
@vivekmig @miguelmartin75
Fixing CUDA delta issue (pytorch#169)
070f62b 
Summary:
Convergence delta was failing on GPUs since the total attribution tensor was not being created on the same device, this fixes that bug. Also, adds data parallel / CUDA tests for computing deltas whenever available.

Issue: pytorch#163
Pull Request resolved: pytorch#169

Differential Revision: D18440711

Pulled By: vivekmig

fbshipit-source-id: 5b67ba75492eb3c6933d13e1e67914bf1d3e5241
miguelmartin75 pushed a commit to miguelmartin75/captum that referenced this issue Dec 20, 2019
@vivekmig @miguelmartin75
Fixing CUDA delta issue (pytorch#169)
9bf9797 
Summary:
Convergence delta was failing on GPUs since the total attribution tensor was not being created on the same device, this fixes that bug. Also, adds data parallel / CUDA tests for computing deltas whenever available.

Issue: pytorch#163
Pull Request resolved: pytorch#169

Differential Revision: D18440711

Pulled By: vivekmig

fbshipit-source-id: 5b67ba75492eb3c6933d13e1e67914bf1d3e5241
NarineK pushed a commit to NarineK/captum-1 that referenced this issue Nov 19, 2020
@vivekmig @facebook-github-bot
Fixing CUDA delta issue (pytorch#169)
902c5c9 
Summary:
Convergence delta was failing on GPUs since the total attribution tensor was not being created on the same device, this fixes that bug. Also, adds data parallel / CUDA tests for computing deltas whenever available.

Issue: pytorch#163
Pull Request resolved: pytorch#169

Differential Revision: D18440711

Pulled By: vivekmig

fbshipit-source-id: 5b67ba75492eb3c6933d13e1e67914bf1d3e5241
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vivekmig vivekmig

Labels
bug Something isn't working triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@suragnair @vivekmig