Leaf Warning Fix #597
Leaf Warning Fix #597
Conversation
tests/attr/test_saliency.py
Outdated
| inp.grad = torch.randn_like(inp) | ||
| grad = inp.grad.detach().clone() | ||
| self._saliency_base_assert(model, inp, grads, add_args) | ||
| assertTensorTuplesAlmostEqual(self, inp.grad, grad) |
There was a problem hiding this comment.
nit: Can we, please, do exact equal here to be sure that the grads didn't change ?
It would be interesting to use backward and compare with the grads computed with the backward to contrast the difference.
There was a problem hiding this comment.
Sure, updated with delta=0.0 for exact equality. From the docs, I think backward should always compute the same gradients just also updating the grad attribute for leaf nodes, but can double-check this.
There was a problem hiding this comment.
Thank you! I was thinking that if we use backward instead autograd.grad then we will see the difference, right ?
There was a problem hiding this comment.
Makes sense, yup, if we used backward instead, this test should fail! Although backward doesn't take inputs as parameters and just requires accessing grad for the gradients, so would be a different structure from what we currently use and might fail for non-leaf inputs.
There was a problem hiding this comment.
@vivekmig has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
@vivekmig has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
This removes the resetting of grad attribute to zero, which is causing warnings as mentioned in #491 and #421 . Based on torch documentation, resetting of grad is only needed when using torch.autograd.backward, which accumulates results into the grad attribute for leaf nodes. Since we only utilize torch.autograd.grad (with only_inputs always set to True), the gradients obtained in Captum are never actually accumulated into grad attributes, so resetting the attribute is not actually necessary.
This also adds a test to confirm that the grad attribute is not altered when gradients are utilized through Saliency.