clip_gradient with clip_grad_value

[dhkim0225]

🚀 Feature

Same issue with #4927 #5456

The current clip_gradient uses clip_grad_norm; can we add clip_grad_value?

https://github.com/PyTorchLightning/pytorch-lightning/blob/f2e99d617f05ec65fded81ccc6d0d59807c47573/pytorch_lightning/plugins/native_amp.py#L63-L65

============================================================
@tchaton

As far as I know, there is a difference between clip_grad_by_value and clip_grad_by_norm.
All of the implementations in PL only use clip_grad_by_norm.
clip_grad_by_value does not perform clipping with norm value but just performs clipping by value, so it is useful when learning model with noisy data.
Please let me know if you think I'm wrong.

pytorch clip by norm link
pytorch clip by value link

Sincerely,
Anthony Kim.

[dhkim0225]

This blog is useful. https://neptune.ai/blog/understanding-gradient-clipping-and-how-it-can-fix-exploding-gradients-problem

[priancho]

Moved the following comment from #5456 since reopening issue was disabled :-)

---

Hi @tchaton,

There are two popular gradient clipping methods: one that limits the maximum gradient value of each model parameter and the other one that scales the gradient value based on the p-norm of a (sub-)set of model parameters.

PyTorch Lightning implements the second option which can be used with Trainer's gradient_clip_val parameter as you mentioned.
This clipping algorithm is useful when the norm of gradients is large, but not when only a small sub-set of model parameters have abnormal gradient values since the norm will still be reasonably small considering the number of all model parameters.
I think that @dhkim0225 wants an alternative gradient clipping method that uses torch.nn.utils.clip_grad_value_() instead of torch.nn.utils.clip_grad_norm_() for such a scenario.

By the way, I don't think that this functionality is something that can break BC.
What about to add gradient_clip_algorithm parameter to Trainer, which is by default 'norm' but can be set to 'value'?

[tchaton]

Hey @priancho @dhkim0225 ,

Yes, I understand now. Sounds like a great idea ! Would @priancho or @dhkim0225 want to make a PR for such feature ?

Best,
T.C

[dhkim0225]

@tchaton Okay, I will.
@priancho If you want to develop this feature together, please feel free to reach out to me.

[dhkim0225]

@tchaton Sorry for bothering you. This is the first time to contribute a huge open-source project.
I made a PR for this. I would like to ask for your advice on the next things to do.