Added Hardtanh activation function #14
Conversation
Very exciting! I'll have to do some looking into this as well 🙂 |
| const long j_start, const long j_end) { | ||
| for (long j = j_start; j < j_end; j++) | ||
| for (long i = i_start; i < i_end; i++) | ||
| x(i, j) = hard_tanh_(x(i, j)); |
There was a problem hiding this comment.
I wonder if it would perform better if you used unaryExpr instead? Like so:
x = x.unaryExpr([](float in) {return hard_tanh(in);});
Seems to do better according to: https://godbolt.org/z/zfePz8MTa
There was a problem hiding this comment.
Well, you don't want to update the entire matrix, so some x.middleCols() are needed
There was a problem hiding this comment.
I wonder if it would perform better if you used unaryExpr instead? Like so:
x = x.unaryExpr([](float in) {return hard_tanh(in);});
Agreed - I just copy/pasted a duplicate of the tanh function structure, but didn't worry about it for now as that code path isn't used by WaveNet - it just uses the full matrix overload and I've already optimized those (for both tanh and hardtanh).
My plan is to refactor the activation code in general soon, but I wanted to get the hardtanh in there asap so we can start think about targeting it with training models.
There was a problem hiding this comment.
Btw, I tried using unaryExpr(), and if I recall it performed slightly worse than just rolling over the data directly.
That Compiler Explorer tool you linked to is pretty cool.
This adds support for the "Hardtanh" activation function in WaveNet models. It will have no impact on current models.
My (admittedly limited so far) testing indicates that using a hard tanh activation function (basically clamp to -1/1) results in the same ESR as using a regular tanh. But it is much faster to compute.
It seems to perform about the same as the fast tanh function that is currently disabled, but might be able to be further optimized with SSE magic. It also has the benefit of being available as an activation fuction in pytorch.
If we get it implemented here, now - once it has been out in the wild for a bit we can switch to using it in training (maybe by default, maybe an option, maybe only on lite/feather?)
If you want to test training a model with it, just swap out "Tanh" for "Hardtanh" (lowercase "t") in the WaveNet layer config.