Leaky_relu -> tf.maximum(x, alpha*x)

[albertfaromatics]

I've been looking for information about the quantization and support on Leaky_relu of the coral, but I cannot see anything about it being available nor a possible future date.

Looking at the implementation of leaky_relu, and the list of supported ops, I've seen that Maximum is supported. Could we implement tf.nn.leaky_relu as tf.maximum(x, alpha * x), being alpha < 1?

Will it work after quantization + compile?

[Namburger]

@albertfaromatics I'm not sure about leaky-relu, but we do have prelu in our roadmap which is quite similar.

tf.maximum(x, alpha * x), being alpha < 1

maximum is definitely a supported op, so in theory should works!

[albertfaromatics]

@Namburger thanks for your response! We will wait then. Model with ReLU is working fine, but as we are using YOLO, I think that being able to eliminate de dying neurons problem will improve quite a bit the detection, in particular of small objects!

[Namburger]

yes, the problem with small objects is complicated, right now we're working on a pure application side solution demo where we split images into smaller chunk and applies nms to get resulting small objects. It has been working well for us, but it's not ideal

[albertfaromatics]

That was our first approach: split image in 4 subimages and do multiple inference.
Our main problem with this is that with the Coral we achieved a x25 speed inference increase but when splitting images in 4 subimages, our increase was only ~x5: not enough for our solution.

The implementation seems to be training and possible to quantize, but not tested it yet. I will report back when I test it and see if there's an improvement or not.

@Namburger I also have another question: I'm trying to do the same with mish
y = x * tanh(ln(1 * e^x))

Looking at the supported ops on coral:

• tanh appears
• both ln and e^x does not appear. Does it mean they are not supported?

Thanks for your incredible work and help!

[Namburger]

@albertfaromatics
:/ so I don't think tflite's interger quantization supported tf.math.log yet (assuming this is what you want to use in place of ln).

This page will give you a good list of what ops are supported for quantization and this page shows their roadmap.
The compiler supports are usually blocked until we at least have the quantization from tflite, so maybe this would be a good feature request