Support Quantized Tensorflow lite model

[kris-himax]

Search before asking

• I have searched the HUB issues and discussions and found no similar questions.

Question

Hi,
The environment is amazing.
But the TensorFlow Lite model is only supported floating.
We will deploy the tflite model to the devices which are resource constrained.
Could you support quantized TensorFlow Lite model with uint8 or int8?

Thanks

Additional

No response

[github-actions]

👋 Hello @kris-himax, thank you for raising an issue about Ultralytics HUB 🚀! Please visit https://ultralytics.com/hub to learn more, and see our ⭐️ HUB Guidelines to quickly get started uploading datasets and training YOLOv5 models.

If this is a 🐛 Bug Report, please provide screenshots and steps to recreate your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

[glenn-jocher]

@kris-himax thanks for the feedback! TFLite int8 models are part of our future roadmap, currently all TFLite exports are FP16.

[kris-himax]

@glenn-jocher thanks for the reply!
I am the member at the Himax Technologies, Inc (https://www.himax.com.tw/) which are researching and developing next generation ultra low power deep learning IC with M55 and U55.
We are really interested in your model with no code environment to support quantized TFlite int models.

[glenn-jocher]

@kris-himax got it! One solution you could use is to download the PyTorch model instead and export it to TFLite INT8 using YOLOv5, i.e.

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

python export.py --weights model.pt --include tflite  # FP16
python export.py --weights model.pt --include tflite --int8  # INT8

One problem we've seen though is that INT8 models do not produce identical validation results. FP32 and FP16 models will produce identical mAP, but quantizing to INT8 results in some drop, so you should definitely validate both exports first to verify you are ok with the accuracy drop.

Also if you have any ideas or solutions to fixing the INT8 validation drop please let us know, thank you!

[kris-himax]

@glenn-jocher hi,
While we try to quantize the other model which the activation function uses silu, the result of quantized int8 model is worse.
In our opinion, maybe the reason of quantization INT8 results dropping is the silu activation function.
Would you try to replace silu to other activation function (ex. relu or relu6...) and then do the quantization?

Thanks

[glenn-jocher]

@kris-himax TFLite seems to suffer from poor results at INT8, likely due to their specific quantization strategy. Other formats like CoreML suffer no significant drop in mAP when moving from FP16 to INT8.

BTW I just added the ability to add any activation to YOLOv5 models with a new activation: field in a model yaml. This works with any PyTorch activation, including ones that require arguments. See ultralytics/yolov5#9371 for more details.

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/hub/wiki
• Tutorials – https://github.com/ultralytics/hub#tutorials
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!