Got error when export TFlite with NMS

[chinaandylee]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

When I export tflite model with opt --nms
python export.py --weights yolov5s.pt --include tflite --nms
I got the following error:
.....
tf.expand_dims (TFOpLambda) (1, 25200, 1, 4) 0 tf.concat[0][0]

---

tf.math.multiply (TFOpLambda) (1, 25200, 2) 0 tf.operators.getitem_1[0][0]
tf.operators.getitem_2[0][0]

---

tf.image.combined_non_max_suppr CombinedNonMaxSuppre 0 tf.expand_dims[0][0]
tf.math.multiply[0][0]

Total params: 7,015,519
Trainable params: 0
Non-trainable params: 7,015,519

---

2022-02-27 14:40:41.858021: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider a
voiding using them.
Found untraced functions such as tf_conv_2_layer_call_fn, tf_conv_2_layer_call_and_return_conditional_losses, tf_conv_3_layer_call_fn, tf_conv_3_layer_call_and
_return_conditional_losses, tf_conv_4_layer_call_fn while saving (showing 5 of 520). These functions will not be directly callable after loading.
Found untraced functions such as tf_conv_2_layer_call_fn, tf_conv_2_layer_call_and_return_conditional_losses, tf_conv_3_layer_call_fn, tf_conv_3_layer_call_and
_return_conditional_losses, tf_conv_4_layer_call_fn while saving (showing 5 of 520). These functions will not be directly callable after loading.
Assets written to: last_saved_model\assets
TensorFlow SavedModel: export success, saved as last_saved_model (238.6 MB)

TensorFlow Lite: starting export with tensorflow 2.4.0...
Found untraced functions such as tf_conv_2_layer_call_fn, tf_conv_2_layer_call_and_return_conditional_losses, tf_conv_3_layer_call_fn, tf_conv_3_layer_call_and
_return_conditional_losses, tf_conv_4_layer_call_fn while saving (showing 5 of 520). These functions will not be directly callable after loading.
Found untraced functions such as tf_conv_2_layer_call_fn, tf_conv_2_layer_call_and_return_conditional_losses, tf_conv_3_layer_call_fn, tf_conv_3_layer_call_and
_return_conditional_losses, tf_conv_4_layer_call_fn while saving (showing 5 of 520). These functions will not be directly callable after loading.
Assets written to: C:\Users\kk\AppData\Local\Temp\tmp7pgu55vr\assets
2022-02-27 14:41:35.278574: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2022-02-27 14:41:35.281165: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2022-02-27 14:41:35.283499: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-02-27 14:41:35.297874: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:928] Optimization results for grappler item: graph_to_optimize
function_optimizer: function_optimizer did nothing. time = 0.001ms.
function_optimizer: function_optimizer did nothing. time = 0ms.

2022-02-27 14:41:36.286075: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:316] Ignored output_format.
2022-02-27 14:41:36.288583: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:319] Ignored drop_control_dependency.
2022-02-27 14:41:36.360661: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
loc(callsite("model/tf.image.combined_non_max_suppression/combined_non_max_suppression/CombinedNonMaxSuppression"("C:\Users\kk\Anaconda3\envs\pytorch_gpu\lib\s
ite-packages\tensorflow\python\eager\def_function.py":1205:0) at callsite("C:\Users\kk\Anaconda3\envs\pytorch_gpu\lib\site-packages\tensorflow\python\eager\def
_function.py":1299:0 at callsite("C:\Users\kk\Anaconda3\envs\pytorch_gpu\lib\site-packages\tensorflow\lite\python\lite.py":847:0 at callsite("export.py":324:0
at callsite("export.py":488:0 at callsite("C:\Users\kk\Anaconda3\envs\pytorch_gpu\lib\site-packages\torch\autograd\grad_mode.py":28:0 at callsite("export.py":5
39:0 at "export.py":544:0)))))))): error: 'tf.CombinedNonMaxSuppression' op is neither a custom op nor a flex op
error: failed while converting: 'main': Ops that can be supported by the flex runtime (enabled via setting the -emit-select-tf-ops flag):
tf.CombinedNonMaxSuppression {clip_boxes = false, device = "", pad_per_class = false}

TensorFlow Lite: export failure: C:\Users\kk\Anaconda3\envs\pytorch_gpu\lib\site-packages\tensorflow\python\eager\def_function.py:1205:0: error: 'tf.CombinedNo
nMaxSuppression' op is neither a custom op nor a flex op
C:\Users\kk\Anaconda3\envs\pytorch_gpu\lib\site-packages\tensorflow\python\eager\def_function.py:1299:0: note: called from
C:\Users\kk\Anaconda3\envs\pytorch_gpu\lib\site-packages\tensorflow\lite\python\lite.py:847:0: note: called from
export.py:324:0: note: called from
export.py:488:0: note: called from
C:\Users\kk\Anaconda3\envs\pytorch_gpu\lib\site-packages\torch\autograd\grad_mode.py:28:0: note: called from
export.py:539:0: note: called from
export.py:544:0: note: called from
### :0: error: failed while converting: 'main': Ops that can be supported by the flex runtime (enabled via setting the -emit-select-tf-ops flag):
tf.CombinedNonMaxSuppression {clip_boxes = false, device = "", pad_per_class = false}

Export complete (65.58s)
Results saved to E:\python_resource\code_resource\pytorch\yolov5-master
Detect: python detect.py --weights last_saved_model
PyTorch Hub: model = torch.hub.load('ultralytics/yolov5', 'custom', 'last_saved_model')
Validate: python val.py --weights last_saved_model
Visualize: https://netron.app

Any help will be appreciated!

Additional

No response

[github-actions]

👋 Hello @chinaandylee, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[leeflix]

You should be able to fix this by changing line 323 in export.py from:

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]

to:

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]

[chinaandylee]

According to your suggestion, I revised that line to converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS], but the problem remains. See the following for details.
TensorFlow Lite: export failure: C:\Users\kk\Anaconda3\envs\pytorch_gpu\lib\site-packages\tensorflow\python\eager\def_function.py:1205:0: error: 'tf.CombinedNo
nMaxSuppression' op is neither a custom op nor a flex op

[leeflix]

Can you please double check that you changed the correct line, because the line appears twice in the file.

[chinaandylee]

I have rechecked the revised lines. but the error still remain.

[glenn-jocher]

You should be able to fix this by changing line 323 in export.py from:

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]

to:

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]

@leeflix should we make this a permanent change in the export code? Will this affect non-nms exports?

[leeflix]

@glenn-jocher CombinedNonMaxSuppression (--nms) and NonMaxSuppression (--agnostic-nms) are both Supported Select TensorFlow operators according to this. This suggests the changes I proposed. Not adding them seems to produce this issue so I would make this a permanent change in the export code. I made a PR #7281 that won't affect non-nms exports.

[leeflix]

@chinaandylee Mh, that is strange. Maybe make a fresh clone from my fork that tries to fix the issue and try again.

[chinaandylee]

With your fork, it works! Thanks!

[glenn-jocher]

@chinaandylee @leeflix good news 😃! Your original issue may now be fixed ✅ in PR #7281 by @leeflix. To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

[Euijune]

@glenn-jocher

Hi, I understand that when I use export.py to change the .pt file to a .tflite file, if I give --nms option, the output comes in the form of nmsed_boxes (1,100), nmsed_scores=(1,100), nmsed_classes (1,100), valid_detections (1,100).

However, when I checked the output of tflite through detect.py, only nmsed_boxes existed, and scores, classes, and valid_detections were not printed.

The tflite model calculates the output properly, but did I make a mistake in the output?
Or is it just the original nmsed_boxes?

[Euijune]

@glenn-jocher

The way I checked the output of the .tflite file is as follows.
Line 122 of detect.py: pred = model (im, augment=augment, visualize=visualize)
print(pred) and print(pred.shape) on the next line

result: pred shape: (1, 100, 4) <-- where is nmsed_score, nmsed_class, nmsed_valid detection??

[AntoineChauviere]

Hello! Did you find how to apply detect.py after exporting with --nms?

[glenn-jocher]

@AntoineChauviere detect.py works with all exports as long as they don't have NMS, as it applies torch NMS to all outputs.

[AntoineChauviere]

Ok thank you @glenn-jocher, how can I test an export model with nms then please?

[glenn-jocher]

@AntoineChauviere you'd have to write some custom code.

You can always test exports following the usage example after export, and of course we also run export benchmarks every 24 hours as part of our CI:

https://github.com/ultralytics/yolov5/actions/runs/3071491167

[AntoineChauviere]

Ok thanks @glenn-jocher I'll try that then.

And last question, is it normal that I have 7 outputs? I thought I only had 4.

[glenn-jocher]

@AntoineChauviere Yes, for YOLOv5 the model typically produces 7 outputs: nmsed_boxes, nmsed_scores, nmsed_classes, valid_detections, output1, output2, output3.