TFlite output data

[Tommydw]

❔Question

Hi, I have successfully trained a custom model based on YOLOv5s and converted the model to TFlite. I feel silly asking, but how do you use the output data?

I get as output:

• StatefulPartitionedCall: 0 = [1,25200,7]
  from the converted YOLOv5 model
  

But I expect an output like:

• StatefulPartitionedCall:3 = [1, 10, 4] # boxes
• StatefulPartitionedCall:2 = [1, 10] # classes
• StatefulPartitionedCall:1 = [1, 10] #scores
• StatefulPartitionedCall:0 = [1] #count
  (this one is from a tensorflow lite mobilenet model (trained to give 10 output data, default for tflite))
  

It may also be some other form of output, but I honestly have no idea how to get the boxes, classes, scores from a [1,25200,7] array.
Can anyone help me with this??
(on 15-January-2021 I updated pytorch, tensorflow and yolov5 to the latest version)

[github-actions]

👋 Hello @Tommydw, thank you for your interest in 🚀 YOLOv5! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://www.ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab Notebook with free GPU:
• Kaggle Notebook with free GPU: https://www.kaggle.com/ultralytics/yolov5
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Docker Image https://hub.docker.com/r/ultralytics/yolov5. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[Tommydw]

The data contained in the [1, 25200, 7] array can be found in this file: outputdata.txt

0.011428807862102985, 0.006756599526852369, 0.04274776205420494, 0.034441519528627396, 0.00012877583503723145, 0.33658933639526367, 0.4722323715686798
0.023071227595210075, 0.006947836373001337, 0.046426184475421906, 0.023744791746139526, 0.0002465546131134033, 0.29862138628959656, 0.4498370885848999
0.03636947274208069, 0.006819264497607946, 0.04913407564163208, 0.025004519149661064, 0.00013208389282226562, 0.3155967593193054, 0.4081345796585083
0.04930267855525017, 0.007249316666275263, 0.04969717934727669, 0.023645592853426933, 0.0001222355494974181, 0.3123127520084381, 0.40113094449043274
...

Should I add a Non Max Suppression or something else, can someone help me please?

[glenn-jocher]

@Tommydw thresholding and NMS are not part of the TFLite model.

The TFLite_detection_posprocess module you show is only available for SSD I believe, this is the first time I've seen it for mobilenet. I've had discussions with Google about helping us apply it to YOLOv5 but the implementation is a bit detailed and we have not made recent progress on it, so currently TFLite models require additional NMS.

[glenn-jocher]

@Tommydw where did you get your mobilenet model from with the included TFLite_detection_postprocess?

The column labels by the way to get you started are [xywh, conf, class0, class1, ...]
https://github.com/zldrobit/yolov5/blob/c761637b51701565a9efbd585a05f093f8aa5f41/models/tf.py#L328

[Tommydw]

The column labels by the way to get you started are [xywh, conf, class0, class1, ...]
https://github.com/zldrobit/yolov5/blob/c761637b51701565a9efbd585a05f093f8aa5f41/models/tf.py#L328

@glenn-jocher Thank you very much!
Now I can continue to experimenting XD

where did you get your mobilenet model from with the included TFLite_detection_postprocess?

I downloaded the SSD MobileNet V2 FPNLite 320x320 from the TF2 Model ZOO and trained it with Tensorflow 2.4, exported it with "export_tflite_graph_tf2.py" and converted it to TFLite with "convert-to-tflite.py" in TF-nightly (2.5.0-dev20210120)

[glenn-jocher]

Ah, ok yes it's SSD based then. As the note mentions the NMS pipeline only works with SSD models today I believe:
https://github.com/armaanpriyadarshan/Training-a-Custom-TensorFlow-2.X-Object-Detector/blob/75ed99b2f1c51a43ed17df2b670443b9cc5c105d/workspace/training_demo/export_tflite_graph_tf2.py#L21

[Tommydw]

@glenn-jocher the code works now (if it is good), but my tflite model detects objects very badly. Also when I convert the yolov5s.pt in the same and other ways

My current code:

    """Output data"""
    output_data = interpreter.get_tensor(output_details[0]['index'])  # get tensor  x(1, 25200, 7)
    output_data = output_data[0]  # x(1, 25200, 7) to x(25200, 7)
    xywh = output_data[..., :4]  # boxes  [25200, 4]
    conf = output_data[..., 4:5]  # confidences  [25200, 1]
    cls = tf.reshape(tf.cast(tf.argmax(output_data[..., 5:], axis=1), tf.float32), (-1, 1))  # classes  x(25200, 1) 
    output = np.squeeze(tf.concat([conf, cls, xywh], 1))  #  [25200, 1], x(25200, 1), [25200, 4] to [25200, 6] (confidences, classes, x, y, w, h)

    scores = output[..., 0]  # scores [25200]
    classes = output[..., 1]  # classes [25200]
    boxes = output[..., 2:]  # boxes [25200, 4]
    # Convert nx4 boxes from [x, y, w, h] to [x1, y1, x2, y2] where xy1=top-left, xy2=bottom-right
    x, y, w, h = boxes[..., 0], boxes[..., 1], boxes[..., 2], boxes[..., 3] #xywh
    xyxy = [x - w / 2, y - h / 2, x + w / 2, y + h / 2]  # xywh to xyxy   [25200, 4]

[glenn-jocher]

@Tommydw TFLite models produce the same outputs as their PyTorch counterparts. We've verified this using detect.py directly with the TFLite models on #1127

[Tommydw]

@glenn-jocher I found the problem, I converted the input data to 1.0 to -1.0 instead of 1.0 to 0.0 😅 Works fine now, thanks!

[AsmaJegham]

I got a quite similar output after converting my custom trained Yolov5 to tflite. Can you please explain more how to get to :
StatefulPartitionedCall:3 = [1, 10, 4] # boxes
StatefulPartitionedCall:2 = [1, 10] # classes
StatefulPartitionedCall:1 = [1, 10] #scores
StatefulPartitionedCall:0 = [1] #4248

[Tommydw]

@AsmaJegham I wrote the solution for my problem at stack overflow: https://stackoverflow.com/a/65953473/15050934
I hope it also helps you?

[AsmaJegham]

what did you exactly do ? did you add these functions to the tf.py file and rerun the conversion or took the existing outputs and somehow gave them as an input to these functions ? sorry, my question might look a bit silly for you.

[yahoouser2021]

@Tommydw Your solution really helped me. May i know if you have implemented nms on the same piece of code?

[ebdjesus]

@Tommydw I have the same problem, this solution really help us. If you could give more details, I would be very grateful.

[chainyo]

@Tommydw Are you using your tflite model with Android Studio ?

Because I need to add metadata to mine in order to import it in Android Studio and it seems that tflite-support (the package used to add metadata to tflite model file) doesn't handle models with only one output tensor...

I'm trying to build my own script to add metadata to the model, but if you already did it, maybe you could share it here :)

[mmsumapas]

@Tommydw where did you get the output_details?

"output_data = interpreter.get_tensor(output_details[0]['index'])"

[Tommydw]

@Tommydw where did you get the output_details?

"output_data = interpreter.get_tensor(output_details[0]['index'])"

output_details = interpreter.get_output_details()

[mmsumapas]

@Tommydw where did you get the output_details?
"output_data = interpreter.get_tensor(output_details[0]['index'])"

output_details = interpreter.get_output_details()

thank you @Tommydw

[ebdjesus]

@mmsumapas Hi bro, good morning!
Did it work for you?
Could share which file moved and line?
Thank you so much

[mmsumapas]

I just followed the code provided by @Tommydw. I implemented it in raspberry pi 4 and the code works perfectly.

…

On Mon, Jan 3, 2022 at 10:03 PM ebdjesus ***@***.***> wrote: @mmsumapas <https://github.com/mmsumapas> Hi bro, good morning! Did it work for you? Could share which file moved and line? Thank you so much — Reply to this email directly, view it on GitHub <#1981 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKZNX4B4LBD66DZKHQCDPODUUGUD7ANCNFSM4WIR3V5A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[ebdjesus]

@mmsumapas In my case I'm using on android mobile generating tflite file. But thanks for the reply I will try to follow what @Tommydw did
Thank you so much

[AnkitMyelin]

what did you exactly do ? did you add these functions to the tf.py file and rerun the conversion or took the existing outputs and somehow gave them as an input to these functions ? sorry, my question might look a bit silly for you.

@AsmaJegham How did you resolve your issue ? I'm facing same issue and didn't able to figure it out .

[matinmoezzi]

Hi @Tommydw
Could you please explain in which file you inserted your code snippet? Thanks.

[d3ath-add3r]

@glenn-jocher I found the problem, I converted the input data to 1.0 to -1.0 instead of 1.0 to 0.0 😅 Works fine now, thanks!

@Tommydw Thank You Man, Your solution helped a lot. I was searching for the solution for long time. But I am facing a problem, hoping you can help. I am getting multiple bounding boxes for same object, What can be done?

[glenn-jocher]

@d3ath-add3r seems like you need to apply NMS to your results.

[vyang2968]

    output_data = interpreter.get_tensor(output_details[0]['index'])  # get tensor  x(1, 25200, 7)
    output_data = output_data[0]  # x(1, 25200, 7) to x(25200, 7)
    xywh = output_data[..., :4]  # boxes  [25200, 4]
    conf = output_data[..., 4:5]  # confidences  [25200, 1]
    cls = tf.reshape(tf.cast(tf.argmax(output_data[..., 5:], axis=1), tf.float32), (-1, 1))  # classes  x(25200, 1) 
    output = np.squeeze(tf.concat([conf, cls, xywh], 1))  #  [25200, 1], x(25200, 1), [25200, 4] to [25200, 6] (confidences, classes, x, y, w, h)

    scores = output[..., 0]  # scores [25200]
    classes = output[..., 1]  # classes [25200]
    boxes = output[..., 2:]  # boxes [25200, 4]
    # Convert nx4 boxes from [x, y, w, h] to [x1, y1, x2, y2] where xy1=top-left, xy2=bottom-right
    x, y, w, h = boxes[..., 0], boxes[..., 1], boxes[..., 2], boxes[..., 3] #xywh
    xyxy = [x - w / 2, y - h / 2, x + w / 2, y + h / 2]  # xywh to xyxy   [25200, 4]`

Is there any way to replicate @Tommydw 's code without using tf math methods? Can we substitute, for example, numpy.concatenate for tf.concat?

[smtalds]

Hi , i want to ask something , i want to inference tflite with c++ interpreter , how can i be sure get true output sorting ? Is the same python ? xywh conf class ? Or are there any example for c++ microcontroller ? Thanks for answering.

[ryecries]

hi i want to ask something, is anyone succesfully create a model from custom data with yolov5, convert it to tflite and using it in android using flutter framework?

[ryecries]

❔Question

Hi, I have successfully trained a custom model based on YOLOv5s and converted the model to TFlite. I feel silly asking, but how do you use the output data?

I get as output:

• StatefulPartitionedCall: 0 = [1,25200,7]
  from the converted YOLOv5 model
  

But I expect an output like:

• StatefulPartitionedCall:3 = [1, 10, 4] # boxes
• StatefulPartitionedCall:2 = [1, 10] # classes
• StatefulPartitionedCall:1 = [1, 10] #scores
• StatefulPartitionedCall:0 = [1] #count
  (this one is from a tensorflow lite mobilenet model (trained to give 10 output data, default for tflite))
  

It may also be some other form of output, but I honestly have no idea how to get the boxes, classes, scores from a [1,25200,7] array. Can anyone help me with this?? (on 15-January-2021 I updated pytorch, tensorflow and yolov5 to the latest version)

hi, can i ask what script you use to convert yolov5 model to tflite? pls i want to use tflite model in flutter

[zldrobit]

@ryecries plz refer to the document of exporting models.

[glenn-jocher]

@ryecries, to convert a YOLOv5 model to TFLite format, you can follow the steps outlined in the document on exporting models. This document provides detailed instructions on how to export your trained YOLOv5 model to TFLite. Once you have the TFLite model, you can use it in your Flutter project by referring to the Flutter TFLite plugin's documentation on how to load and run the TFLite model within your app.

[RohanEmpire]

@Tommydw @glenn-jocher @AsmaJegham @zldrobit @matinmoezzi @ovshake @mmsumapas @

Hi Guys Please give me some time and Help Me

i hv trained yolov5s model and converted into tflite which is giving 1 output array when interpreted on an image
also i hv trained a ssd mobilenet model of tensorflowzoo and converted into tflite which is giving 4 arrays

please help to convert yolov5 to tflite to get 4 arrays

@Tommydw i hv seen your solution in stackoverflow didnt get it how to do and where to do in flow

please help me and let me know the flow and process and code when to convert ,I mean inbetween yolo to tflite or after tflite

please help

the export of yolov5s to tflite should give 4 arrays ........because i will upload tflite in ai camera

[glenn-jocher]

@RohanEmpire hello! It sounds like you're looking for a way to modify the output of your YOLOv5 TFLite model to match the output format of an SSD MobileNet model. The YOLOv5 model typically outputs a single tensor with the shape [1, num_boxes, 7] where each box is represented by [x_center, y_center, width, height, confidence, class1_prob, class2_prob, ...]. On the other hand, SSD models often output multiple tensors for boxes, classes, scores, and the number of detections.

To achieve this, you would need to modify the post-processing step of your YOLOv5 TFLite model. This involves interpreting the single output tensor and splitting it into the desired four arrays. This is not a straightforward export option and requires custom code to be written.

Here's a high-level overview of the steps you might take:

1. Interpret YOLOv5 Output: After running inference with the TFLite interpreter, you'll get a single output tensor. You'll need to write code to interpret this tensor and extract the bounding box coordinates, confidence scores, and class probabilities.

2. Convert to Desired Format: Once you have the raw data from the YOLOv5 output, you'll need to convert it into the four separate arrays that you're looking for. This will involve thresholding the confidence scores to determine the count of detections and then populating the arrays with the corresponding data.

3. Implement in TFLite Model: If you want these four arrays to be the direct output of the TFLite model, you would need to implement this post-processing as custom operations within the TFLite graph. This is advanced and typically requires knowledge of TensorFlow's graph operations.

4. Custom Wrapper or Function: A simpler approach might be to keep the TFLite model as is and write a custom wrapper or function in your application code that performs this post-processing after inference.

Please note that this is a non-trivial task and requires a good understanding of both the YOLOv5 output format and TensorFlow/TFLite operations. If you're not familiar with these concepts, it might be helpful to work with someone who has experience in deep learning and TensorFlow to assist you with this task.

Remember that the YOLOv5 repository and documentation are great resources, and you can often find help and advice by engaging with the community there. Good luck with your project!

[zldrobit]

@RohanEmpire Maybe you could refer to the repo. confidence, class, and bbox are extracted from the output of the tflite model, as follows:
https://github.com/zldrobit/yolov5/blob/e796283b53a6d4198a2b7067bfa976b542395d25/android/app/src/main/java/org/tensorflow/lite/examples/detection/tflite/YoloV5ClassifierDetect.java#L421-L451

[RohanEmpire]

@RohanEmpire hello! It sounds like you're looking for a way to modify the output of your YOLOv5 TFLite model to match the output format of an SSD MobileNet model. The YOLOv5 model typically outputs a single tensor with the shape [1, num_boxes, 7] where each box is represented by [x_center, y_center, width, height, confidence, class1_prob, class2_prob, ...]. On the other hand, SSD models often output multiple tensors for boxes, classes, scores, and the number of detections.

To achieve this, you would need to modify the post-processing step of your YOLOv5 TFLite model. This involves interpreting the single output tensor and splitting it into the desired four arrays. This is not a straightforward export option and requires custom code to be written.

Here's a high-level overview of the steps you might take:

1. Interpret YOLOv5 Output: After running inference with the TFLite interpreter, you'll get a single output tensor. You'll need to write code to interpret this tensor and extract the bounding box coordinates, confidence scores, and class probabilities.
2. Convert to Desired Format: Once you have the raw data from the YOLOv5 output, you'll need to convert it into the four separate arrays that you're looking for. This will involve thresholding the confidence scores to determine the count of detections and then populating the arrays with the corresponding data.
3. Implement in TFLite Model: If you want these four arrays to be the direct output of the TFLite model, you would need to implement this post-processing as custom operations within the TFLite graph. This is advanced and typically requires knowledge of TensorFlow's graph operations.
4. Custom Wrapper or Function: A simpler approach might be to keep the TFLite model as is and write a custom wrapper or function in your application code that performs this post-processing after inference.

Please note that this is a non-trivial task and requires a good understanding of both the YOLOv5 output format and TensorFlow/TFLite operations. If you're not familiar with these concepts, it might be helpful to work with someone who has experience in deep learning and TensorFlow to assist you with this task.

Remember that the YOLOv5 repository and documentation are great resources, and you can often find help and advice by engaging with the community there. Good luck with your project!

Thanks
@glenn-jocher

The camera where i need to upload tflite model with take only 4 arrays as input boxes,confidence,class,num_detections

initially i use to upload ssd mobnet tflite into cam which gives 4 arrays as output

the results are very bad with ssd mobnet where yolo gives me very good results

so i need to get a readymade tflite which me 4 arrays same like ssd so that i can upload to camera( camera takes tflite which give only 4 arrays ouput)
readymade tflite which give me 4 arrays on an image
beacuse my camera will give 4 arrays as ouput so it takes tflite as input which gives 4 arrays as ouput(location,confidence,class,num_detections)

Implement in TFLite Model: If you want these four arrays to be the direct output of the TFLite model, you would need to implement this post-processing as custom operations within the TFLite graph. This is advanced and typically requires knowledge of TensorFlow's graph operations

HELP ME IN MODIFYING THE FILES REQUIRED TO CONVERT TFLITE(READYMADE TFLITE THAT GIVES 4 ARRAYS(LOCATION,CONFIDENCE,CLASS,NUM_DETECTIONS)

please give me your time ,and help me which files i need to modify and run tflite lite conversion code so that i can get camera required tflite
thanks

[RohanEmpire]

@RohanEmpire Maybe you could refer to the repo. confidence, class, and bbox are extracted from the output of the tflite model, as follows: https://github.com/zldrobit/yolov5/blob/e796283b53a6d4198a2b7067bfa976b542395d25/android/app/src/main/java/org/tensorflow/lite/examples/detection/tflite/YoloV5ClassifierDetect.java#L421-L451

@zldrobit really thanks for your time

but i need a readymade tflite which give me 4 arrays on an image
beacuse my camera will give 4 arrays as ouput so it takes tflite as input which gives 4 arrays as ouput(location,confidence,class,num_detections)

thanks

[RohanEmpire]

@glenn-jocher Hey
Num_detections of tflite(yolo to tflite) on an image im getting 25000 ..how can i control it 10 for example?

[glenn-jocher]

Hello @RohanEmpire,

The num_detections you're seeing is related to the total number of predictions made by the model before any filtering. To control the number of detections, you'll need to apply Non-Maximum Suppression (NMS) and a confidence threshold to filter out less likely predictions and reduce the number of final detections.

Here's a brief outline of what you need to do:

1. Apply a Confidence Threshold: Filter out detections that have a confidence score lower than a certain threshold. This will remove a lot of low-confidence predictions.

2. Apply NMS: Use Non-Maximum Suppression to eliminate overlapping boxes. This will keep only the best bounding box when multiple boxes are detected for the same object.

3. Limit the Number of Detections: After applying the confidence threshold and NMS, you can sort the remaining detections by confidence and keep only the top N detections, where N is the number you want (e.g., 10).

This post-processing needs to be done after you run inference with the TFLite model and before you interpret the results. If you want to integrate this directly into the TFLite model, it would require custom operations, which is quite complex.

For a camera system that requires a TFLite model outputting a fixed number of detections, you would typically handle this in the post-processing step within the camera's software. If the camera's software is not modifiable, you might need to create a custom TFLite model with integrated post-processing, which is an advanced task.

If you're not familiar with these operations, I would recommend working with someone who has experience in TensorFlow and TFLite to help you modify your model accordingly.

[RohanEmpire]

Hello @RohanEmpire,

The num_detections you're seeing is related to the total number of predictions made by the model before any filtering. To control the number of detections, you'll need to apply Non-Maximum Suppression (NMS) and a confidence threshold to filter out less likely predictions and reduce the number of final detections.

Here's a brief outline of what you need to do:

1. Apply a Confidence Threshold: Filter out detections that have a confidence score lower than a certain threshold. This will remove a lot of low-confidence predictions.
2. Apply NMS: Use Non-Maximum Suppression to eliminate overlapping boxes. This will keep only the best bounding box when multiple boxes are detected for the same object.
3. Limit the Number of Detections: After applying the confidence threshold and NMS, you can sort the remaining detections by confidence and keep only the top N detections, where N is the number you want (e.g., 10).

This post-processing needs to be done after you run inference with the TFLite model and before you interpret the results. If you want to integrate this directly into the TFLite model, it would require custom operations, which is quite complex.

For a camera system that requires a TFLite model outputting a fixed number of detections, you would typically handle this in the post-processing step within the camera's software. If the camera's software is not modifiable, you might need to create a custom TFLite model with integrated post-processing, which is an advanced task.

If you're not familiar with these operations, I would recommend working with someone who has experience in TensorFlow and TFLite to help you modify your model accordingly.

@glenn-jocher Hey Thanks
Actually Im applying nms while converting pt to tflite
so when i infer on a image the result 4 ouput arrays are getting jumbled

[glenn-jocher]

Hello @RohanEmpire,

If the outputs are getting jumbled after applying NMS during the conversion from .pt to .tflite, it's possible that there might be an issue with the conversion process or the post-processing code.

To resolve this, you should:

1. Review the Conversion Code: Ensure that the conversion script correctly applies NMS and confidence thresholding. The conversion process should preserve the order of operations as it is in the original model.

2. Check Post-Processing: After inference, make sure that the post-processing steps are correctly implemented. The outputs should be sorted and filtered in a consistent manner.

3. Debugging: Run the model on a set of test images and carefully check the intermediate steps of your post-processing to see where the mix-up might be happening.

4. Consistency: Ensure that the NMS and confidence thresholding are applied in the same way during both training and inference.

If you're still facing issues, you might need to step through your code with a debugger or add logging to better understand where the outputs are getting mixed up.

Remember, if the camera system you're working with has specific requirements for the model output, it's crucial that the model's post-processing aligns with these requirements. If the camera software cannot be modified, you may need to adapt your model's output format to match the camera's input expectations.

Since this can be quite complex, if you're not experienced with these processes, consider reaching out to the community or collaborating with someone who has expertise in model conversion and TFLite to assist you further.

[RohanEmpire]

Hello @RohanEmpire,

If the outputs are getting jumbled after applying NMS during the conversion from .pt to .tflite, it's possible that there might be an issue with the conversion process or the post-processing code.

To resolve this, you should:

1. Review the Conversion Code: Ensure that the conversion script correctly applies NMS and confidence thresholding. The conversion process should preserve the order of operations as it is in the original model.
2. Check Post-Processing: After inference, make sure that the post-processing steps are correctly implemented. The outputs should be sorted and filtered in a consistent manner.
3. Debugging: Run the model on a set of test images and carefully check the intermediate steps of your post-processing to see where the mix-up might be happening.
4. Consistency: Ensure that the NMS and confidence thresholding are applied in the same way during both training and inference.

If you're still facing issues, you might need to step through your code with a debugger or add logging to better understand where the outputs are getting mixed up.

Remember, if the camera system you're working with has specific requirements for the model output, it's crucial that the model's post-processing aligns with these requirements. If the camera software cannot be modified, you may need to adapt your model's output format to match the camera's input expectations.

Since this can be quite complex, if you're not experienced with these processes, consider reaching out to the community or collaborating with someone who has expertise in model conversion and TFLite to assist you further.

@glenn-jocher Thanks Brother

[glenn-jocher]

You're welcome, @RohanEmpire! If you have any more questions or need further assistance, feel free to reach out. Best of luck with your project! 😊👍

[RohanEmpire]

!cd yolo && python export.py --weights "C:\Users\Desktop\YOLO_25\HELMET_JACKET_TF_25\yolo\runs\train\exp\weights\best.pt" --img 640 --data dataset.yaml --include tflite

Output Array 0:
[[[ 0.00361872 0.00522317 0.03131891 0.0228551 ]
[ 0.00417319 -0.01452317 0.02296825 0.04225811]
[-0.00829614 -0.00607556 0.03775904 0.02981148]
...
[ 0.90323806 0.8960819 1.0104626 1.0134667 ]
[ 0.8346647 0.8692871 1.0523373 1.046476 ]
[ 0.7383431 0.70022464 1.2261554 1.2492236 ]]]
Output Array 1:
[[1. 2. 2. ... 3. 3. 0.]]
Output Array 2:
[[1.6012788e-04 6.1989995e-05 3.1957163e-05 ... 1.3974309e-04
8.0671816e-05 1.6522408e-04]]
Output Array 3:
[25200.]

!cd yolo && python export.py --weights "C:\Users\Desktop\YOLO_25\HELMET_JACKET_TF_25\yolo\runs\train\exp\weights\best.pt" --img 640 --nms --data dataset.yaml --include tflite

Output Array 0:
[[[0.4320571 0.16799404 0.47489187 0.20888616]
[0.40980852 0.23323964 0.5173153 0.37525183]
[0.5744248 0.23921803 0.6130417 0.27299497]
[0. 0. 0. 0. ]
[0. 0. 0. 0. ]
[0. 0. 0. 0. ]
[0. 0. 0. 0. ]
[0. 0. 0. 0. ]
[0. 0. 0. 0. ]
[0. 0. 0. 0. ]]]
Output Array 1:
[[0.47958136 0.45514858 0.29846266 0. 0. 0.
0. 0. 0. 0. ]]
Output Array 2:
[[3. 1. 3. 0. 0. 0. 0. 0. 0. 0.]]
Output Array 3:
[3]

@glenn-jocher When I include nms in the code im getting good results but ouput order is getting changed
ouput 1 and output 4 are ok but output 2 and 3 getting swaped

with out nms im getting right order but box upon box issue
with nms no box upon box issue but order is getting jumbled

I need location,class,score,num_detections

Please help

[glenn-jocher]

Hello @RohanEmpire,

It seems like the output order is not matching your expectations when NMS is included. The typical output order for object detection models is [boxes, scores, classes, num_detections]. If the outputs are getting swapped, it's likely an issue with the export script or the post-processing code within the TFLite conversion.

Here's what you can do:

1. Review Export Script: Check the export.py script to ensure that the outputs are being set in the correct order during the TFLite conversion process.

2. Post-Processing: If the export script seems correct, review the post-processing code that's applied after inference. Ensure that the outputs are being interpreted and sorted in the correct order.

3. Manual Adjustment: As a temporary fix, you can manually adjust the order of the outputs in your application code after inference to match the expected [boxes, scores, classes, num_detections] format.

4. Consistency Check: Make sure that the NMS is being applied consistently and that the output tensors are being handled in the same way across different parts of the code.

If you continue to face issues, you may need to debug the export process step by step to identify where the discrepancy is occurring. Since this involves digging into the code, if you're not comfortable with it, consider reaching out to the community or someone with experience in model conversion for assistance.

Remember to keep backups of your original working scripts so you can always revert to a known good state if needed.

[zldrobit]

@RohanEmpire TFLite >=2.7 does change the order of output tensors. According to tensorflow/tensorflow#33303 (comment), this is a regression of TFLite >=2.7. BTW, what's the version of TFLite you are using?

[RohanEmpire]

@glenn-jocher Thanks
@zldrobit im using tensorflow 2.5 in one env
By default tf 2.5 gvies the order i want but after applying nms the order is getting changed

giving jumbled ouputs

I want location,class,scores,num_detections order(tf 2.5 default order)
after nms im getting loca,score,class,detections.........a[1],a[2] getting swapped

[glenn-jocher]

Hello @RohanEmpire,

It appears that the issue you're encountering with the output order after applying NMS is related to a known behavior in TensorFlow Lite. Since you're using TensorFlow 2.5 and experiencing a change in the output order after NMS, it's likely due to the way the TFLite conversion or the NMS operation is implemented.

Here's what you can do to address this:

1. Custom Post-Processing: After running inference with the TFLite model, you can manually reorder the outputs in your application code to match the desired [location, class, scores, num_detections] format.

2. Modify Export Script: If possible, modify the export.py script to enforce the output order during the conversion process. This might involve explicitly setting the order of the output tensors or adjusting the model's output layers.

3. TensorFlow Version: Although you're using TensorFlow 2.5, you might want to experiment with different versions of TensorFlow to see if the issue persists. Sometimes, upgrading or downgrading the TensorFlow version can resolve unexpected behaviors.

4. Report the Issue: If you believe this is a regression or an unintended side effect of NMS in TensorFlow Lite, consider reporting it to the TensorFlow repository so that the maintainers are aware and can provide a fix or guidance.

Remember to test your changes thoroughly to ensure that the output order is consistent across different inputs and scenarios. If you're still having trouble, you may need to seek further assistance from the TensorFlow community or from someone with deep expertise in TensorFlow Lite conversions.

[RohanEmpire]

@glenn-jocher Thanks for your time

[glenn-jocher]

You're welcome, @RohanEmpire! If you have any more questions in the future or need further assistance, don't hesitate to ask. Good luck with your project! 😊🚀

[mp051998]

Select TensorFlow op(s), included in the given model, is(are) not supported by this interpreter. Make sure you apply/link the Flex delegate before inference. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-lite-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/guide/ops_select
E/tflite  (24127): Node number 402 (FlexCombinedNonMaxSuppression) failed to prepare.
E/flutter (24127): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Invalid argument(s): Unable to create interpreter.
E/flutter (24127): #0      checkArgument (package:quiver/check.dart:45:5)
E/flutter (24127): #1      new Interpreter._create (package:tflite_flutter/src/interpreter.dart:58:5)
E/flutter (24127): #2      new Interpreter.fromBuffer (package:tflite_flutter/src/interpreter.dart:109:37)
E/flutter (24127): #3      Interpreter.fromAsset (package:tflite_flutter/src/interpreter.dart:126:24)
E/flutter (24127): <asynchronous suspension>
E/flutter (24127): #4      Detector.start (package:my_app/modules/trial_object_detector.dart:166:7)
E/flutter (24127): <asynchronous suspension>
E/flutter (24127): #5      _RecordVideoState.initState.<anonymous closure> (package:my_app/screens/record_video.dart:73:27)
E/flutter (24127): <asynchronous suspension>
E/flutter (24127):

I exported the model using --nms.
I exported the yolov5n.pt model with --nms flag, but I get this error when I try to load it into the interpreter.
I followed the link, but it's somewhat confusing, not sure what I need to do. Any of you know what to do?

Note: This is in flutter.

@RohanEmpire @zldrobit @glenn-jocher

[glenn-jocher]

@mp051998 hello,

The error message indicates that the TensorFlow Lite interpreter is encountering an operation (in this case, FlexCombinedNonMaxSuppression) that it doesn't natively support. The TensorFlow Lite Flex delegate allows you to use certain TensorFlow ops that are not included in the standard TensorFlow Lite runtime.

Here's what you can do:

1. Flex Delegate: If you haven't already, you'll need to include the TensorFlow Lite Flex delegate in your application. This will enable the use of the TensorFlow ops that are not natively supported by TFLite.

2. Dependency: For Android, as mentioned in the error message, you should add the "org.tensorflow:tensorflow-lite-select-tf-ops" dependency to your build.gradle file to include the necessary TensorFlow ops.

3. Interpreter Options: When creating the TFLite interpreter in your Flutter code, make sure to apply the Flex delegate. This typically involves setting the appropriate interpreter options before initializing the interpreter.

4. Model Conversion: If possible, consider re-exporting your model without using TensorFlow ops that require the Flex delegate. This can simplify deployment and potentially improve performance.

5. Documentation: Review the TensorFlow Lite documentation on ops select (Flex delegate) for detailed instructions on how to include and use the Flex delegate in your project.

If you're still having trouble after trying these steps, you may need to seek further assistance from the TensorFlow Lite or Flutter communities, as they might have more specific guidance for your use case.

[mp051998]

Hi @glenn-jocher,

Thanks for the reply. I was under the impression the nms was important and it had to be handled at export time itself (since I'm a beginner at all this), but yes, I ended up writing a function using the regular output and converted it into the format I required. It's working now I think.

Also, I may be wrong but I don't think the FlexDelegate support is there on flutter tflite.

[glenn-jocher]

Hello @mp051998,

I'm glad to hear that you've managed to resolve the issue by writing a custom function to format the output as needed. Handling NMS at export time can be convenient, but it's not strictly necessary, and performing NMS post-inference gives you more control over the process.

Regarding the FlexDelegate, you are correct that TensorFlow Lite for Flutter might not support all TensorFlow ops out of the box, and the FlexDelegate integration may not be as straightforward as in other environments. It's great that you found a workaround that doesn't require the use of unsupported ops.

If you have any more questions or run into further issues, feel free to reach out. Best of luck with your project! 🌟

[mahirk27]

@Tommydw can you provide full python code with nms?

[glenn-jocher]

Hello @mahirk27,

Certainly! Here's a simple example of how you can apply Non-Maximum Suppression (NMS) using PyTorch:

import torch

def nms(boxes, scores, iou_threshold):
    # Sort scores and corresponding boxes
    idxs = scores.argsort(descending=True)
    keep = []
    while idxs.numel() > 0:
        # Take the box with the highest score
        max_score_idx = idxs[0]
        max_score_box = boxes[max_score_idx]
        keep.append(max_score_idx.item())
        
        # Compute IoU of the remaining boxes with the max score box
        other_boxes = boxes[idxs[1:]]
        ious = bbox_iou(max_score_box.unsqueeze(0), other_boxes)
        
        # Remove boxes with IoU above the threshold
        idxs = idxs[1:][ious < iou_threshold]
    
    return keep

# Example usage
boxes = torch.tensor([[10, 10, 50, 50], [20, 20, 40, 40], [30, 30, 60, 60]])
scores = torch.tensor([0.9, 0.8, 0.7])
iou_threshold = 0.1

keep_idxs = nms(boxes, scores, iou_threshold)
print("Boxes to keep:", keep_idxs)

This is a basic implementation. For production, you might want to use optimized libraries like torchvision which include a built-in NMS function. Hope this helps! 😊