tflite model throwing issue while loading on Android (NCHW --> NHWC)

[sandeepangara]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hello,

I have exported the model to tflite and tried loading the model on the Android app based on TensorFlow. I am getting the following error, and I tried various ways to fix it, but I have no luck yet. anyone faced the same issue?

E/TaskJniUtils: Error getting native address of native library: task_vision_jni
java.lang.IllegalArgumentException: Error occurred when initializing ImageSegmenter: The input tensor should have dimensions 1 x height x width x 3. Got 1 x 3 x 256 x 256.
at org.tensorflow.lite.task.vision.segmenter.ImageSegmenter.initJniWithModelFdAndOptions(Native Method)
at org.tensorflow.lite.task.vision.segmenter.ImageSegmenter.access$100(ImageSegmenter.java:77)
at org.tensorflow.lite.task.vision.segmenter.ImageSegmenter$3.createHandle(ImageSegmenter.java:419)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
at org.tensorflow.lite.task.vision.segmenter.ImageSegmenter.createFromModelFdAndOptions(ImageSegmenter.java:415)
at org.tensorflow.lite.task.vision.segmenter.ImageSegmenter.createFromFileAndOptions(ImageSegmenter.java:139)
at org.tensorflow.lite.examples.imagesegmentation.ImageSegmentationHelper.setupImageSegmenter(ImageSegmentationHelper.kt:99)
at org.tensorflow.lite.examples.imagesegmentation.ImageSegmentationHelper.segment(ImageSegmentationHelper.kt:116)
at org.tensorflow.lite.examples.imagesegmentation.fragments.CameraFragment.segmentImage(CameraFragment.kt:266)
at org.tensorflow.lite.examples.imagesegmentation.fragments.CameraFragment.bindCameraUseCases$lambda-6$lambda-5(CameraFragment.kt:241)
at org.tensorflow.lite.examples.imagesegmentation.fragments.CameraFragment.$r8$lambda$XZey5CsMADmZQ6yO-iU2A2TMxzQ(Unknown Source:0)
at org.tensorflow.lite.examples.imagesegmentation.fragments.CameraFragment$$ExternalSyntheticLambda2.analyze(Unknown Source:2)
at androidx.camera.core.ImageAnalysis.lambda$setAnalyzer$2(ImageAnalysis.java:477)
at androidx.camera.core.ImageAnalysis$$ExternalSyntheticLambda0.analyze(Unknown Source:2)
at androidx.camera.core.ImageAnalysisAbstractAnalyzer.lambda$analyzeImage$0$androidx-camera-core-ImageAnalysisAbstractAnalyzer(ImageAnalysisAbstractAnalyzer.java:285)
at androidx.camera.core.ImageAnalysisAbstractAnalyzer$$ExternalSyntheticLambda1.run(Unknown Source:14)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
at java.lang.Thread.run(Thread.java:1012)
E/Image Segmentation Helper: TFLite failed to load model with error: Error getting native address of native library: task_vision_jni

Additional

No response

[github-actions]

👋 Hello @sandeepangara, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

Dear @sandeepangara,

I'm happy to help you with the issue you are facing while exporting the YOLOv5 model to tflite.

Regarding the error message you posted, it appears that the input tensor dimensions for the image segmentation model are incorrect on Android. The dimensions should be 1 x height x width x 3. It seems the shape of the input tensor you provided is incorrect: Got 1 x 3 x 256 x 256 instead of 1 x height x width x 3.

Could you please check the dimensions of the input tensor of your model and ensure that they match the dimensions required by the ImageSegmenter?

Additionally, I would suggest checking the version of TensorFlow and TensorFlow Lite that you are using, as well as the version of the Android Studio. This could help you to narrow down the issue.

Let me know if you have any further questions or concerns.

Best,
Glenn Jocher

[sandeepangara]

Hello @glenn-jocher

Thanks for replying. Yes, the issue is with the input tensor. I am using Tensorflow 2.8 and onnx version 1.131.
Is there a way to change the format in Kotlin?

I am using the following code for running the model in the Android
https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/android

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[glenn-jocher]

Hello @sandeepangara,

Great to hear that you have identified the issue with the input tensor. In order to change the format of the input tensor in Kotlin, you would need to adjust the code where you load and process the input image before passing it to the model. Specifically, you may need to rearrange the dimensions to match the required format (1 x height x width x 3).

You can use Kotlin's native array and array manipulation functions to reshape the input image array accordingly.

I would recommend reviewing the code where you process and pass the input tensor to the model in the Android app. You may need to modify this part of the code to ensure that the input tensor has the correct dimensions before invoking the model.

Let me know how it goes or if you have any more questions.

[akuma308]

@sandeepangara Did you happened to solve this ?
I am also using yolo and converted it to tflite and getting the same error.

@glenn-jocher JFYI

[glenn-jocher]

Hello @akuma308 and the second user,

If both of you are encountering the same issue with the input tensor dimensions, it's likely that the model conversion process may not be correctly setting the input shape for the TFLite model.

When converting the YOLO model to TFLite, ensure that you specify the correct input shape and data format (NHWC for TFLite) during the conversion process. If you're using a script for conversion, check the parameters related to input shape and format.

For the TensorFlow Lite model to work on Android, the input must be in NHWC format. If the model is in NCHW format, you will need to transpose the dimensions to NHWC. This can be done during preprocessing of the input image in Kotlin before passing it to the model.

Here's a simplified example of how you might transpose an image tensor from NCHW to NHWC in Kotlin:

// Assuming 'image' is a 4D FloatArray in NCHW format (1 x 3 x height x width)
val nchwImage: FloatArray = ...

// Transpose to NHWC format (1 x height x width x 3)
val nhwcImage = FloatArray(nchwImage.size)
val height = ...
val width = ...

for (n in 0 until 1) {
    for (c in 0 until 3) {
        for (h in 0 until height) {
            for (w in 0 until width) {
                val nchwIndex = n * (3 * height * width) + c * (height * width) + h * width + w
                val nhwcIndex = n * (height * width * 3) + h * (width * 3) + w * 3 + c
                nhwcImage[nhwcIndex] = nchwImage[nchwIndex]
            }
        }
    }
}

// Now 'nhwcImage' can be used as input to the TFLite model

Please note that this is just a conceptual example and you will need to adapt it to your specific use case, ensuring that the dimensions and data types match your model's requirements.

If you continue to face issues, please check the model conversion script for any potential issues with input shape specification and ensure that the TensorFlow and ONNX versions you are using are compatible with the TFLite conversion process.

[sandeepangara]

@akuma308 I couldn't fix it. I used a model from the TensorFlow repo directly. Let me know if you fix it.

@glenn-jocher I tried something like this, but it did not work for me.

[glenn-jocher]

Hello @sandeepangara,

I understand that the issue persists despite your efforts. Transposing the tensor in the correct manner is crucial, and it can be tricky to get right. If the code snippet you tried did not resolve the issue, there might be a problem with the way the transposition is being handled or with the model conversion itself.

Here are a few steps you can take to troubleshoot the issue:

1. Review Model Conversion: Ensure that during the conversion to TFLite, the input shape is explicitly set to NHWC format. The conversion script should include a step that specifies the input format.

2. Preprocessing: Double-check the preprocessing code in Kotlin to ensure that the image is being correctly reshaped to the NHWC format before being passed to the model.

3. Model Input Details: Use the TensorFlow Lite Interpreter in your Android code to check the expected input shape of the model. This can help confirm whether the model expects NHWC input.

4. TensorFlow Lite Interpreter: Here's an example of how you might use the TensorFlow Lite Interpreter to check the input shape and ensure your input tensor matches:

val tfliteModel = FileUtil.loadMappedFile(context, "your_model.tflite")
val tflite = Interpreter(tfliteModel)
val inputTensor = tflite.getInputTensor(0)
val inputShape = inputTensor.shape() // This should give you the expected input shape

// Ensure that your input image tensor matches this shape

5. Testing: If possible, test the TFLite model on a desktop environment with a known good input to ensure the model itself is functioning correctly after conversion.

6. TensorFlow Lite Support: Since the issue might be related to TensorFlow Lite's handling of the model, consider reaching out to TensorFlow Lite support or community forums for more targeted assistance.

If you continue to face difficulties, please provide more details about the steps you've taken and any error messages you're receiving. This can help in diagnosing the problem more effectively.

Keep in mind that the YOLOv5 model is complex, and converting it to TFLite for use on mobile devices can introduce challenges. Patience and careful debugging are key.

[akuma308]

@glenn-jocher

ultralytics == '8.0.145'
tensorflow==2.10.0

I trained Yolov8 and exported via
yolo export model=last.pt format=tflite and i am able to see model perfectly in netron.app .

When i am using last_float32.tflite in Kotlin i am getting error as further specified.
""Error occurred when initializing ImageSegmenter: Type mismatch for input tensor serving_default_images:0. Requested one of these types: kTfLiteUint8/kTfLiteFloat32, got INT8." .

Please let me know what are we doing wrong ?

[glenn-jocher]

@akuma308 hello,

It seems like there's a data type mismatch between what your TFLite model expects and what is being provided by your Kotlin code. The error message indicates that the model is expecting a UINT8 or FLOAT32 input tensor, but it's receiving an INT8 tensor instead.

Here are a few steps to resolve this issue:

1. Check Model Export: Ensure that when you're exporting the model using the yolo export command, you're not quantizing the model to INT8 unintentionally. If you want a FLOAT32 model, make sure not to apply any quantization that would change the data type to INT8.

2. Input Data Preparation: In your Kotlin code, make sure that the input image is being converted to the correct data type (UINT8 or FLOAT32) before being passed to the model. If you're using image data that is in UINT8 format, you don't need to do anything, but if it's in another format, you'll need to convert it.

3. TensorFlow Lite Interpreter: Use the TensorFlow Lite Interpreter in Kotlin to set the input tensor data type explicitly. Here's an example snippet:

val interpreter = Interpreter(tfliteModel)
val inputIndex = interpreter.getInputIndex("serving_default_images:0")
val inputTensor = interpreter.getInputTensor(inputIndex)

// If your model expects FLOAT32, make sure to provide a FloatArray
val input = FloatArray(requiredSize)
// ... fill 'input' with your image data in FLOAT32 format

// Run inference
interpreter.run(input, output)

4. Model Compatibility: Double-check that the YOLOv8 model you trained is compatible with the TensorFlow Lite conversion process and that the export command is generating the correct TFLite model without any quantization if that's not what you desire.

5. TensorFlow Lite Version: Ensure that the version of TensorFlow Lite you're using in your Kotlin project is compatible with the exported TFLite model. Mismatches in versions can sometimes lead to unexpected errors.

If you continue to encounter issues, please provide more details about the steps you've taken, including the exact export command and any additional flags you used, as well as the relevant portions of your Kotlin code that prepare and pass the input tensor to the model.

[akuma308]

@glenn-jocher Used below command to convert without any flags.
yolo export model=last.pt format=tflite it generated "last_float32.tflite" and below is the screenshot of the graph in netron.app .

I am assuming conversion looks fine.
When we tried in Kotlin ,we are getting below error
'''Input tensor has type kTfLiteFloat32: it requires specifying NormalizationOptions metadata to preprocess input images.'''

I understand we need to write "metadata" to the tflite model and convert again.
When i try using this tutorial
https://www.tensorflow.org/lite/models/convert/metadata_writer_tutorial#image_segmenters ,
I get error stating "The number of output tensors (2) should match the number of output tensor metadata (1)"
As i can understand we have 2 output tensor instead of 1 , and we may need to add metadata for another output tensor ?

Is there a way to bypass this or add metadata in tflite model while conversion or possible solutions.

[glenn-jocher]

Hello @akuma308,

It appears that you're facing a metadata-related issue with your TFLite model. TensorFlow Lite models can include metadata that describes the model's input and output data, which can be useful for downstream applications to understand how to preprocess and postprocess the data.

Here are some steps to address the issue:

1. Metadata Populator: You may need to use the TensorFlow Lite Metadata Populator to add the necessary metadata to your model. This includes specifying the normalization options for the input tensor and ensuring that the metadata for all output tensors is included.

2. Multiple Outputs: Since your model has multiple outputs, you'll need to add metadata for each output tensor. The TensorFlow Lite Metadata Populator allows you to add information for multiple input and output tensors.

3. Bypassing Metadata: While metadata is helpful, it is not strictly necessary for running inference with a TFLite model. You can bypass the need for metadata by manually handling the preprocessing and postprocessing in your Kotlin code. This means you'll need to normalize the input image data as expected by the model and interpret the output tensors correctly without relying on metadata.

4. Normalization Options: If the error message is indicating that you need to specify normalization options, you'll need to ensure that your input image data is normalized according to the model's expectations. This typically involves scaling pixel values to a range that the model was trained on (e.g., [0, 1] or [-1, 1]).

5. Conversion Script: There is no built-in way to add metadata during the yolo export command. You'll need to use the TensorFlow Lite Metadata Populator after the model has been converted to TFLite format.

6. Documentation: Refer to the TensorFlow Lite documentation on how to add metadata to a model. The tutorial you mentioned is a good starting point, but you'll need to adapt it to handle multiple outputs.

Here's a simplified example of how you might add metadata for multiple outputs:

from tflite_support.metadata_writers import image_segmenter
from tflite_support.metadata_writers import writer_utils

ImageSegmenterWriter = image_segmenter.MetadataWriter
_MODEL_PATH = "path/to/your/model.tflite"
_LABEL_FILE_PATH = "path/to/your/labels.txt"
_SAVE_TO_PATH = "path/to/save/the/model_with_metadata.tflite"

# Create the metadata writer.
writer = ImageSegmenterWriter.create_for_inference(
    writer_utils.load_file(_MODEL_PATH), input_norm_mean=[0], input_norm_std=[1],
    label_file_paths=[_LABEL_FILE_PATH])

# Add metadata for the second output tensor.
writer_utils.save_file(writer.populate(), _SAVE_TO_PATH)

Please note that the above code is just an example and may not work out of the box. You'll need to adjust it according to your specific model and outputs.

If you continue to face difficulties, consider reaching out to the TensorFlow Lite community for more targeted assistance with metadata population for models with multiple outputs.

[tirgei]

Hi @glenn-jocher following up on the issue raised by @akuma308 , we are using the tensorflow lite repository sample to run our model.

The issue we are facing is that we are getting the Input tensor has type kTfLiteFloat32: it requires specifying NormalizationOptions metadata to preprocess input images. error when trying to create an instance of the ImageSegmenter which can then run the analysis on input images. Here is the code sample of initialising the ImageSegmenter.

 fun segment(image: Bitmap, imageRotation: Int) {

        if (imageSegmenter == null) {
            val optionsBuilder =
                ImageSegmenter.ImageSegmenterOptions.builder()

            // Set general segmentation options, including number of used threads
            val baseOptionsBuilder = BaseOptions.builder().setNumThreads(numThreads)

            // Use the specified hardware for running the model. Default to CPU
            when (currentDelegate) {
                DELEGATE_CPU -> {
                    // Default
                }
                DELEGATE_GPU -> {
                    if (CompatibilityList().isDelegateSupportedOnThisDevice) {
                        baseOptionsBuilder.useGpu()
                    } else {
                        imageSegmentationListener?.onError("GPU is not supported on this device")
                    }
                }
                DELEGATE_NNAPI -> {
                    baseOptionsBuilder.useNnapi()
                }
            }

            optionsBuilder.setBaseOptions(baseOptionsBuilder.build())

            /*
            CATEGORY_MASK is being specifically used to predict the available objects
            based on individual pixels in this sample. The other option available for
            OutputType, CONFIDENCE_MAP, provides a gray scale mapping of the image
            where each pixel has a confidence score applied to it from 0.0f to 1.0f
             */
            optionsBuilder.setOutputType(OutputType.CATEGORY_MASK)
            try {
                imageSegmenter =
                    ImageSegmenter.createFromFileAndOptions(
                        context,
                        model.tflite,
                        optionsBuilder.build()
                    )
            } catch (e: IllegalStateException) {
                imageSegmentationListener?.onError(
                    "Image segmentation failed to initialize. See error logs for details"
                )
                Log.e(TAG, "TFLite failed to load model with error: " + e.message)
            }
        }

        // Inference time is the difference between the system time at the start and finish of the
        // process
        var inferenceTime = SystemClock.uptimeMillis()

        // Create preprocessor for the image.
        // See https://www.tensorflow.org/lite/inference_with_metadata/
        //            lite_support#imageprocessor_architecture
        val imageProcessor =
            ImageProcessor.Builder()
                .add(Rot90Op(-imageRotation / 90))
                .add(NormalizeOp(127.5f, 127.5f))
                .build()

        // Preprocess the image and convert it into a TensorImage for segmentation.
        val tensorImage = imageProcessor.process(TensorImage.fromBitmap(image))

        val segmentResult = imageSegmenter?.segment(tensorImage)
        inferenceTime = SystemClock.uptimeMillis() - inferenceTime

        imageSegmentationListener?.onResults(
            segmentResult,
            inferenceTime,
            tensorImage.height,
            tensorImage.width
        )
    }

In the code snippet we invoke this function with the bitmap of the image captured by the user and the device rotation angle (portrait vs landscape). The first step is to check whether we have an instance of the ImageSegmenter which if we don't we initialize it first and this is where it is failing.

imageSegmenter = ImageSegmenter.createFromFileAndOptions(
      context,
      model.tflite,
      optionsBuilder.build()
  )

Is there something we are missing? cc @akuma308

[glenn-jocher]

@tirgei hello,

The error you're encountering suggests that the TensorFlow Lite model you're using requires specific normalization options to be specified in its metadata for preprocessing input images. The TensorFlow Lite Task Library, which includes the ImageSegmenter, relies on this metadata to automatically preprocess the input images.

Since you're using a custom model, you'll need to ensure that the model's metadata includes the NormalizationOptions. Here's what you can do:

1. Add Metadata: Use the TensorFlow Lite Metadata Populator to add the necessary normalization options to your model's metadata. This will typically include the mean and standard deviation values used during model training for image normalization.

2. Manual Preprocessing: Alternatively, if adding metadata is not feasible, you can manually preprocess the images in Kotlin before passing them to the ImageSegmenter. This would involve normalizing the pixel values of the input Bitmap to the expected range (e.g., scaling pixel values to [0, 1] or [-1, 1] if that's what the model was trained with).

Here's an example of how you might manually preprocess the image:

val imageProcessor = ImageProcessor.Builder()
    .add(Rot90Op(-imageRotation / 90))
    // Assuming the model was trained with pixel values in range [-1, 1]
    .add(NormalizeOp(127.5f, 127.5f)) // Adjust these values based on your model's training
    .build()

val tensorImage = imageProcessor.process(TensorImage.fromBitmap(image))

In the above code, NormalizeOp(127.5f, 127.5f) is used to scale pixel values from [0, 255] to [-1, 1]. You'll need to adjust the mean and std values based on how your model was trained.

3. Check Model Training: Ensure that you know the exact normalization parameters used during the training of your YOLOv8 model. These values need to be consistent during both training and inference.

4. Update TensorFlow Lite Task Library: Make sure you're using the latest version of the TensorFlow Lite Task Library, as updates may include fixes or changes related to model metadata and preprocessing.

5. Review TensorFlow Lite Examples: Review the TensorFlow Lite examples and documentation to ensure that you're following the recommended practices for using custom models with the Task Library.

If you continue to face issues, you may need to reach out to the TensorFlow Lite community for more specific guidance on working with the Task Library and custom models with multiple outputs.

[tirgei]

@glenn-jocher thanks for the response.

For the manual preprocessing step that is currently not applicable since the app does not get to this point. It fails earlier when initialising the ImageSegmenter which we then use to process the image.

Let us explore the other options recommended. Thanks.

[glenn-jocher]

@tirgei, you're welcome.

I understand that the issue occurs during the initialization of the ImageSegmenter and not during the manual preprocessing step. The initialization fails because the model lacks the required metadata, which the ImageSegmenter expects for automatic preprocessing.

To resolve the initialization issue, you will indeed need to focus on adding the appropriate metadata to your TFLite model. Once the metadata is correctly embedded in the model, the ImageSegmenter should be able to initialize without errors, as it will have the information it needs to preprocess the input images.

If you encounter any further issues or have additional questions as you explore the recommended options, please feel free to reach out. Good luck with your implementation, and I hope you'll be able to get the ImageSegmenter up and running smoothly with your custom model.