how to calculate inference time ?

[quocnhat]

Search before asking

• I have searched the HUB issues and discussions and found no similar questions.

Question

Hi, Thanks for your excellent job.
I've tested another great deployment with the same tflite model, But the speed is significantly slower than this (x4). May I ask where is the main gap?
Does your inference time cover all detection steps? (preprocess, inference, nms)
or Does this come from the queue of camera capture?

Additional

No response

[github-actions]

👋 Hello @quocnhat, thank you for raising an issue about Ultralytics HUB 🚀! Please visit our HUB Docs to learn more:

• Quickstart. Start training and deploying YOLO models with HUB in seconds.
• Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
• Projects: Creating and Managing. Group your models into projects for improved organization.
• Models: Training and Exporting. Train YOLOv5 and YOLOv8 models on your custom datasets and export them to various formats for deployment.
• Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
• Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
  • iOS. Learn about YOLO CoreML models accelerated on Apple's Neural Engine on iPhones and iPads.
  • Android. Explore TFLite acceleration on mobile devices.
• Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.

If this is a 🐛 Bug Report, please provide screenshots and steps to reproduce your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

[pderrenger]

@quocnhat hello,

Thank you for your kind words and for reaching out with your question! It's great to hear that you're exploring different deployment options for your YOLOv8 model.

To calculate inference time accurately, it's essential to consider all the steps involved in the detection process. Typically, the total inference time should include:

1. Preprocessing: Preparing the input data (e.g., resizing, normalization).
2. Inference: Running the model to get predictions.
3. Postprocessing: Applying non-maximum suppression (NMS) and other post-processing steps to filter and refine the predictions.

Here's a basic example of how you can measure the inference time in Python:

import time
import cv2
import numpy as np
import tensorflow as tf

# Load your TFLite model
interpreter = tf.lite.Interpreter(model_path="model.tflite")
interpreter.allocate_tensors()

# Get input and output tensors
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Load an example image
image = cv2.imread('example.jpg')
input_data = cv2.resize(image, (input_details[0]['shape'][1], input_details[0]['shape'][2]))
input_data = np.expand_dims(input_data, axis=0).astype(np.float32)

# Measure preprocessing time
start_time = time.time()
# Preprocessing steps (e.g., normalization)
input_data = input_data / 255.0
preprocess_time = time.time() - start_time

# Measure inference time
start_time = time.time()
interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
inference_time = time.time() - start_time

# Measure postprocessing time
start_time = time.time()
output_data = interpreter.get_tensor(output_details[0]['index'])
# Postprocessing steps (e.g., NMS)
# Assuming you have a function `apply_nms` for non-maximum suppression
# output_data = apply_nms(output_data)
postprocess_time = time.time() - start_time

total_time = preprocess_time + inference_time + postprocess_time

print(f"Preprocessing Time: {preprocess_time:.4f} seconds")
print(f"Inference Time: {inference_time:.4f} seconds")
print(f"Postprocessing Time: {postprocess_time:.4f} seconds")
print(f"Total Inference Time: {total_time:.4f} seconds")

Regarding the discrepancy in speed, several factors could contribute to this:

• Model Optimization: Different implementations might have varying levels of optimization.
• Hardware Utilization: Ensure that both deployments are utilizing the same hardware resources effectively.
• Framework Overheads: Different frameworks (e.g., TensorFlow Lite vs. other deployment tools) might introduce different overheads.

To get a more accurate comparison, ensure that both deployments are measured under similar conditions and that all steps (preprocessing, inference, postprocessing) are accounted for.

If you encounter any specific issues or need further assistance, please provide a minimum reproducible example as outlined here. Additionally, make sure you are using the latest versions of the packages involved.

I hope this helps! If you have any more questions, feel free to ask.

[quocnhat]

Hi, Thanks for your quick and very clear information.
Closing question now, Thank you!

[pderrenger]

Hello @quocnhat,

You're very welcome! I'm glad the information was helpful to you. 😊

If you have any more questions in the future or need further assistance, feel free to open a new issue. The YOLO community and the Ultralytics team are always here to help.

Happy coding and best of luck with your projects!