NVIDIA Jetson Nano Deployment - Ultralytics YOLOv8 Docs

[giscus[bot]]

NVIDIA Jetson Nano Deployment - Ultralytics YOLOv8 Docs

📚 This guide explains how to deploy a trained model into NVIDIA Jetson Platform and perform inference using TensorRT and DeepStream SDK. Here we use TensorRT to maximize the inference performance on the Jetson platform.
UPDATED 18 November 2022.

https://docs.ultralytics.com/yolov5/tutorials/running_on_jetson_nano/

[sergii-matiukha]

How to make INT8 engine for yolov8 model with imgsz=3040, for use in deepstream python app on jetson? And how much does the accuracy drop when switching from Fp32 to INT8?

[pderrenger]

@sergii-matiukha to create an INT8 engine for your YOLOv8 model with an image size of 3040 for use in a DeepStream Python app on Jetson, you'll need to follow these general steps:

1. Prepare Calibration Data: Collect a representative dataset for calibration. This dataset should reflect the diversity of input that the model will encounter in production.

2. Convert Model to ONNX: Export your YOLOv8 model to ONNX format with the desired image size. You can do this using the export mode in YOLOv8.

   from ultralytics import YOLO
   
   # Load your trained YOLOv8 model
   model = YOLO('path/to/your/model.pt')
   
   # Export the model to ONNX format
   model.export(format='onnx', imgsz=(3040, 3040), dynamic=False)

3. Build INT8 Engine: Use TensorRT's Python API to convert the ONNX model to a TensorRT engine with INT8 precision. During this process, you'll need to provide the calibration dataset to the TensorRT builder.

   import tensorrt as trt
   
   # Set up the TensorRT logger, builder, and network
   TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
   builder = trt.Builder(TRT_LOGGER)
   network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
   
   # Define the builder configurations
   config = builder.create_builder_config()
   config.max_workspace_size = 1 << 30  # 1GB
   config.set_flag(trt.BuilderFlag.INT8)
   config.int8_calibrator = YourCalibratorClass(calibration_data)  # Implement the calibrator class
   
   # Parse the ONNX model
   parser = trt.OnnxParser(network, TRT_LOGGER)
   with open('path/to/your/model.onnx', 'rb') as model_file:
       parser.parse(model_file.read())
   
   # Build the TensorRT engine
   engine = builder.build_engine(network, config)

4. Integrate with DeepStream: Once you have the TensorRT engine, you can integrate it with your DeepStream Python app by loading the engine and using it for inference.

As for the accuracy drop when switching from FP32 to INT8, it can vary depending on the model and the dataset. Typically, INT8 quantization can lead to a slight decrease in accuracy due to the reduced numerical precision. However, TensorRT provides advanced calibration techniques to minimize this loss. The actual impact on accuracy can only be determined empirically by running evaluations on your validation set with both the FP32 and INT8 models.

For detailed instructions and code examples, please refer to the Ultralytics documentation on model export and TensorRT optimization.

[MadhavKrishna1]

Can you please provide the code for 4th step also "Integrate with DeepStream" as an example.

[pderrenger]

Hi @MadhavKrishna1,

Certainly! Below is an example of how you can integrate the TensorRT engine with a DeepStream Python app. This example assumes you have already created the TensorRT engine as described in the previous steps.

1. Install DeepStream Python Bindings: Ensure you have the DeepStream Python bindings installed. You can follow the installation instructions from the DeepStream Python Apps repository.

2. DeepStream Python App Integration: Here is a basic example of how to load the TensorRT engine and use it for inference within a DeepStream pipeline.

import pyds
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject

# Initialize GStreamer
Gst.init(None)

# Create the pipeline
pipeline = Gst.Pipeline()

# Create elements
source = Gst.ElementFactory.make('filesrc', 'file-source')
h264parser = Gst.ElementFactory.make('h264parse', 'h264-parser')
decoder = Gst.ElementFactory.make('nvv4l2decoder', 'nvv4l2-decoder')
streammux = Gst.ElementFactory.make('nvstreammux', 'Stream-muxer')
pgie = Gst.ElementFactory.make('nvinfer', 'primary-inference')
nvvidconv = Gst.ElementFactory.make('nvvideoconvert', 'nvvideo-converter')
nvosd = Gst.ElementFactory.make('nvdsosd', 'nv-onscreendisplay')
sink = Gst.ElementFactory.make('nveglglessink', 'nvvideo-renderer')

# Set properties
source.set_property('location', 'path/to/your/video.mp4')
streammux.set_property('width', 3040)
streammux.set_property('height', 3040)
streammux.set_property('batch-size', 1)
streammux.set_property('batched-push-timeout', 4000000)
pgie.set_property('config-file-path', 'path/to/config_infer_primary_yoloV8.txt')

# Add elements to the pipeline
pipeline.add(source)
pipeline.add(h264parser)
pipeline.add(decoder)
pipeline.add(streammux)
pipeline.add(pgie)
pipeline.add(nvvidconv)
pipeline.add(nvosd)
pipeline.add(sink)

# Link the elements
source.link(h264parser)
h264parser.link(decoder)

# Link the decoder to the streammux
sinkpad = streammux.get_request_pad('sink_0')
srcpad = decoder.get_static_pad('src')
srcpad.link(sinkpad)

# Link the remaining elements
streammux.link(pgie)
pgie.link(nvvidconv)
nvvidconv.link(nvosd)
nvosd.link(sink)

# Create an event loop and feed GStreamer bus messages to it
loop = GObject.MainLoop()
bus = pipeline.get_bus()
bus.add_signal_watch()
bus.connect('message', bus_call, loop)

# Start the pipeline
pipeline.set_state(Gst.State.PLAYING)

try:
    loop.run()
except:
    pass

# Cleanup
pipeline.set_state(Gst.State.NULL)

This script sets up a basic DeepStream pipeline that reads a video file, decodes it, performs inference using the TensorRT engine, and displays the results. Make sure to adjust the paths and properties according to your setup.

For more detailed information, you can refer to the DeepStream SDK documentation.

If you encounter any issues, please ensure you are using the latest versions of the packages and provide a reproducible example if the problem persists. You can find more information on creating a minimum reproducible example here.

Happy coding! 😊

[slopezba]

Hello, I am trying to follow the tutorial, with a new jetson nano and the SDK developer kit and I got this error when installing the requirements

pip3 install -r requirements.txt

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see pypa/pip#5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement gitpython>=3.1.30 (from versions: 0.1.7, 0.2.0b1, 0.3.0b1, 0.3.0b2, 0.3.1b2, 0.3.2rc1, 0.3.2, 0.3.2.1, 0.3.3, 0.3.4, 0.3.5, 0.3.6, 0.3.7, 1.0.0, 1.0.1, 1.0.2, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.7, 2.0.8, 2.0.9.dev0, 2.0.9.dev1, 2.0.9, 2.1.0, 2.1.1, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, 2.1.8, 2.1.9, 2.1.10, 2.1.11, 2.1.12, 2.1.13, 2.1.14, 2.1.15, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.0.6, 3.0.7, 3.0.8, 3.0.9, 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.1.4, 3.1.5, 3.1.6, 3.1.7, 3.1.8, 3.1.9, 3.1.10, 3.1.11, 3.1.12, 3.1.13, 3.1.14, 3.1.15, 3.1.16, 3.1.17, 3.1.18, 3.1.19, 3.1.20)
ERROR: No matching distribution found for gitpython>=3.1.30
Thank you for your help :)

[shanek16]

Jetson nano supports JetPack 4.6 which default python version is 3.6. However current yolov5 repository supports python 3.7>=.

One way is to try python3.8 and pip3.8.(However failed)
I tried installing python 3.8 and pip3.8

• installed python3.8 with
  '''sudo apt install python3.8
  sudo apt-get install python3.8-dev'''
• installed pip3.8 with
  '''curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
  python3.7 get-pip.py'''
  solving other errors through installing testresources, etc, '''pip3.8 install -r requirements.txt'''. succeeded finally.

However when installing torchvision, I encoutered "OSError: libcublas.so.11: cannot open shared object file: No such file or directory"
Which is related to CUDA.
Therefore using 3.8 for JetPack 4.6 supported Jetson nano cause compatibility issues.

Second choice is to use https://github.com/fei4xu/yolov5-python3.6.9-jetson
which has been modified to work with Python 3.6.9 on Jetson devices.
Installed
'''sudo apt-get install libopenmpi-dev
sudo apt-get install libopenblas-dev'''
before installing torchvision and followed the tutorial.
However when installing torchvision: '''sudo python3 setup.py install'''
I got Illegal instruction error.
This looks like a problem that occurs when the code is compiled for a different architecture(CPU) than the one it is being run on.

Now I got stuck here..
If anyone figured out how to follow this tutorial with Jetson Nano, please share your experience.
It will help a lot.
Thank you.

[pderrenger]

@shanek16 it seems you're encountering a few issues related to package compatibility and architecture differences on the NVIDIA Jetson Nano. Here are some steps that might help you resolve these issues:

1. Pip Warning: The warning about pip can be resolved by using Python's -m module flag as suggested. Try installing the requirements using:

   python3 -m pip install -r requirements.txt

2. GitPython Error: The error message indicates that the version of GitPython required by the requirements.txt file is not available for your Python version. You might need to install a compatible version manually:

   python3 -m pip install GitPython==3.1.20

3. Python 3.8 Installation: Installing Python 3.8 on the Jetson Nano can lead to compatibility issues with the CUDA toolkit provided by NVIDIA's JetPack. It's generally recommended to use the Python version that comes with JetPack to avoid such issues.

4. Using a Modified Repository: The repository you mentioned (fei4xu/yolov5-python3.6.9-jetson) is a fork specifically modified for Python 3.6.9 on Jetson devices. If you're encountering an "Illegal instruction" error, it could be due to the compiled binaries not matching the Jetson Nano's architecture.

5. CUDA Compatibility: The libcublas.so.11 error suggests that the CUDA toolkit version required by the torchvision package is not present on your system. Ensure that you have the correct version of CUDA installed that matches the requirements of the torchvision package you're trying to install.

6. General Advice: When working with the Jetson Nano, it's crucial to use compatible versions of Python, CUDA, and other libraries. The JetPack SDK comes with tested and compatible versions of these tools, and deviating from them can lead to the issues you're experiencing.

For now, I would recommend sticking to the Python version that comes with JetPack 4.6 and using the official Ultralytics YOLOv8 documentation for guidance. If you're still having trouble, consider reaching out to the community forums or NVIDIA support for more tailored assistance with the Jetson Nano.

Remember, the YOLOv8 documentation and support can be found at our official docs: https://docs.ultralytics.com. Keep an eye on the documentation for any updates or additional guidance on deploying to platforms like the Jetson Nano. 📘

[Pugnacioius]

I got to the DeepStream configuration, cloned the repository but the gen_wts_yoloV5.py file was not found needed by the next step.

[flmello]

Not sure, but it seems gen_wts_yoloV5.py is in fact, export_yoloV5.py.

[flmello]

However, it gives you Invalid Weights File ....
ubuntu@yolov5:~/yolov5$ python3 export_yoloV5.py -w yolo5s.pt
Invalid weights file

[Pugnacioius]

Thanks and yes, this looks like a probable step but export_yoloV5.py requires onnx which is giving me version problems.

[flmello]

I checked the .md files all over /DeepStream-Yolo, it seems we need to add --dynamic.
However, now I get another error:
ubuntu@yolov5:/yolov5$ python3 export_yoloV5.py -w yolov5s.pt --dynamic

Starting: yolov5s.pt
Opening YOLOv5 model

YOLOv5 🚀 2022-6-11 Python-3.8.10 torch-1.10.1+cu102 CPU

Fusing layers...
YOLOv5s summary: 213 layers, 7225885 parameters, 0 gradients, 16.4 GFLOPs
Traceback (most recent call last):
File "export_yoloV5.py", line 120, in
sys.exit(main(args))
File "export_yoloV5.py", line 54, in main
if len(model.names.keys()) > 0:
AttributeError: 'list' object has no attribute 'keys'

[pderrenger]

It seems like there's a mismatch or an issue with the model file or its handling in the script. The error indicates that the script expects a dictionary with keys, but it's getting a list instead. This could be due to how the model's data is being loaded or interpreted.

Make sure that the model file yolov5s.pt is correct and fully compatible with the script you're using. It's also a good idea to check the version of the script and the model file to ensure they are compatible. If you have recently updated the repository or the model file, consider reverting to a previous version that was working or updating your script to align with the current model structure.

If the issue persists, you might want to look into the script export_yoloV5.py to debug how it processes the model file and ensure it correctly interprets the data as expected. Adjusting the script to accommodate the data structure of yolov5s.pt might be necessary.

[MagnusGabell]

Hi, good tutorial and it works, but how do I take that further? I want to use it in a python app not deepstream-app. The results from the inference needs to be picked up but how? This is good for benchmarking but I dont know which tutorial online I should trust and spend hours on trying to figure out how to get the output, box/class and confidence into python program. Any ideas or finger-pointers?

regards Magnus

[mominali12]

Hi can you share your environment details here?
It would be good to know the setup that got you to run the model on Jetson Nano

[pderrenger]

@mominali12 hello Magnus,

I'm glad to hear that the tutorial worked well for you! To integrate the model's inference results into a Python application, you can use the Python API provided by Ultralytics. After loading your trained model with TensorRT on the Jetson Nano, you can perform inference and obtain the detection results, which include bounding boxes, class labels, and confidence scores.

Here's a simplified example of how you might use the model in a Python app:

from ultralytics import YOLO

# Load your custom trained YOLOv8 model with TensorRT
model = YOLO('path/to/your/best-model.pt', device='cuda', format='engine')

# Perform inference on an image
results = model('path/to/your/image.jpg')

# Parse the inference results
for result in results:
    for box in result.boxes:
        xyxy = box.xyxy  # Bounding box coordinates
        conf = box.conf  # Confidence score
        cls = box.cls    # Class label
        # Now you can use these results as needed in your application

This code snippet assumes you have a trained YOLOv8 model exported as a TensorRT engine. The YOLO class handles loading the model and performing inference. The results object contains the detection outputs, which you can iterate over and integrate into your Python application.

For more detailed information on using the Python API, please refer to our documentation at https://docs.ultralytics.com. If you need further assistance, don't hesitate to reach out to the community on GitHub discussions for YOLOv8.

[Sanelembuli98]

Hii please help after cloning git clone https://github.com/marcoslucianops/DeepStream-Yolo the gen_wts_yoloV5.py doesnt exist any where on this repo it has export_yolov5.py which just exports an onnx model without creating the weights and cfg file was this intentional if so how do I create the cfr and weights file also the onnx that export_yolov5 creates is int64 instead of int32

[mominali12]

Having the same issue. This article is outdated now!

[mominali12]

----------------------- COMMAND --------------------------------
CUDA_VER=10.2 make -C nvdsinfer_custom_impl_Yolo

---------------------------------------- ERROR --------------------------------------------------------------
make: Entering directory '/home/nvidia/DeepStream-Yolo/nvdsinfer_custom_impl_Yolo'
g++ -c -o yolo.o -Wall -std=c++11 -shared -fPIC -Wno-error=deprecated-declarations -I/opt/nvidia/deepstream/deepstream/sources/includes -I/usr/local/cuda-10.2/include yolo.cpp
In file included from yolo.cpp:27:0:
yoloPlugins.h:73:13: error: ‘int32_t YoloLayer::enqueue(int, const void* const*, void* const*, void*, cudaStream_t)’ marked ‘override’, but does not override
int32_t enqueue(int batchSize, void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream)
^~~~~~~
yoloPlugins.h: In member function ‘virtual nvinfer1::IPluginV2* YoloLayerPluginCreator::deserializePlugin(const char*, const void*, size_t)’:
yoloPlugins.h:122:52: error: invalid new-expression of abstract class type ‘YoloLayer’
return new YoloLayer(serialData, serialLength);
^
yoloPlugins.h:46:7: note: because the following virtual functions are pure within ‘YoloLayer’:
class YoloLayer : public nvinfer1::IPluginV2 {
^~~~~~~~~
In file included from /usr/include/aarch64-linux-gnu/NvInferRuntime.h:59:0,
from /usr/include/aarch64-linux-gnu/NvInfer.h:53,
from /usr/include/aarch64-linux-gnu/NvInferPlugin.h:53,
from yolo.h:29,
from yolo.cpp:26:
/usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h:492:17: note: virtual int nvinfer1::IPluginV2::enqueue(int, const void* const*, void**, void*, cudaStream_t)
virtual int enqueue(int batchSize, const void* const* inputs, void** outputs, void* workspace, cudaStream_t stream) TRTNOEXCEPT = 0;
^~~~~~~
yolo.cpp: In member function ‘nvinfer1::ICudaEngine* Yolo::createEngine(nvinfer1::IBuilder*, nvinfer1::IBuilderConfig*)’:
yolo.cpp:57:12: error: ‘virtual nvinfer1::INetworkDefinition::~INetworkDefinition()’ is protected within this context
delete network;
^~~~~~~
In file included from /usr/include/aarch64-linux-gnu/NvInferPlugin.h:53:0,
from yolo.h:29,
from yolo.cpp:26:
/usr/include/aarch64-linux-gnu/NvInfer.h:5340:13: note: declared protected here
virtual ~INetworkDefinition() {}
^
yolo.cpp:108:10: error: ‘virtual nvinfer1::INetworkDefinition::~INetworkDefinition()’ is protected within this context
delete network;
^~~~~~~
In file included from /usr/include/aarch64-linux-gnu/NvInferPlugin.h:53:0,
from yolo.h:29,
from yolo.cpp:26:
/usr/include/aarch64-linux-gnu/NvInfer.h:5340:13: note: declared protected here
virtual ~INetworkDefinition() {}
^
yolo.cpp: In member function ‘NvDsInferStatus Yolo::buildYoloNetwork(std::vector&, nvinfer1::INetworkDefinition&)’:
yolo.cpp:453:36: error: invalid new-expression of abstract class type ‘YoloLayer’
modelType, m_ScoreThreshold);
^
Makefile:70: recipe for target 'yolo.o' failed
make: *** [yolo.o] Error 1
make: Leaving directory '/home/nvidia/DeepStream-Yolo/nvdsinfer_custom_impl_Yolo'

--------------------------------- SYSTEM DETAILS --------------------------------------------
UBUNTU 18.04
Model NVIDIA Jetson Nano Developer
Python 3.8 (using venv)
CUDA 10.2.89
cuDNN 8.0.0.180
TensorRT 7.1.3.0
OpenCV 4.1.1 with CUDA: NO
SoC tegra210
L4T 32.5.0
Jetpack 4.5

[mominali12]

Got the workaound @Sanelembuli98 issue by checkout at commit git checkout e7c77ee9fe78d8b82caa5b01574792af429fe923

But now stuck at CUDA_VER=10.2 make -C nvdsinfer_custom_impl_Yolo

[mominali12]

Steps for successful setup for custom model deployment on NVIDIA Jetson Nano:

1. Download SD card image for Jetson Nano Jetpack 4.6.4.
   Link: https://developer.nvidia.com/jetpack-sdk-464

2. Download and install NVIDIA Deepstream 6.0 SDK. (Download using .deb file)
   Link: https://docs.nvidia.com/metropolis/deepstream/6.0/dev-guide/text/DS_Quickstart.html

3. Clone YoloV5 - Ultralytics repo from GitHub:
   Bash Command: git clone https://github.com/ultralytics/yolov5

4. Go into yolov5 directory and then checkout to v7.0 branch for yolov5 - ultralytics repo
   Bash Command: cd yolov5
   Bash Command: git checkout v7.0

5. Update requirements.txt file in yolov5 folder:
   Bash Command: vi requirements.txt

Edit the following lines. Here you need to press i first to enter editing mode. Press ESC, then type :wq to save and quit

torch>=1.8.0

torchvision>=0.9.0

Note: torch and torchvision are excluded for now because they will be installed later.

6. Install the below dependency
   Bash Command: sudo apt install -y libfreetype6-dev

7. Install the necessary packages
   Bash Command: pip3 install -r requirements.txt

8. Install numpy 1.19.4 using following command:
   Bash Command: pip3 install --force-reinstall --no-cache-dir numpy=1.19.4

9. Install PyTorch
   First install PyTorch v1.10.0
   (Supported by JetPack 4.4 (L4T R32.4.3) / JetPack 4.4.1 (L4T R32.4.4) / JetPack 4.5 (L4T R32.5.0) / JetPack 4.5.1 (L4T R32.5.1) / JetPack 4.6 (L4T R32.6.1) with Python 3.6)
   Bash Command:
   cd ~
   sudo apt-get install -y libopenblas-base libopenmpi-dev
   wget https://nvidia.box.com/shared/static/fjtbno0vpo676a25cgvuqc1wty0fkkg6.whl -O torch-1.10.0-cp36-cp36m-linux_aarch64.whl
   pip3 install torch-1.10.0-cp36-cp36m-linux_aarch64.whl

10. Install TorchVision
    Bash Command:
    sudo apt install -y libjpeg-dev zlib1g-dev
    git clone --branch v0.11.1 https://github.com/pytorch/vision torchvision
    cd torchvision
    sudo python3 setup.py install

11. DeepStream Configuration for YOLOv5
    Bash Command:
    cd ~
    git clone https://github.com/marcoslucianops/DeepStream-Yolo
    git checkout e7c77ee9fe78d8b82caa5b01574792af429fe923
    cp DeepStream-Yolo/utils/gen_wts_yoloV5.py yolov5
    cd yolov5
    wget https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt
    python3 gen_wts_yoloV5.py -w yolov5s.pt

12. To change the inference size (default: 640) use flags as mentioned below
    -s SIZE
    --size SIZE
    -s HEIGHT WIDTH
    --size HEIGHT WIDTH
    Example for 1280:
    -s 1280
    or
    -s 1280 1280

13. Copy the generated cfg and wts files into the DeepStream-Yolo folder
    Bash Command:
    cp yolov5s.cfg ~/DeepStream-Yolo
    cp yolov5s.wts ~/DeepStream-Yolo

14. Open the DeepStream-Yolo folder and compile the library
    Bash Command:
    cd ~/DeepStream-Yolo
    CUDA_VER=11.4 make -C nvdsinfer_custom_impl_Yolo # for DeepStream 6.1
    CUDA_VER=10.2 make -C nvdsinfer_custom_impl_Yolo # for DeepStream 6.0.1 / 6.0 (Used this one in my case)

15. Edit the config_infer_primary_yoloV5.txt file according to your
    [property]
    ...
    custom-network-config=yolov5s.cfg
    model-file=yolov5s.wts
    ...

16. Edit the deepstream_app_config file

...
[primary-gie]
...
config-file=config_infer_primary_yoloV5.txt (CHECK THIS USUALLY THE DEFAULT VALUE IS DIFFERENT)

17. Change the video source in deepstream_app_config file. Here a default video file is loaded as you can see below

...
[source0]
...
uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4

18. Run the Inference
    deepstream-app -c deepstream_app_config.txt

[Sanelembuli98]

Thank you this seems to have solved my problem

[mominali12]

Can anyone guide on how to do int 8 calibration. The method mentioned in this article does not seems to be working for my Jetson Nano device.

Getting following error:

File does not exist: /home/nvidia/DeepStream-Yolo/calib.table
deepstream-app: yolo.cpp:76: nvinfer1::ICudaEngine* Yolo::createEngine(nvinfer1::IBuilder*, nvinfer1::IBuilderConfig*): Assertion `builder->platformHasFastInt8()' failed.

[mominali12]

Turns out Jetson Nano doesn't support int8 quantization. 😔

[pderrenger]

@mominali12 ah, I see you've discovered the limitation regarding INT8 quantization on the Jetson Nano. It's true that the Nano's hardware doesn't support INT8 operations, which is why you're encountering that error. For deployment on the Nano, you'll want to stick with FP16 or FP32 precision, which the Nano does support and can still offer a good balance between performance and accuracy.

If you're looking to optimize your model further, you might consider pruning or simplifying your network, using TensorRT for FP16 inference, and optimizing your input pipeline to ensure you're getting the best possible performance from the Nano. Keep in mind that while these methods won't give you the speedup of INT8, they can still make a significant difference in inference times. 🚀

If you have any more questions or need further assistance with your YOLOv8 model deployment on the Jetson Nano, feel free to ask!

[mominali12]

Hey, I am getting ~5 FPS on Jetson Nano with FP16 quantization. 🫤

I would be thankful if someone can share the benchmark results for the same with YoloV5 on Jetson Nano

[Sanelembuli98]

also having the same issue the general performance sucks ive back tracked and attempted to fix this by specifying additional flags during the training and generation of weight and cfg files but doesnt improve

[MagnusGabell]

I dont know much about Nano but running the stuff with Deepstream increase proformance quite alot.

[pderrenger]

@MagnusGabell it's great to hear that you're exploring YOLOv8 on the NVIDIA Jetson Nano platform. Achieving higher FPS can be challenging due to the hardware constraints of the Jetson Nano. However, there are a few strategies you could try to improve performance:

1. Model Optimization: Ensure that you're using a model optimized for edge devices, like YOLOv8n (nano), which is designed to be lightweight.

2. TensorRT Optimization: Converting your model to a TensorRT-optimized version can provide significant speedups. Make sure to follow the Export guide to properly convert your model.

3. DeepStream SDK: Integrating with NVIDIA's DeepStream SDK can indeed help to maximize throughput and efficiency on Jetson devices.

4. Inference Settings: Adjusting inference settings, such as reducing the input resolution or tweaking the NMS threshold, can also lead to performance gains.

5. Batch Inference: If your use case allows, running inference in batches rather than on individual images can improve FPS.

6. System Optimization: Close unnecessary applications and background processes to free up resources on the Jetson Nano.

Regarding benchmarks for YOLOv5 on Jetson Nano, we focus on YOLOv8 and its ecosystem. For YOLOv5 benchmarks, you might want to check the YOLOv5 repository or community discussions for shared experiences and results.

If you're experiencing suboptimal general performance, it's worth revisiting the training process to ensure that the model is well-optimized. Consider using the Benchmark mode to profile different aspects of your model and identify potential bottlenecks.

Lastly, if you find that DeepStream is giving you better performance, it's a good indication that leveraging NVIDIA's optimized pipelines is the right approach. Keep experimenting with DeepStream configurations to fine-tune the performance further. 🚀

If you have specific questions or need further assistance with YOLOv8, feel free to reach out!

[flmello]

Thanks to these posts, this was the first time I managed to run Yolo on Deepstream. However, it works fine with yolov5s.pt, but I see a weird behavior with custom models based on a transfer learning from yolov5s. When using a custom model, it detects the objects from yolov5.pt, not the ones the model was trained for. This is a description of my experiment:

Take a look at this extract from label files of yolov5s.pt and my custom model epi.pt

yolov5s.pt labels          epi.pt labels
person                    person
bicycle                   helmet
car                       suit
motorcycle                boot
airplane                  gloves
bus                       mask
train                     glasses
truck                     earplug
boat
.....

And this is what I obtain when exporting epi.pt to epi.onnx. The DeepStream-Yolo folder now contains just the epi.onnx and label.txt. This is what I get:

Clearly, it is using the original weights, the ones from yolov5s. For instance, it detects an object whose class are bicycles ("1") and by label file says it is helmet, or an object belonging to trucks ("7") and my label file says it is an earplug. It look like it is ignoring my training and using just the original weights from yolov5s.

When I trained my custom model (epi.pt) I made a transfer learning from yolov5s. This is what I did for training my custom model (epi.pt): python3 train.py --img 640 --batch 16 --epochs 1000 --data epi.yaml --weights yolov5s.pt --cache --save-period 10

And this is what my custom model epi.pt returns when I run: python3 ~/yolov5/detect.py --weights epi.pt --source tegra.mp4.

What's going wrong? Is the export script creating an incomplete onnx file?

[flmello]

Yes, I did. Deleted all *.pt, labels.txt and *.onnx. Just to be sure I was working with a brand new engine.

[MagnusGabell]

But it looks like its doing its job? How many images did you train on?
The bottom images clearly pics the boots or?

[Sanelembuli98]

Was the model trained from scratch?

[flmello]

But it looks like its doing its job? How many images did you train on?
The bottom images clearly pics the boots or?
The second pair of images use ultralytics script, with no Nvidia Deepstreem, and it works perfect.
My problem is with the usage of Deepstream, the first pair of images. It was expected a behavior like the 2nd pair of images, but it doesn't provide the correct detections.

And I used 6,700 images for training, although I believe this is no an issue here.

Was the model trained from scratch?
No, it wasn't .
When I do python3 train.py --img 640 --batch 16 --epochs 1000 --data epi.yaml --weights yolov5s.pt --cache --save-period 10, I am taking the yolov5s weight and using them no initialize my network.

[pderrenger]

@flmello it seems like there might be an issue with the way the custom model epi.pt is being converted to ONNX and subsequently used with DeepStream. The fact that the model works correctly with the Ultralytics script but not with DeepStream suggests that the problem could be with the export process or the DeepStream configuration.

Here are a few steps you can take to troubleshoot the issue:

1. Re-export the ONNX Model: Ensure that you're exporting the epi.pt model correctly to ONNX format. Use the export.py script provided by Ultralytics YOLOv8 to export the model and verify that the process completes without errors.

2. Check the DeepStream Configuration: Review the DeepStream configuration files, especially the config_infer_primary.txt or similar, to ensure that the correct ONNX model and labels file are being referenced.

3. Verify the Labels: Double-check the labels.txt file used by DeepStream to ensure that it matches the classes of your custom model. The order of classes in this file should align with the class indices that your model predicts.

4. Inspect the ONNX Model: Use tools like Netron to visualize the ONNX model and confirm that the output layer corresponds to the expected number of classes.

5. Test the ONNX Model: Before deploying to DeepStream, test the ONNX model using a standalone ONNX runtime to ensure that it predicts the correct classes.

6. DeepStream Model Cache: After deleting the old engine files, DeepStream might still use a cached version. Clear the TensorRT engine cache located at /var/nvidia/tensorrt/cache on the Jetson Nano and try running the DeepStream pipeline again.

7. DeepStream Version Compatibility: Ensure that the version of DeepStream SDK you're using is compatible with the exported ONNX model. Sometimes, newer or older versions of DeepStream might handle ONNX models differently.

8. Training Data Check: Although you've mentioned that the number of images is likely not the issue, it's still worth ensuring that the training data was correctly labeled and that the model had a chance to learn the custom classes effectively.

If you've gone through these steps and the issue persists, it might be helpful to reach out to the community or the maintainers of the Ultralytics YOLOv8 repository for more specific guidance, as they might have encountered similar issues before.

[Edisharing]

The tensor acceleration and the speed of recognizing images are quite amazing, I would like to use the tensor to accelerate the process of detecting vehicles in real time (real-time video) in Jetson Nano, but now I can't find any information about it, how should I start to carry out my work?

[pderrenger]

@Edisharing it's great to hear about your interest in using YOLOv8 for real-time vehicle detection on the Jetson Nano! To get started, you'll want to follow these general steps:

1. Model Preparation: Ensure you have a trained YOLOv8 model ready for deployment. If you haven't trained your model yet, you can do so following the training instructions in our documentation.

2. Export to ONNX: Convert your trained YOLOv8 model to ONNX format using the export mode. This is necessary for compatibility with TensorRT.

   from ultralytics import YOLO
   
   # Load your trained model
   model = YOLO('path/to/your/trained/model.pt')
   
   # Export the model to ONNX format
   model.export(format='onnx')

3. TensorRT Optimization: Once you have the ONNX model, you can optimize it with TensorRT on your Jetson Nano. This step involves creating a TensorRT engine that is specifically optimized for the Nano's GPU architecture.

4. DeepStream Integration: With the TensorRT engine ready, you can integrate it with NVIDIA's DeepStream SDK, which is designed for building efficient video analytics applications. DeepStream provides a pipeline for ingesting video streams, running inference, and processing the results.

5. Real-Time Inference: Use the DeepStream pipeline to process your video stream in real-time. You'll need to set up the pipeline to use your TensorRT engine and define how the detections should be handled and displayed.

6. Performance Tuning: Once everything is set up, you may need to fine-tune your setup to achieve the desired performance. This could involve adjusting the model's input size, the batch size, or other inference settings.

For detailed instructions and code examples, you can refer to our documentation at https://docs.ultralytics.com. While the documentation currently focuses on YOLOv5, the process for YOLOv8 will be similar, especially regarding model export and TensorRT optimization. Keep an eye on the documentation for any updates specific to YOLOv8.

If you encounter any issues or have specific questions during the process, feel free to reach out to the community or check the Ultralytics Docs for guidance. Good luck with your vehicle detection project! 🚗💨

[zohaibjan]

Is there a config_infer_secondary_yolov8 for the yolov8_cls deep stream integration ?

[pderrenger]

@zohaibjan currently, there isn't a specific config_infer_secondary_yolov8 file provided for YOLOv8 classification (yolov8_cls) integration with DeepStream. The configuration files and secondary inference engines for DeepStream are typically tailored for detection tasks.

However, if you're looking to integrate YOLOv8 classification with DeepStream, you would need to create a custom configuration file that specifies the network architecture, input dimensions, and other relevant parameters for the classification model. This process would involve adapting the existing YOLO configuration files used for detection to work with the classification model.

For guidance on creating and modifying DeepStream configuration files, you can refer to the DeepStream documentation and the sample config files provided with the SDK. Additionally, you might find it helpful to look at the existing YOLO configuration files for detection as a starting point.

If you have further questions or need assistance with YOLOv8 and DeepStream integration, feel free to reach out to the community or check the Ultralytics documentation for updates and more detailed instructions.

[mgabell]

would this help?
https://github.com/marcoslucianops/DeepStream-Yolo

[glenn-jocher]

@mgabell the repository you mentioned, marcoslucianops/DeepStream-Yolo, is indeed a helpful resource for integrating YOLO models with NVIDIA DeepStream for object detection tasks. It provides a custom parser for YOLO layers and allows you to convert YOLO models to a format compatible with DeepStream.

However, this repository is primarily focused on object detection and may not directly support classification models like yolov8_cls out of the box. DeepStream's primary use case revolves around detection, tracking, and other tasks that involve spatial information, which is different from the classification task that typically outputs only class labels without spatial bounding boxes.

To integrate a YOLOv8 classification model with DeepStream, you would need to adapt the existing code and configuration files to handle the classification output. This might involve:

1. Modifying the parser to handle the output format of the classification model.
2. Adjusting the configuration files to reflect the input size and output format expected by the classification model.
3. Implementing a custom post-processing step to interpret the classification results within the DeepStream pipeline.

If you're looking to perform secondary classification on detected objects within DeepStream, you might still be able to use parts of the repository as a starting point, but you would need to implement the classification-specific logic yourself.

For a classification task, you might not find a ready-made solution within the DeepStream-Yolo repository, and it could require custom development work. If you're not sure how to proceed, consider reaching out to the repository maintainers or the NVIDIA DeepStream forums for guidance on how to adapt the code for classification tasks.

[MadhavKrishna1]

can you explain a bit more.Actually i have build engine(yolov8.eninge fike)(and used tensorrt into it) now i want to do inference using this for yolov8 for 3 different ways (i)a single picture(ii)a batch of pictures (iii) using camera.Can you please help me with this?

…

On Mon, 8 Jul 2024 at 17:32, Paula Derrenger ***@***.***> wrote: Hi @MadhavKrishna1 <https://github.com/MadhavKrishna1>, Certainly! Below is an example of how you can integrate the TensorRT engine with a DeepStream Python app. This example assumes you have already created the TensorRT engine as described in the previous steps. 1. *Install DeepStream Python Bindings*: Ensure you have the DeepStream Python bindings installed. You can follow the installation instructions from the DeepStream Python Apps <https://github.com/NVIDIA-AI-IOT/deepstream_python_apps> repository. 2. *DeepStream Python App Integration*: Here is a basic example of how to load the TensorRT engine and use it for inference within a DeepStream pipeline. import pydsimport gigi.require_version('Gst', '1.0')from gi.repository import Gst, GObject # Initialize GStreamerGst.init(None) # Create the pipelinepipeline = Gst.Pipeline() # Create elementssource = Gst.ElementFactory.make('filesrc', 'file-source')h264parser = Gst.ElementFactory.make('h264parse', 'h264-parser')decoder = Gst.ElementFactory.make('nvv4l2decoder', 'nvv4l2-decoder')streammux = Gst.ElementFactory.make('nvstreammux', 'Stream-muxer')pgie = Gst.ElementFactory.make('nvinfer', 'primary-inference')nvvidconv = Gst.ElementFactory.make('nvvideoconvert', 'nvvideo-converter')nvosd = Gst.ElementFactory.make('nvdsosd', 'nv-onscreendisplay')sink = Gst.ElementFactory.make('nveglglessink', 'nvvideo-renderer') # Set propertiessource.set_property('location', 'path/to/your/video.mp4')streammux.set_property('width', 3040)streammux.set_property('height', 3040)streammux.set_property('batch-size', 1)streammux.set_property('batched-push-timeout', 4000000)pgie.set_property('config-file-path', 'path/to/config_infer_primary_yoloV8.txt') # Add elements to the pipelinepipeline.add(source)pipeline.add(h264parser)pipeline.add(decoder)pipeline.add(streammux)pipeline.add(pgie)pipeline.add(nvvidconv)pipeline.add(nvosd)pipeline.add(sink) # Link the elementssource.link(h264parser)h264parser.link(decoder) # Link the decoder to the streammuxsinkpad = streammux.get_request_pad('sink_0')srcpad = decoder.get_static_pad('src')srcpad.link(sinkpad) # Link the remaining elementsstreammux.link(pgie)pgie.link(nvvidconv)nvvidconv.link(nvosd)nvosd.link(sink) # Create an event loop and feed GStreamer bus messages to itloop = GObject.MainLoop()bus = pipeline.get_bus()bus.add_signal_watch()bus.connect('message', bus_call, loop) # Start the pipelinepipeline.set_state(Gst.State.PLAYING) try: loop.run()except: pass # Cleanuppipeline.set_state(Gst.State.NULL) This script sets up a basic DeepStream pipeline that reads a video file, decodes it, performs inference using the TensorRT engine, and displays the results. Make sure to adjust the paths and properties according to your setup. For more detailed information, you can refer to the DeepStream SDK documentation <https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html>. If you encounter any issues, please ensure you are using the latest versions of the packages and provide a reproducible example if the problem persists. You can find more information on creating a minimum reproducible example here <https://docs.ultralytics.com/help/minimum_reproducible_example>. Happy coding! 😊 — Reply to this email directly, view it on GitHub <#2475 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCVBHHXBIKA6RYBRSAV436TZLJ5VBAVCNFSM6AAAAAAX2BWOU6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TSOBWGY4TG> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[pderrenger]

Hi @MadhavKrishna1,

Certainly! Let's break down the process of performing inference using your TensorRT engine (yolov8.engine) for three different scenarios: a single image, a batch of images, and a live camera feed.

1. Single Image Inference

To perform inference on a single image, you can use the following Python script. This script loads the TensorRT engine and runs inference on a specified image.

import cv2
import numpy as np
import tensorrt as trt

# Load the TensorRT engine
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
with open("yolov8.engine", "rb") as f, trt.Runtime(TRT_LOGGER) as runtime:
    engine = runtime.deserialize_cuda_engine(f.read())

# Allocate buffers
context = engine.create_execution_context()
input_shape = engine.get_binding_shape(0)
output_shape = engine.get_binding_shape(1)
input_size = trt.volume(input_shape) * engine.max_batch_size
output_size = trt.volume(output_shape) * engine.max_batch_size
d_input = cuda.mem_alloc(input_size * np.dtype(np.float32).itemsize)
d_output = cuda.mem_alloc(output_size * np.dtype(np.float32).itemsize)
bindings = [int(d_input), int(d_output)]

# Load and preprocess the image
image = cv2.imread("path/to/your/image.jpg")
image_resized = cv2.resize(image, (input_shape[2], input_shape[1]))
image_resized = image_resized.astype(np.float32)
image_resized = np.transpose(image_resized, (2, 0, 1))
image_resized = np.expand_dims(image_resized, axis=0)

# Run inference
cuda.memcpy_htod(d_input, image_resized)
context.execute_v2(bindings)
output = np.empty(output_shape, dtype=np.float32)
cuda.memcpy_dtoh(output, d_output)

# Process the output
# (Add your post-processing code here)

print("Inference completed on single image.")

2. Batch of Images Inference

For batch inference, you can modify the above script to handle multiple images at once. Ensure that your TensorRT engine supports batch processing.

# Load and preprocess the batch of images
images = ["path/to/image1.jpg", "path/to/image2.jpg", ...]
batch_images = []
for img_path in images:
    image = cv2.imread(img_path)
    image_resized = cv2.resize(image, (input_shape[2], input_shape[1]))
    image_resized = image_resized.astype(np.float32)
    image_resized = np.transpose(image_resized, (2, 0, 1))
    batch_images.append(image_resized)

batch_images = np.stack(batch_images)
batch_images = np.expand_dims(batch_images, axis=0)

# Run inference on the batch
cuda.memcpy_htod(d_input, batch_images)
context.execute_v2(bindings)
output = np.empty(output_shape, dtype=np.float32)
cuda.memcpy_dtoh(output, d_output)

# Process the output
# (Add your post-processing code here)

print("Inference completed on batch of images.")

3. Camera Feed Inference

To perform inference on a live camera feed, you can use OpenCV to capture frames from the camera and run inference on each frame.

import cv2
import numpy as np
import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit

# Load the TensorRT engine
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
with open("yolov8.engine", "rb") as f, trt.Runtime(TRT_LOGGER) as runtime:
    engine = runtime.deserialize_cuda_engine(f.read())

# Allocate buffers
context = engine.create_execution_context()
input_shape = engine.get_binding_shape(0)
output_shape = engine.get_binding_shape(1)
input_size = trt.volume(input_shape) * engine.max_batch_size
output_size = trt.volume(output_shape) * engine.max_batch_size
d_input = cuda.mem_alloc(input_size * np.dtype(np.float32).itemsize)
d_output = cuda.mem_alloc(output_size * np.dtype(np.float32).itemsize)
bindings = [int(d_input), int(d_output)]

# Open the camera
cap = cv2.VideoCapture(0)

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    # Preprocess the frame
    frame_resized = cv2.resize(frame, (input_shape[2], input_shape[1]))
    frame_resized = frame_resized.astype(np.float32)
    frame_resized = np.transpose(frame_resized, (2, 0, 1))
    frame_resized = np.expand_dims(frame_resized, axis=0)

    # Run inference
    cuda.memcpy_htod(d_input, frame_resized)
    context.execute_v2(bindings)
    output = np.empty(output_shape, dtype=np.float32)
    cuda.memcpy_dtoh(output, d_output)

    # Process the output
    # (Add your post-processing code here)

    # Display the frame
    cv2.imshow("Camera Feed", frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

print("Inference completed on camera feed.")

Additional Resources

For more detailed information on deploying YOLOv8 with DeepStream and TensorRT, please refer to our DeepStream and NVIDIA Jetson Guide.

If you encounter any issues, please ensure you are using the latest versions of the packages and provide a reproducible example if the problem persists. You can find more information on creating a minimum reproducible example here.

Happy coding! 😊