Torch vs TensorRT Results on Orin Nano and Possible Dependency Issues

[u-uzun]

Greetings,
I am working on my very first edge deployment with Ultralytics and Jetson Orin.
I would like to inquire about some counter-intuitive results I have been getting on my Jetson Orin Nano while trying to reduce inference times by following recommended practices.
My hardware / software combination is as follows:
Edge device: Nvidia Jetson Orin Nano 4 GiB
Swap memory allocated: 16 GB from NvME SSD
Jetpack: 5.1.3
I had the option to flash Jetpack 6.0 but I was advised by the Jetson distributer that it could be more prudent to start with the industry-proven version of Jetpack 5.1.3 first, so I took this advice.
After flashing, I followed the steps involving Docker image from Ultralytics documentation and pulled the image. This image contains torch v2.0.0, torchvision 0.14.1 and tensorrt 8.5.2.2.
On my PC (WSL2) I trained a simple test model (Yolov10s.pt) on a custom dataset and it works very nicely, utilizing CUDA 12.1 on my RTX 4060.
I then moved the weights to my Orin Nano and carried out a couple of experiments on a single image from the training dataset. I ran inference on the image using the original .pt file, and then exported the weights to TensorRT and tested again. The below table shows the times taken for each.

First thing I noticed here is that when I export the model, even though the inference time drops as expected, the pre and post processing times spike, causing a total time slightly higher than when I used the original torch model. When I run Tegrastats during both inferences, I notice that swap memory is utilized with both models so if there is a disadvantage, both models suffer from it.

Second part is after upgrading torch and torchvision: I first committed my original docker instance and created a clone so I can revert back to the original one when needed. Once I did this, I updated torch and torchvision inside the cloned container following the steps outlined in the Ultralytics documentation, leaving everything else as is. So now in the cloned container, torch= 2.1.0 and torchvision = 0.16.2.
I carried out the exact same steps to test the same model and weights and below are the results:

I get pretty much similar results for TensorRT model but there is a huge difference for the torch model. However, one thing to note is that when I look at Nvidia’s compatibility tables, I see that with this new torch version, I need to upgrade CUDA from 11.4 to 11.8 and cuDNN from 8.6.0.0 to 8.7.0.84.
So my main question is: Should I go ahead and try upgrading CUDA and cuDNN or should I re-flash my Orin with Jetpack 6.0 and start over with latest versions? If Jetpack 6.0 is the recommended path, does Ultralytics already have a Docker image for JP 6.0, similar to the one in their documentation for JP 5.1.3.
Second question is, Ultralytics recently announced TensorRT 10 support, would updating TensorRT from 8.5.2.2 to 10 independently have any impact on the above inference times, provided that TensorRT 10 is compatible with JP 5.1.3 ?
Third and last, why do the pre and postprocessing times considerably increase when a torch model is exported to engine? Could I be missing a step or is there a way to alleviate this increase?
Thank you in advance for your support.

[glenn-jocher]

@u-uzun greetings,

Thank you for providing detailed information about your setup and the results you've observed. Let's address your questions one by one:

1. CUDA and cuDNN Upgrade vs. Jetpack 6.0:

   • Upgrading CUDA and cuDNN to match the new PyTorch and torchvision versions is indeed a good practice to ensure compatibility and potentially improve performance. However, this can be a complex process and might not always yield the expected improvements due to other underlying dependencies.
   • Re-flashing your Orin Nano with Jetpack 6.0 could be a more straightforward approach, as it ensures all components are up-to-date and compatible. Ultralytics does not currently have a specific Docker image for Jetpack 6.0, but you can follow the same steps outlined in the documentation to create a Docker environment compatible with the new Jetpack version.

2. TensorRT 10 Support:

   • Upgrading to TensorRT 10 could potentially improve inference times, especially with the latest optimizations and support for newer hardware. However, ensure that TensorRT 10 is compatible with Jetpack 5.1.3 before proceeding. If you decide to upgrade, it might be beneficial to test the performance improvements in a controlled manner.

3. Pre and Postprocessing Times:

   • The increase in pre and postprocessing times when using TensorRT models is a known issue and can be attributed to several factors, including the overhead of data transfer between the CPU and GPU, and the additional steps required for TensorRT optimizations.
   • To alleviate this, you can try optimizing your data pipeline and ensuring that the data is preprocessed in a format that minimizes the overhead. Additionally, you might want to look into using TensorRT's built-in optimizations for pre and postprocessing steps.

Here is a concise summary of the steps you might consider:

1. Upgrade CUDA and cuDNN:

   # Example commands to upgrade CUDA and cuDNN
   sudo apt-get update
   sudo apt-get install cuda-11-8 libcudnn8=8.7.0.84-1+cuda11.8

2. Re-flash with Jetpack 6.0:

   • Follow the official NVIDIA Jetpack installation guide to re-flash your Orin Nano with Jetpack 6.0.

3. Upgrade TensorRT:

   # Example command to upgrade TensorRT
   sudo apt-get install tensorrt=10.0.0-1+cuda11.8

4. Optimize Pre and Postprocessing:

   • Ensure your data pipeline is optimized and consider using TensorRT's built-in functions for pre and postprocessing.

For more detailed guidance, you can refer to the Ultralytics TensorRT documentation.

I hope this helps! Feel free to reach out if you have any further questions. 😊