Inference in TensorRT fails

[lodm94]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Validation, Detection, Export

Bug

Traceback (most recent call last):
  File "detect.py", line 261, in <module>
    main(opt)
  File "detect.py", line 256, in main
    run(**vars(opt))
  File "/home/leonardo/miniconda3/envs/yolo_env/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "detect.py", line 92, in run
    model = DetectMultiBackend(weights, device=device, dnn=dnn, data=data)
  File "/home/leonardo/Scrivania/yolov5/models/common.py", line 351, in __init__
    context = model.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'

Environment

I am working in a Conda env on Ubuntu 20.04 host

Minimal Reproducible Example

I exported onnx and engine model with export.py using the following command:
python export.py --weights path-to-pt-weights --imgsz 540 960 --batch-size 1 --device 0 --include engine onnx
Then i tried to do inference with engine file using:
python detect.py --weights path-to-engine-file --source val/images/ --device 0 --imgsz 960
Or i tried to run validation with:
python val.py --weights path-to-engine-file --data path-to-yaml-file --device 0 --imgsz 960 --task val
But in both case it returns the error i mentioned above.

Additional

I have some doubt about image-size declaration while exporting the model.
The model should infer on image with fixed size [540,960] and that is the size i used in export.py.
But during the inference with detect.py or during validation with val.py i can only declare one size, so i use the biggest, 960.
If i try to do inference with onnx model with:
python detect.py --weights pesi_sar/best_model_1502.onnx --source val_new/images/ --device 0 --imgsz 960
It obviously returns:

Traceback (most recent call last):
  File "detect.py", line 261, in <module>
    main(opt)
  File "detect.py", line 256, in main
    run(**vars(opt))
  File "/home/leonardo/miniconda3/envs/yolo_env/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "detect.py", line 117, in run
    model.warmup(imgsz=(1 if pt else bs, 3, *imgsz), half=half)  # warmup
  File "/home/leonardo/Scrivania/yolov5/models/common.py", line 474, in warmup
    self.forward(im)  # warmup
  File "/home/leonardo/Scrivania/yolov5/models/common.py", line 421, in forward
    y = self.session.run([self.session.get_outputs()[0].name], {self.session.get_inputs()[0].name: im})[0]
  File "/home/leonardo/miniconda3/envs/yolo_env/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: images for the following indices
 index: 2 Got: 960 Expected: 544
 Please fix either the inputs or the model.

So i tried to export the model declaring only the biggest size with the following command:
python export.py --weights path-to-pt-weights --imgsz 960 --batch-size 1 --device 0 --include engine onnx
The engine model still doesn't work and throw the error mentioned on top.
The onnx model works but it shows different perfomances respect to pytorch model.
That's probably because with onnx weigths the model infers at 960x960, while with pytorch weigths it infers at 544x960 (and thats what i desire!!).

Can you clarify the best practice to use for exporting the model to onnx or engine?
If it is still not clear, i trained my model at 960 image size, but the images i need to process are 540,960 (a quarter of the FHD format btw). I would like to export the model in TRT and keep the same performances as in PyTorch while of course decreasing the inference time.

Cheers

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[github-actions]

👋 Hello @lodm94, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@lodm94 TensorRT export and inference examples are shown in Colab notebook Appendix section:
https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb

Examples work correctly in a recent test:

[lodm94]

Hi @glenn-jocher
In my case it does not work:

(yolo_env) leonardo@leonardo-Precision-7750:~/Scrivania/yolov5$ python export.py --weights pesi_sar/best_model_1502.pt --include engine --imgsz 540 960 --device 0
export: data=data/coco128.yaml, weights=['pesi_sar/best_model_1502.pt'], imgsz=[540, 960], batch_size=1, device=0, half=False, inplace=False, train=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=12, verbose=False, workspace=4, nms=False, agnostic_nms=False, topk_per_class=100, topk_all=100, iou_thres=0.45, conf_thres=0.25, include=['engine']
YOLOv5 🚀 v6.1-17-g596de6d torch 1.10.2 CUDA:0 (Quadro RTX 5000, 16125MiB)

Fusing layers... 
Model Summary: 290 layers, 20852934 parameters, 0 gradients, 47.9 GFLOPs
WARNING: --img-size 540 must be multiple of max stride 32, updating to 544

PyTorch: starting from pesi_sar/best_model_1502.pt with output shape (1, 32130, 6) (42.3 MB)

ONNX: starting export with onnx 1.11.0...
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
ONNX: export success, saved as pesi_sar/best_model_1502.onnx (84.0 MB)

TensorRT: starting export with TensorRT 8.4.0.6...
[03/10/2022-16:20:22] [TRT] [I] [MemUsageChange] Init CUDA: CPU +306, GPU +0, now: CPU 3049, GPU 2967 (MiB)
[03/10/2022-16:20:23] [TRT] [I] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 3068 MiB, GPU 2968 MiB
[03/10/2022-16:20:23] [TRT] [I] [MemUsageSnapshot] End constructing builder kernel library: CPU 3296 MiB, GPU 3036 MiB
export.py:220: DeprecationWarning: Use set_memory_pool_limit instead.
  config.max_workspace_size = workspace * 1 << 30
[03/10/2022-16:20:23] [TRT] [I] ----------------------------------------------------------------
[03/10/2022-16:20:23] [TRT] [I] Input filename:   pesi_sar/best_model_1502.onnx
[03/10/2022-16:20:23] [TRT] [I] ONNX IR version:  0.0.7
[03/10/2022-16:20:23] [TRT] [I] Opset version:    13
[03/10/2022-16:20:23] [TRT] [I] Producer name:    pytorch
[03/10/2022-16:20:23] [TRT] [I] Producer version: 1.10
[03/10/2022-16:20:23] [TRT] [I] Domain:           
[03/10/2022-16:20:23] [TRT] [I] Model version:    0
[03/10/2022-16:20:23] [TRT] [I] Doc string:       
[03/10/2022-16:20:23] [TRT] [I] ----------------------------------------------------------------
[03/10/2022-16:20:23] [TRT] [W] onnx2trt_utils.cpp:365: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/10/2022-16:20:23] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
[03/10/2022-16:20:23] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
[03/10/2022-16:20:23] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
TensorRT: Network Description:
TensorRT:       input "images" with shape (1, 3, 544, 960) and dtype DataType.FLOAT
TensorRT:       output "output" with shape (1, 32130, 6) and dtype DataType.FLOAT
TensorRT:       output "467" with shape (1, 3, 68, 120, 6) and dtype DataType.FLOAT
TensorRT:       output "533" with shape (1, 3, 34, 60, 6) and dtype DataType.FLOAT
TensorRT:       output "599" with shape (1, 3, 17, 30, 6) and dtype DataType.FLOAT
TensorRT: building FP16 engine in pesi_sar/best_model_1502.engine
export.py:239: DeprecationWarning: Use build_serialized_network instead.
  with builder.build_engine(network, config) as engine, open(f, 'wb') as t:
[03/10/2022-16:20:23] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.8.0 but loaded cuBLAS/cuBLAS LT 11.5.1
[03/10/2022-16:20:23] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1, GPU +8, now: CPU 3386, GPU 3044 (MiB)
[03/10/2022-16:20:24] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +114, GPU +50, now: CPU 3500, GPU 3094 (MiB)
[03/10/2022-16:20:24] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[03/10/2022-16:21:04] [TRT] [I] Some tactics do not have sufficient workspace memory to run. Increasing workspace size will enable more tactics, please check verbose output for requested sizes.
[03/10/2022-16:25:16] [TRT] [I] Detected 1 inputs and 7 output network tensors.
[03/10/2022-16:25:17] [TRT] [I] Total Host Persistent Memory: 209424
[03/10/2022-16:25:17] [TRT] [I] Total Device Persistent Memory: 2765312
[03/10/2022-16:25:17] [TRT] [I] Total Scratch Memory: 0
[03/10/2022-16:25:17] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 48 MiB, GPU 2471 MiB
[03/10/2022-16:25:17] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 15.9647ms to assign 9 blocks to 132 nodes requiring 32901636 bytes.
[03/10/2022-16:25:17] [TRT] [I] Total Activation Memory: 32901636
[03/10/2022-16:25:17] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +40, GPU +43, now: CPU 40, GPU 43 (MiB)
TensorRT: export success, saved as pesi_sar/best_model_1502.engine (44.8 MB)

Export complete (301.55s)
Results saved to /home/leonardo/Scrivania/yolov5/pesi_sar
Detect:          python detect.py --weights pesi_sar/best_model_1502.engine
PyTorch Hub:     model = torch.hub.load('ultralytics/yolov5', 'custom', 'pesi_sar/best_model_1502.engine')
Validate:        python val.py --weights pesi_sar/best_model_1502.engine
Visualize:       https://netron.app
(yolo_env) leonardo@leonardo-Precision-7750:~/Scrivania/yolov5$ python detect.py --weights pesi_sar/best_model_1502.engine --imgsz 540 960 --source val_new/images/ --device 0
detect: weights=['pesi_sar/best_model_1502.engine'], source=val_new/images/, data=data/coco128.yaml, imgsz=[540, 960], conf_thres=0.25, iou_thres=0.45, max_det=1000, device=0, view_img=False, save_txt=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=runs/detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False
YOLOv5 🚀 v6.1-17-g596de6d torch 1.10.2 CUDA:0 (Quadro RTX 5000, 16125MiB)

Loading pesi_sar/best_model_1502.engine for TensorRT inference...
[03/10/2022-16:27:32] [TRT] [I] [MemUsageChange] Init CUDA: CPU +306, GPU +0, now: CPU 421, GPU 1594 (MiB)
[03/10/2022-16:27:32] [TRT] [I] Loaded engine size: 41 MiB
[03/10/2022-16:27:32] [TRT] [E] 1: [pluginV2Runner.cpp::load::290] Error Code 1: Serialization (Serialization assertion creator failed.Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
[03/10/2022-16:27:32] [TRT] [E] 4: [runtime.cpp::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.)
Traceback (most recent call last):
  File "detect.py", line 261, in <module>
    main(opt)
  File "detect.py", line 256, in main
    run(**vars(opt))
  File "/home/leonardo/miniconda3/envs/yolo_env/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "detect.py", line 92, in run
    model = DetectMultiBackend(weights, device=device, dnn=dnn, data=data)
  File "/home/leonardo/Scrivania/yolov5/models/common.py", line 351, in __init__
    context = model.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'
-------------------------------------------------------------------
PyCUDA ERROR: The context stack was not empty upon module cleanup.
-------------------------------------------------------------------
A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.
-------------------------------------------------------------------
Annullato (core dump creato)

What about this?

[lodm94]

Even your example does not work:

(yolo_env) leonardo@leonardo-Precision-7750:~/Scrivania/yolov5$ python export.py --weights weights/yolov5s.pt --include engine --imgsz 640 640 --device 0
export: data=data/coco128.yaml, weights=['weights/yolov5s.pt'], imgsz=[640, 640], batch_size=1, device=0, half=False, inplace=False, train=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=12, verbose=False, workspace=4, nms=False, agnostic_nms=False, topk_per_class=100, topk_all=100, iou_thres=0.45, conf_thres=0.25, include=['engine']
YOLOv5 🚀 v6.1-17-g596de6d torch 1.10.2 CUDA:0 (Quadro RTX 5000, 16125MiB)

Fusing layers... 
Model Summary: 213 layers, 7225885 parameters, 0 gradients, 16.5 GFLOPs

PyTorch: starting from weights/yolov5s.pt with output shape (1, 25200, 85) (14.8 MB)

ONNX: starting export with onnx 1.11.0...
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
ONNX: export success, saved as weights/yolov5s.onnx (29.3 MB)

TensorRT: starting export with TensorRT 8.4.0.6...
[03/10/2022-16:31:38] [TRT] [I] [MemUsageChange] Init CUDA: CPU +306, GPU +0, now: CPU 3077, GPU 2836 (MiB)
[03/10/2022-16:31:39] [TRT] [I] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 3097 MiB, GPU 2856 MiB
[03/10/2022-16:31:39] [TRT] [I] [MemUsageSnapshot] End constructing builder kernel library: CPU 3324 MiB, GPU 2920 MiB
export.py:220: DeprecationWarning: Use set_memory_pool_limit instead.
  config.max_workspace_size = workspace * 1 << 30
[03/10/2022-16:31:39] [TRT] [I] ----------------------------------------------------------------
[03/10/2022-16:31:39] [TRT] [I] Input filename:   weights/yolov5s.onnx
[03/10/2022-16:31:39] [TRT] [I] ONNX IR version:  0.0.7
[03/10/2022-16:31:39] [TRT] [I] Opset version:    13
[03/10/2022-16:31:39] [TRT] [I] Producer name:    pytorch
[03/10/2022-16:31:39] [TRT] [I] Producer version: 1.10
[03/10/2022-16:31:39] [TRT] [I] Domain:           
[03/10/2022-16:31:39] [TRT] [I] Model version:    0
[03/10/2022-16:31:39] [TRT] [I] Doc string:       
[03/10/2022-16:31:39] [TRT] [I] ----------------------------------------------------------------
[03/10/2022-16:31:39] [TRT] [W] onnx2trt_utils.cpp:365: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/10/2022-16:31:39] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
[03/10/2022-16:31:39] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
[03/10/2022-16:31:39] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
TensorRT: Network Description:
TensorRT:       input "images" with shape (1, 3, 640, 640) and dtype DataType.FLOAT
TensorRT:       output "output" with shape (1, 25200, 85) and dtype DataType.FLOAT
TensorRT:       output "350" with shape (1, 3, 80, 80, 85) and dtype DataType.FLOAT
TensorRT:       output "416" with shape (1, 3, 40, 40, 85) and dtype DataType.FLOAT
TensorRT:       output "482" with shape (1, 3, 20, 20, 85) and dtype DataType.FLOAT
TensorRT: building FP16 engine in weights/yolov5s.engine
export.py:239: DeprecationWarning: Use build_serialized_network instead.
  with builder.build_engine(network, config) as engine, open(f, 'wb') as t:
[03/10/2022-16:31:40] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.8.0 but loaded cuBLAS/cuBLAS LT 11.5.1
[03/10/2022-16:31:40] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 3358, GPU 2940 (MiB)
[03/10/2022-16:31:40] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +114, GPU +40, now: CPU 3472, GPU 2980 (MiB)
[03/10/2022-16:31:40] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[03/10/2022-16:33:54] [TRT] [I] Some tactics do not have sufficient workspace memory to run. Increasing workspace size will enable more tactics, please check verbose output for requested sizes.
[03/10/2022-16:36:08] [TRT] [I] Detected 1 inputs and 7 output network tensors.
[03/10/2022-16:36:09] [TRT] [I] Total Host Persistent Memory: 153872
[03/10/2022-16:36:09] [TRT] [I] Total Device Persistent Memory: 909824
[03/10/2022-16:36:09] [TRT] [I] Total Scratch Memory: 3264000
[03/10/2022-16:36:09] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 17 MiB, GPU 2207 MiB
[03/10/2022-16:36:09] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 10.5558ms to assign 9 blocks to 103 nodes requiring 18931204 bytes.
[03/10/2022-16:36:09] [TRT] [I] Total Activation Memory: 18931204
[03/10/2022-16:36:09] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +14, GPU +15, now: CPU 14, GPU 15 (MiB)
TensorRT: export success, saved as weights/yolov5s.engine (17.1 MB)

Export complete (273.91s)
Results saved to /home/leonardo/Scrivania/yolov5/weights
Detect:          python detect.py --weights weights/yolov5s.engine
PyTorch Hub:     model = torch.hub.load('ultralytics/yolov5', 'custom', 'weights/yolov5s.engine')
Validate:        python val.py --weights weights/yolov5s.engine
Visualize:       https://netron.app
(yolo_env) leonardo@leonardo-Precision-7750:~/Scrivania/yolov5$ python detect.py --weights weights/yolov5s.engine --imgsz 640 640 --device 0
detect: weights=['weights/yolov5s.engine'], source=data/images, data=data/coco128.yaml, imgsz=[640, 640], conf_thres=0.25, iou_thres=0.45, max_det=1000, device=0, view_img=False, save_txt=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=runs/detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False
YOLOv5 🚀 v6.1-17-g596de6d torch 1.10.2 CUDA:0 (Quadro RTX 5000, 16125MiB)

Loading weights/yolov5s.engine for TensorRT inference...
[03/10/2022-16:37:18] [TRT] [I] [MemUsageChange] Init CUDA: CPU +306, GPU +0, now: CPU 421, GPU 1614 (MiB)
[03/10/2022-16:37:18] [TRT] [I] Loaded engine size: 41 MiB
[03/10/2022-16:37:18] [TRT] [E] 1: [pluginV2Runner.cpp::load::290] Error Code 1: Serialization (Serialization assertion creator failed.Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
[03/10/2022-16:37:18] [TRT] [E] 4: [runtime.cpp::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.)
Traceback (most recent call last):
  File "detect.py", line 261, in <module>
    main(opt)
  File "detect.py", line 256, in main
    run(**vars(opt))
  File "/home/leonardo/miniconda3/envs/yolo_env/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "detect.py", line 92, in run
    model = DetectMultiBackend(weights, device=device, dnn=dnn, data=data)
  File "/home/leonardo/Scrivania/yolov5/models/common.py", line 351, in __init__
    context = model.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'
-------------------------------------------------------------------
PyCUDA ERROR: The context stack was not empty upon module cleanup.
-------------------------------------------------------------------
A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.
-------------------------------------------------------------------
Annullato (core dump creato)

As you can see, the model instantiation fails returning a NoneType object.

[lodm94]

Even the example on Colab you mentioned does not work:

[glenn-jocher]

@lodm94 your code is out of date, before doing anything I would make sure you are using the latest code.

In terms of Colab example, it works correctly. The screenshot I just posted in #6934 (comment) is from a test I ran right then (2 hours ago).

[glenn-jocher]

@lodm94 it's likely your GPU (K80) is simply too old for compatibility with current TensorRT.

EDIT: this GPU was launched almost 8 years ago, 2014.

[lodm94]

@lodm94 it's likely your GPU (K80) is simply too old for compatibility with current TensorRT.

EDIT: this GPU was launched almost 8 years ago, 2014.

This is likely to be true

@lodm94 your code is out of date, before doing anything I would make sure you are using the latest code.

Excuse me but YOLOv5 🚀 v6.1-17-g596de6d is the latest release, isn't it?

EDIT: the problem was common.py, once i updated it it works fine. I saw last commit was 4 days ago, i think i cloned the repo like 4,5 days ago :-)

[glenn-jocher]

@lodm94 yours is a few commits back (code changes frequently), because my L351 looks different than yours in your error message. Current is v6.1-24-g055e72a (7 commits/PRs ahead).

Note there's been recent updates to TensorRT export in #6798

[lodm94]

Hi @glenn-jocher thanks for your support, now inference in TRT with detect.py is working.
But there is something i can't understand in image-size definition.

I trained my model at --imgsz 960 with fixed-size training images (540, 960).
If i run detect.py --imgsz 960 with PyTorch weights the logger return:
image 6/241 /home/leonardo/Scrivania/yolov5/val_new/images/1013_0.png: 544x960 1 person, Done. (0.015s)
Note image size 544x960.

I exported two model in TRT with different imgsz: one at (544,960) and the other one at (960,960).
If i run detect.py --imgsz 960 with engine file generated at (544,960) it throws an obvious error:
AssertionError: (torch.Size([1, 3, 960, 960]), (1, 3, 544, 960))

If i run detect.py --imgsz 960 with engine file generated at (960,960) it works but the logger returns:
image 19/241 /home/leonardo/Scrivania/yolov5/val_new/images/1075_2.png: 960x960 1 person, Done. (0.007s)
Note image size 960x960

What i am trying to underline is that with detect.py i can just set one dimension for image size. So if i exported the trt model with same size for height and width, there is no problem. If, otherwise, i exported the trt model with different H and W, i don't understand how to infer on this model. With PyTorch weights there is no problem: i declare one size (960) in detect.py but inference is done at the expected image-size (544x960).

I hope you understand what i am trying to explain

[glenn-jocher]

@lodm94 I don't understand the problem. My TRT screenshot from above already shows how to export and detect at specific height and widths.

[lodm94]

@glenn-jocher Actually i figured out but i didn't manage to delete the comment that you already replied!
Export and detect are ok. Still struggling in set two different size in val.py. Differently from detect, this script require a single value. I managed to validate TRT model exported @ 960x960 but can't validate TRT model exported with H != W.

[rglkt]

hello，may be the problem is because common.py set default stride as 64 which mean the imgsz must be the multiple of 64,but you use 32 multiple imgsz 544 . you can try as my pr #7342

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

[AndreiBarsan]

For what it's worth, I had a similar this morning. The inference environment naively depended on nvidia-tensorrt without constraining its version, and the dep might have slipped to a newer one due to a new release causing the issue.

Forcing nvidia-tensorrt==8.4.1.5 (the one I confirmed was being used before I got the error) in my dependencies solved the issue in my case.

[glenn-jocher]

Thank you all for your valuable points. @AndreiBarsan, to accommodate different size inputs during validation with val.py, you can consider using the imgsz load argument to explicitly specify the desired image size. Furthermore, I recommend considering the insights provided by the community members regarding the default stride in common.py. Finally, @lodm94, your input regarding the version constraint for nvidia-tensorrt may be helpful, and others might also find it useful.

I appreciate everyone's contribution to this discussion and the collaborative effort to resolve this issue. If you have any further questions or encounter any issues, please feel free to ask.