Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yololayer cuda file for batch processing with V2DynamicExt #20

Open
MariJothi opened this issue Mar 16, 2023 · 2 comments
Open

Yololayer cuda file for batch processing with V2DynamicExt #20

MariJothi opened this issue Mar 16, 2023 · 2 comments

Comments

@MariJothi
Copy link

Hi @nanmi,
I want to run the tensorrt model with batchsize 8, but when i infer with batch 8, i am getting results for first batch and the remaining batch outputs are zero. because this model only support for batch 1. So i successfully exported onnx model with dynamic batch size, but when i convert to tensorrt i am gettting below error

ERROR: [TRT]: YoloLayer_TRT_0: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
ERROR: [TRT]: ModelImporter.cpp:726: While parsing node number 495 [YoloLayer_TRT -> "output0"]:
ERROR: [TRT]: ModelImporter.cpp:727: --- Begin node ---
ERROR: [TRT]: ModelImporter.cpp:728: input: "745"
input: "783"
input: "821"
input: "859"
output: "output0"
name: "YoloLayer_TRT_0"
op_type: "YoloLayer_TRT"

ERROR: [TRT]: ModelImporter.cpp:729: --- End node ---
ERROR: [TRT]: ModelImporter.cpp:732: ERROR: ModelImporter.cpp:185 In function parseGraph:
[6] Invalid Node - YoloLayer_TRT_0
YoloLayer_TRT_0: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.

@YunghuiHsu
Copy link

I'm having the same problem with dynamic batch conversion (same error message), has anyone solved this problem yet?

Here is my procedure:

  1. pt > onnx
    python3 export.py --weights yolov7-w6-pose.pt --img-size 960 --fp16 --dynamic-batch --device=0

  2. insert yololayer
    python3 yolov7-pose/YoloLayer_TRT_v7.0/script/add_custom_yolo_op.py
    with

dim_outputs = n_maxoutput * n_outputs + 1  # (1000 x 57 +1)
if dynamic_batch :
    out_shape = (gs.Tensor.DYNAMIC, dim_outputs, 1, 1) 
  1. onnx > trt engine
/usr/src/tensorrt/bin/trtexec --verbose --onnx=yolov7-w6-pose-dynamic_960-yololayer.onnx --fp16 \
 --minShapes=images:1x3x960x960 \
--optShapes=images:12x3x960x960 \
 --maxShapes=images:16x3x960x960 \
 --saveEngine=yolov7_w6_pose_dy_fp16_960-yololayer.engine \
 --plugins=yolov7-pose/YoloLayer_TRT_v7.0/build/libyolo.so 

@YunghuiHsu
Copy link

Rewriting the IPluginv2DynamicExt Class instead of IPluginv2Ext Class in yololayer.cu is one solution to support dynamic batch.

But I used yolov8-pose (which supports dynamic batch processing) and integrated it into the Deepstream Python API.
Detailed in deepstream-yolo-pose

reference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants