-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Paddle-TRT] upgrade EnqueueV2 to EnqueueV3 #60807
Conversation
你的PR提交成功,感谢你对开源项目的贡献! |
手机坏了,暂时登陆不了Lizexu123的号。 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM for const_cast
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR types
Others
PR changes
Others
Description
Pcard-71501
TensorRT执行器升级,enqueueV2升级enqueueV3
使用条件:TensorRT>8.5
在A10上测量了54个模型,与enqueueV2推理时间持平。Paddle-trt的执行器功能升级。
注意:以后TensorRT 8.5版本以后,Paddle-trt都使用enqueueV3接口推理,请注意不要使用setBindingDimension和setInputShapeBinding,请使用setInputShape和setTensorAddress替代。
测试模型,单测不要删除任何op,否则会导致core down。
enqueue: oldest api, support implicit batch, is deprecated.
enqueueV2: replacement of enqueue, support explict batch. is deprecated now.
enqueueV3: latest api, support data dependent shape, recommend to use now.
Please check TensorRT: nvinfer1::IExecutionContext Class Reference 153 for details.
TensorRT runtime time (inference)
execution tensor: Runtime: IExecutionContext->setInputShape + setTensorAddress
shape tensor: IExecutionContext->setTensorAdress
注意:两者使用的API是不同的,shape tensor的时候,只使用setTensorAddress,shape_tensor I/O的address需要的是cpu point