Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support model-only inference #733

Merged
merged 5 commits into from
Apr 20, 2023
Merged

Support model-only inference #733

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
dbf329b
Support model-only inference
triple-Mu Apr 17, 2023
0d796d7
Fix ppyoloe std
triple-Mu Apr 17, 2023
7d85cee
Add doc
triple-Mu Apr 17, 2023
6fec243
Fix typo
triple-Mu Apr 17, 2023
daa1d47
Rename
triple-Mu Apr 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 101 additions & 2 deletions projects/easydeploy/docs/model_convert.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# MMYOLO 模型 ONNX 转换

## 1. 导出后端支持的 ONNX

## 环境依赖

- [onnx](https://github.com/onnx/onnx)
Expand All @@ -14,6 +16,8 @@
pip install onnx-simplifier
```

\*\*\* 请确保您在 `MMYOLO` 根目录下运行相关脚本，避免无法找到相关依赖包。\*\*\*

## 使用方法

[模型导出脚本](./projects/easydeploy/tools/export_onnx.py)用于将 `MMYOLO` 模型转换为 `onnx` 。
Expand All @@ -28,7 +32,7 @@
- `--device`: 转换模型使用的设备，默认为 `cuda:0`。
- `--simplify`: 是否简化导出的 `onnx` 模型，需要安装 [onnx-simplifier](https://github.com/daquexian/onnx-simplifier)，默认关闭。
- `--opset`: 指定导出 `onnx` 的 `opset`，默认为 `11` 。
- `--backend`: 指定导出 `onnx` 用于的后端 id，`ONNXRuntime`: `onnxruntime`, `TensorRT8`: `tensorrt8`, `TensorRT7`: `tensorrt7`，默认为`onnxruntime`即 `ONNXRuntime`。
- `--backend`: 指定导出 `onnx` 用于的后端名称，`ONNXRuntime`: `onnxruntime`, `TensorRT8`: `tensorrt8`, `TensorRT7`: `tensorrt7`，默认为`onnxruntime`即 `ONNXRuntime`。
- `--pre-topk`: 指定导出 `onnx` 的后处理筛选候选框个数阈值，默认为 `1000`。
- `--keep-topk`: 指定导出 `onnx` 的非极大值抑制输出的候选框个数阈值，默认为 `100`。
- `--iou-threshold`: 非极大值抑制中过滤重复候选框的 `iou` 阈值，默认为 `0.65`。
Expand All @@ -54,4 +58,99 @@ python ./projects/easydeploy/tools/export.py \
--score-threshold 0.25
```

然后利用后端支持的工具如 `TensorRT` 读取 `onnx` 再次转换为后端支持的模型格式如 `.engine/.plan` 等
然后利用后端支持的工具如 `TensorRT` 读取 `onnx` 再次转换为后端支持的模型格式如 `.engine/.plan` 等。

`MMYOLO` 目前支持 `TensorRT8`, `TensorRT7`, `ONNXRuntime` 后端的端到端模型转换，目前仅支持静态 shape 模型的导出和转换，动态 batch 或动态长宽的模型端到端转换会在未来继续支持。

端到端转换得到的 `onnx` 模型输入输出如图：

<div align=center>
<img src="https://user-images.githubusercontent.com/92794867/232403745-101ca999-2003-46fa-bc5b-6b0eb2b2d41b.png"/>
</div>

输入名: `images`, 尺寸 640x640

输出名: `num_dets`, 尺寸 1x1，表示检测目标数量。

输出名: `boxes`, 尺寸 1x100x4，表示检测框的坐标，格式为 `x1y1x2y1`。

输出名: `scores`, 尺寸 1x100，表示检测框的分数。

输出名: `labels`, 尺寸 1x100，表示检测框的类别 id。

可以利用 `num_dets` 中的个数对 `boxes`, `scores`, `labels` 进行截断，从 100 个检测结果中抽取前 `num_dets` 个目标作为最终检测结果。

## 2. 仅导出模型 Backbone + Neck

当您需要部署在非 `TensorRT`, `ONNXRuntime` 等支持端到端部署的平台时，您可以考虑使用`--model-only` 参数并且不要传递 `--backend` 参数，您将会导出仅包含 `Backbone` + `neck` 的模型，模型的部分输出如图:

<div align=center>
<img src="https://user-images.githubusercontent.com/92794867/232406169-40eee9fd-bc53-4fdc-bd37-d0e9033826f9.png"/>
</div>

这种导出方式获取的 `ONNX` 模型具有如下优点:

- 算子简单，一般而言只包含 `Conv`，激活函数等简单算子，几乎不存在无法正确导出的情况，对于嵌入式部署更加友好。
- 方便不同算法之间对比速度性能，由于不同的算法后处理不同，仅对比 `backbone` + `Neck` 的速度更加公平。

也有如下缺点:

- 后处理逻辑需要单独完成，会有额外的 `decode` + `nms` 的操作需要实现。
- 与 `TensorRT` 相比，由于 `TensorRT` 可以利用多核优势并行进行后处理，使用 `--model-only` 方式导出的模型性能会差很多。

### 使用方法

```shell
python ./projects/easydeploy/tools/export.py \
configs/yolov5/yolov5_s-v61_syncbn_fast_8xb16-300e_coco.py \
yolov5s.pth \
--work-dir work_dir \
--img-size 640 640 \
--batch 1 \
--device cpu \
--simplify \
--opset 11 \
--model-only
```

## 使用 `model-only` 导出的 ONNX 进行推理

[模型推理脚本](./projects/easydeploy/examples/main_onnxruntime.py)用于推理导出的 `ONNX` 模型，需要安装基础依赖环境:

[`onnxruntime`](https://github.com/microsoft/onnxruntime) 和 [`opencv-python`](https://github.com/opencv/opencv-python)

```shell
pip install onnxruntime
pip install opencv-python==4.7.0.72 # 建议使用最新的 opencv
```

### 参数介绍:

- `img` : 待检测的图片路径或图片文件夹路径。
- `onnx` : 导出的 `model-only` ONNX 模型。
- `--type` : 模型名称，目前支持 `yolov5`, `yolox`, `yolov6`, `ppyoloe`, `ppyoloep`, `yolov7`, `rtmdet`, `yolov8`。
- `--img-size`: 转换模型时输入的尺寸，如 `640 640`。
- `--out-dir`: 保存检测结果的路径 。
- `--show`: 是否可视化检测结果。
- `--score-thr`: 模型检测后处理的置信度分数 。
- `--iou-thr`: 模型检测后处理的 IOU 分数 。

## 使用方法

```shell
cd ./projects/easydeploy/examples
python main_onnxruntime.py \
"image_path_to_detect" \
yolov5_s_model-only.onnx \
--out-dir work_dir \
--img-size 640 640 \
--show \
--score-thr 0.3 \
--iou-thr 0.7
```

*注意！！！*

当您使用自定义数据集训练得到的模型时，请修改 [`config.py`](./projects/easydeploy/examples/config.py) 中 `CLASS_NAMES` 和 `CLASS_COLORS`，如果是 `yolov5` 或者 `yolov7` 基于 `anchor` 的模型请同时修改 `YOLOv5_ANCHORS` 和 `YOLOv7_ANCHORS`。

[`numpy_coder.py`](./projects/easydeploy/examples/numpy_coder.py) 是目前所有算法仅使用 `numpy` 实现的 `decoder`，如果您对性能有较高的要求，可以参照相关代码改写为 `c/c++`。
64 changes: 64 additions & 0 deletions projects/easydeploy/examples/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
from enum import Enum


class TASK_TYPE(Enum):
DET = 'det'
SEG = 'seg'
POSE = 'pose'


class ModelType(Enum):
YOLOV5 = 'yolov5'
YOLOX = 'yolox'
PPYOLOE = 'ppyoloe'
PPYOLOEP = 'ppyoloep'
YOLOV6 = 'yolov6'
YOLOV7 = 'yolov7'
RTMDET = 'rtmdet'
YOLOV8 = 'yolov8'


CLASS_NAMES = ('person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus',
'train', 'truck', 'boat', 'traffic light', 'fire hydrant',
'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog',
'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe',
'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee',
'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat',
'baseball glove', 'skateboard', 'surfboard', 'tennis racket',
'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl',
'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot',
'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch',
'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop',
'mouse', 'remote', 'keyboard', 'cell phone', 'microwave',
'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock',
'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush')

CLASS_COLORS = [(220, 20, 60), (119, 11, 32), (0, 0, 142), (0, 0, 230),
(106, 0, 228), (0, 60, 100), (0, 80, 100), (0, 0, 70),
(0, 0, 192), (250, 170, 30), (100, 170, 30), (220, 220, 0),
(175, 116, 175), (250, 0, 30), (165, 42, 42), (255, 77, 255),
(0, 226, 252), (182, 182, 255), (0, 82, 0), (120, 166, 157),
(110, 76, 0), (174, 57, 255), (199, 100, 0), (72, 0, 118),
(255, 179, 240), (0, 125, 92), (209, 0, 151), (188, 208, 182),
(0, 220, 176), (255, 99, 164), (92, 0, 73), (133, 129, 255),
(78, 180, 255), (0, 228, 0), (174, 255, 243), (45, 89, 255),
(134, 134, 103), (145, 148, 174), (255, 208, 186),
(197, 226, 255), (171, 134, 1), (109, 63, 54), (207, 138, 255),
(151, 0, 95), (9, 80, 61), (84, 105, 51), (74, 65, 105),
(166, 196, 102), (208, 195, 210), (255, 109, 65),
(0, 143, 149), (179, 0, 194), (209, 99, 106), (5, 121, 0),
(227, 255, 205), (147, 186, 208), (153, 69, 1), (3, 95, 161),
(163, 255, 0), (119, 0, 170), (0, 182, 199), (0, 165, 120),
(183, 130, 88), (95, 32, 0), (130, 114, 135), (110, 129, 133),
(166, 74, 118), (219, 142, 185), (79, 210, 114), (178, 90, 62),
(65, 70, 15), (127, 167, 115), (59, 105, 106), (142, 108, 45),
(196, 172, 0), (95, 54, 80), (128, 76, 255), (201, 57, 1),
(246, 0, 122), (191, 162, 208)]

YOLOv5_ANCHORS = [[(10, 13), (16, 30), (33, 23)],
[(30, 61), (62, 45), (59, 119)],
[(116, 90), (156, 198), (373, 326)]]

YOLOv7_ANCHORS = [[(12, 16), (19, 36), (40, 28)],
[(36, 75), (76, 55), (72, 146)],
[(142, 110), (192, 243), (459, 401)]]
36 changes: 36 additions & 0 deletions projects/easydeploy/examples/cv2_nms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
from typing import List, Tuple, Union

import cv2
from numpy import ndarray

MAJOR, MINOR = map(int, cv2.__version__.split('.')[:2])
assert MAJOR == 4


def non_max_suppression(boxes: Union[List[ndarray], Tuple[ndarray]],
scores: Union[List[float], Tuple[float]],
labels: Union[List[int], Tuple[int]],
conf_thres: float = 0.25,
iou_thres: float = 0.65) -> Tuple[List, List, List]:
if MINOR >= 7:
indices = cv2.dnn.NMSBoxesBatched(boxes, scores, labels, conf_thres,
iou_thres)
elif MINOR == 6:
indices = cv2.dnn.NMSBoxes(boxes, scores, conf_thres, iou_thres)
else:
indices = cv2.dnn.NMSBoxes(boxes, scores, conf_thres,
iou_thres).flatten()

nmsd_boxes = []
nmsd_scores = []
nmsd_labels = []
for idx in indices:
box = boxes[idx]
# x0y0wh -> x0y0x1y1
box[2:] = box[:2] + box[2:]
score = scores[idx]
label = labels[idx]
nmsd_boxes.append(box)
nmsd_scores.append(score)
nmsd_labels.append(label)
return nmsd_boxes, nmsd_scores, nmsd_labels
110 changes: 110 additions & 0 deletions projects/easydeploy/examples/main_onnxruntime.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
import math
import sys
from argparse import ArgumentParser
from pathlib import Path

import cv2
import onnxruntime
from config import (CLASS_COLORS, CLASS_NAMES, ModelType, YOLOv5_ANCHORS,
YOLOv7_ANCHORS)
from cv2_nms import non_max_suppression
from numpy_coder import Decoder
from preprocess import Preprocess
from tqdm import tqdm

# Add __FILE__ to sys.path
sys.path.append(str(Path(__file__).resolve().parents[0]))

IMG_EXTENSIONS = ('.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif',
'.tiff', '.webp')


def path_to_list(path: str):
path = Path(path)
if path.is_file() and path.suffix in IMG_EXTENSIONS:
res_list = [str(path.absolute())]
elif path.is_dir():
res_list = [
str(p.absolute()) for p in path.iterdir()
if p.suffix in IMG_EXTENSIONS
]
else:
raise RuntimeError
return res_list


def parse_args():
parser = ArgumentParser()
parser.add_argument(
'img', help='Image path, include image file, dir and URL.')
parser.add_argument('onnx', type=str, help='Onnx file')
parser.add_argument('--type', type=str, help='Model type')
parser.add_argument(
'--img-size',
nargs='+',
type=int,
default=[640, 640],
help='Image size of height and width')
parser.add_argument(
'--out-dir', default='./output', type=str, help='Path to output file')
parser.add_argument(
'--show', action='store_true', help='Show the detection results')
parser.add_argument(
'--score-thr', type=float, default=0.3, help='Bbox score threshold')
parser.add_argument(
'--iou-thr', type=float, default=0.7, help='Bbox iou threshold')
args = parser.parse_args()
return args


def main():
args = parse_args()
out_dir = Path(args.out_dir)
model_type = ModelType(args.type.lower())

if not args.show:
out_dir.mkdir(parents=True, exist_ok=True)

files = path_to_list(args.img)
session = onnxruntime.InferenceSession(
args.onnx, providers=['CPUExecutionProvider'])
preprocessor = Preprocess(model_type)
decoder = Decoder(model_type, model_only=True)
if model_type == ModelType.YOLOV5:
anchors = YOLOv5_ANCHORS
elif model_type == ModelType.YOLOV7:
anchors = YOLOv7_ANCHORS
else:
anchors = None

for file in tqdm(files):
image = cv2.imread(file)
image_h, image_w = image.shape[:2]
img, (ratio_w, ratio_h) = preprocessor(image, args.img_size)
features = session.run(None, {'images': img})
decoder_outputs = decoder(
features,
args.score_thr,
num_labels=len(CLASS_NAMES),
anchors=anchors)
nmsd_boxes, nmsd_scores, nmsd_labels = non_max_suppression(
*decoder_outputs, args.score_thr, args.iou_thr)
for box, score, label in zip(nmsd_boxes, nmsd_scores, nmsd_labels):
x0, y0, x1, y1 = box
x0 = math.floor(min(max(x0 / ratio_w, 1), image_w - 1))
y0 = math.floor(min(max(y0 / ratio_h, 1), image_h - 1))
x1 = math.ceil(min(max(x1 / ratio_w, 1), image_w - 1))
y1 = math.ceil(min(max(y1 / ratio_h, 1), image_h - 1))
cv2.rectangle(image, (x0, y0), (x1, y1), CLASS_COLORS[label], 2)
cv2.putText(image, f'{CLASS_NAMES[label]}: {score:.2f}',
(x0, y0 - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.5,
(0, 255, 255), 2)
if args.show:
cv2.imshow('result', image)
cv2.waitKey(0)
else:
cv2.imwrite(f'{out_dir / Path(file).name}', image)


if __name__ == '__main__':
main()
Loading