Unable to setup with GPU/CUDA

[ad-mos]

I have successfully setup all drivers, CUDA toolkit, cuDNN, builded up openCV with CUDA support, but it's still looks like it ignores system GPU.

I am tested that by running /examples/py/rest_examples/simple_flask/server.py with threaded=True web serving.
Added some debug output for checking GPU/CUDA status for cv2 and pytorch

Performance tested with ab benchmark by sending /detect requests to flask

CPU usage is 100%, but GPU only 4-12%.
Benchmark shows average proccessing time 1.4s.

Tried installing different driver versions, cuda toolkits and etc. Wasted few days but no success.
Does anybody know, what am I doing wrong?
Thanks.

[ad-mos]

Tried even running docker GPU image (/docker/build-gpu.sh + /docker/run-gpu.sh) but it can't start even the basic examples and also seems like it's not using GPU.

[ApelSYN]

We connect YOLO with "default" settings and then change them. In previous versions of yolo you could specify 'cuda', now you have to write the GPU number, for example '0'
You can change the settings for your environment in this file:
pipes/number_plate_localizators/yolo_v5_detector.py

python model = torch.hub.load(repo_path, 'custom', device="0", path=weights, source="local")

We will improve it in future versions.

[ad-mos]

Debug output of your code:

NUMBERPLATE CLASSIFICATION
cuda:0
TEXT DETECORS
cuda:0
cuda:0
cuda:0
cuda:0
cuda:0
cuda:0
cuda:0
cuda:0
cuda:0
cuda:0
YOLO
cuda:0
CRAFT
cuda:0
cuda:0

Performance comparison:

1. Big one - 6MB (3472x4624px) + concurency 10
   GPU Load = 14-22% (before 4-15%)
   Time for processing 1 image = 0.63s (before 1.4s)

2. Big one - 6MB (3472x4624px) + concurency 1
   GPU Load = 2-6% (before 0-5%)
   Time for processing 1 image = 0.2s (before 0.92s)

3. Small compressed - 22.6KB (685x514px) + concurency 10
   GPU Load = 15-22% (before 17-21%)
   Time for processing 1 image = 0.471s (before 0.438s)

4. Small compressed - 22.6KB (685x514px) + concurency 1
   GPU Load = 12-14% (before 12-13%)
   Time for processing 1 image = 0.058s (before 0.063s)

Screenshot of benchmarking (Big one - 6MB + concurency 10)

[ad-mos]

Benchmark timer stats:

Big one - 6MB (3472x4624px) - one shot

Processed 1 photo
One photo process 0.19008541107177734 seconds

detect_bbox_time_all 0.020519256591796875 per one photo
craft_time_all 0.03821420669555664 per one photo
classification_time_all 0.005549192428588867 per one photo
ocr_time_all 0.009549140930175781 per one photo
127.0.0.1 - - [23/Jan/2023 05:43:07] "POST /detect HTTP/1.1" 200 -

Small compressed - 22.6KB (685x514px) - one shot

Processed 1 photo
One photo process 0.05509495735168457 seconds

detect_bbox_time_all 0.020476818084716797 per one photo
craft_time_all 0.016835451126098633 per one photo
classification_time_all 0.005713224411010742 per one photo
ocr_time_all 0.009476423263549805 per one photo
127.0.0.1 - - [23/Jan/2023 05:43:27] "POST /detect HTTP/1.1" 200 -

It's still strange that it can't use 100% of GPU but only 15-25%, but maybe it's okay.

Thank you very much for your help!

[ad-mos]

Noticed that replacing opencv with turbo - greatly reduced recognition accuracy, so unfortunately, it's not usable in most cases for me

[ApelSYN]

Show examples on which this problem can be reproduced.

[ad-mos]

I will send it on your email

[ApelSYN]

This is taking too long to do, I'll go over the settings tomorrow.