half inference bug while using detect.py

[PresageBoat]

🐛 Bug

When using detecto.py to make predictions, if the GPU does not support fp16 operations, an error occurs in the inference and no results can be obtained; when fp16 is turned off, the results are correct

Description

case1：
usage： python.exe .\detect.py --weights .\yolov5s.pt --source .\data\images
output
Namespace(agnostic_nms=False, augment=False, classes=None, conf_thres=0.25, device='', exist_ok=False, hide_conf=False, hide_labels=False, img_size=640, iou_thres=0.45, line_thickness=3, max_det=1000, name='exp', nosave=False, project='runs/detect', save_conf=False, save_crop=False, save_txt=False, source='.\data\images', update=False, view_img=False, weights=['.\yolov5s.pt'])
YOLOv5 v5.0-106-g683cefe torch 1.8.1+cu111 CUDA:0 (Quadro T1000, 4096.0MB)

Fusing layers...
Model Summary: 224 layers, 7266973 parameters, 0 gradients
image 1/2 D:\github\yolov5\data\images\bus.jpg: 640x480 Done. (0.051s)
image 2/2 D:\github\yolov5\data\images\zidane.jpg: 384x640 Done. (0.050s)

Results saved to runs\detect\exp3
Done. (0.172s)

case2:
modify:
change ./detect.py files
line 30 :# half = device.type != 'cpu' # half precision only supported on CUDA
line 30 : half = False # half precision only supported on CUDA

usage： python.exe .\detect.py --weights .\yolov5s.pt --source .\data\images
output
Namespace(agnostic_nms=False, augment=False, classes=None, conf_thres=0.25, device='', exist_ok=False, hide_conf=False, hide_labels=False, img_size=640, iou_thres=0.45, line_thickness=3, max_det=1000, name='exp', nosave=False, project='runs/detect', save_conf=False, save_crop=False, save_txt=False, source='.\data\images', update=False, view_img=False, weights=['.\yolov5s.pt'])
YOLOv5 v5.0-106-g683cefe torch 1.8.1+cu111 CUDA:0 (Quadro T1000, 4096.0MB)

Fusing layers...
Model Summary: 224 layers, 7266973 parameters, 0 gradients
image 1/2 D:\github\yolov5\data\images\bus.jpg: 640x480 4 persons, 1 bus, 1 fire hydrant, Done. (0.044s)
image 2/2 D:\github\yolov5\data\images\zidane.jpg: 384x640 2 persons, 2 ties, Done. (0.027s)
Results saved to runs\detect\exp4
Done. (0.132s)

Expected behavior

support an automatic judgment of whether the GPU supports fp16 operation，

[glenn-jocher]

@edificewang thanks for the bug report! This seems to be a silent error since no error message is printed to screen right?

I'd thought all CUDA devices supported FP16 ops. Do you have any source for a list of unsupported devices, or do you know a way to check in realtime for FP16 support?

[glenn-jocher]

@edificewang I have an idea. Can you test inference on YOLOv5 PyTorch Hub models instead of using detect.py? The Hub models are loaded as FP32 and are run using AMP in a completely different inference pathway rather than converted to FP16. The command is:

python hubconf.py

Correct output is:

Fusing layers... 
Model Summary: 224 layers, 7266973 parameters, 0 gradients
Adding AutoShape... 
YOLOv5 🚀 v5.0-113-g73a92dc torch 1.8.1 CPU

image 1/5: 720x1280 2 persons, 2 ties
image 2/5: 720x1280 2 persons, 2 ties
image 3/5: 1080x810 4 persons, 1 bus
image 4/5: 1080x810 4 persons, 1 bus
image 5/5: 320x640
Speed: 172.5ms pre-process, 276.8ms inference, 1.4ms NMS per image at shape (5, 3, 640, 640)
Saved zidane.jpg, zidane.jpg, image2.jpg, bus.jpg, image4.jpg to runs/hub/exp6

[PresageBoat]

hello, @glenn-jocher
At present, I have not found a suitable method to automatically determine whether the GPU supports the fp16 calculation.
I will test according to your suggestion and provide the test results

[glenn-jocher]

@edificewang great thanks! I also opened up PR #3314 to modify detect.py for autocast usage, but unfortunately there seems to be a GPU speed penalty. You can test this branch also with:

git checkout glenn-jocher-patch-1
python detect.py

[PresageBoat]

@glenn-jocher

python hubconf.py


Fusing layers... 
Model Summary: 224 layers, 7266973 parameters, 0 gradients
Adding AutoShape... 
YOLOv5  v5.0-106-g683cefe torch 1.8.1+cu111 CUDA:0 (Quadro T1000, 4096.0MB)

image 1/5: 720x1280
image 2/5: 720x1280
image 3/5: 1080x810
image 4/5: 1080x810
image 5/5: 320x640
Speed: 4599.0ms pre-process, 234.9ms inference, 0.5ms NMS per image at shape (5, 3, 640, 640)
Saved zidane.jpg, zidane.jpg, image2.jpg, bus.jpg, image4.jpg to runs\hub\exp

It also output nothing!

[glenn-jocher]

@edificewang ah I see, thanks! So autocast doesn't solve the problem either.

[glenn-jocher]

@edificewang good news 😃! Your original issue may now be fixed ✅ in PR #3423. To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload with model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!