training problem

[HakubunLuo]

When I test your code on Tesla K80 (CUDA 11.1, Pytorch 1.8.1, Ubuntu18.04), I meet a problem:

Traceback (most recent call last):
File "rotate_train.py", line 555, in
train(hyp, opt, device, tb_writer, rotate=opt.rotate)
File "rotate_train.py", line 315, in train
pred = model(imgs) # forward
File "/root/miniconda3/envs/myconda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/RotateObjectDetection/rotate-yolov5/models/yolo.py", line 122, in forward
return self.forward_once(x, profile) # single-scale inference, train
File "/root/RotateObjectDetection/rotate-yolov5/models/yolo.py", line 153, in forward_once
x = m(x) # run
File "/root/miniconda3/envs/myconda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/RotateObjectDetection/rotate-yolov5/models/common.py", line 172, in forward
return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
File "/root/miniconda3/envs/myconda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/RotateObjectDetection/rotate-yolov5/models/common.py", line 43, in forward
return self.act(self.bn(self.conv(x)))
File "/root/miniconda3/envs/myconda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/miniconda3/envs/myconda/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 399, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/root/miniconda3/envs/myconda/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc)

[XinzeLee]

Please refer to this link for more details.