CUDA error: an illegal memory access was encountered

[aimerlove4456]

I encounter a runtimeerror about running this program.
version: gpu is 3090 torch is 1.7.1 cuda version is 11.0
pretrained mae_hivit_base_dec512d8b_hifeat_p1600lr10.pth
my command is python ./tools/train.py configs/rotated_imted/rotated_imted_hb1m_oriented_rcnn_hivitdet_base_1x_dota_ms_rr_le90_stdc_xyawh321v.py --work-dir output/outputSTD/1 --gpu-ids 1
then the result is

how can i solve this problem?it is hard for me to read code and find up error in where.

[yuhongtian17]

This error may occur if you are sharing the GPU server with others, and the CPU usage or memory usage of the server is full.