AssertionError: No inf checks were recorded for this optimizer.

[Christian-lyc]

Hi, thank you so much for your code.
I try to use the pretrained model fine-tuning on the other dataset. But I'm facing a problem.
Do you have some idea about it? The error is listed below, thank you in advance. : )

Traceback (most recent call last):
File "main.py", line 284, in
main()
File "main.py", line 278, in main
train(args)
File "main.py", line 148, in train
amp_scale.step(optimizer)
File "/export/home/yliu/.local/lib/python3.8/site-packages/torch/cuda/amp/grad_scaler.py", line 372, in step
assert len(optimizer_state["found_inf_per_device"]) > 0, "No inf checks were recorded for this optimizer."
AssertionError: No inf checks were recorded for this optimizer.

[jahongir7174]

Hello @Christian-lyc

I tested the code on nvcr.io/nvidia/pytorch:22.08-py3 container with torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
Please check with different torch versions such as torch==1.13.1