RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

[ZuoJiaxing]

Training on train env ...
Traceback (most recent call last):
File "tasks/R2R-pano/main.py", line 300, in
main(opts)
File "tasks/R2R-pano/main.py", line 265, in main
trainer.train(epoch, train_env, tb_logger)
File "/home/inceptio/Documents/VLN_github/regretful-agent/tasks/R2R-pano/trainer.py", line 46, in train
loss, traj = self.agent.rollout_regret()
File "/home/inceptio/Documents/VLN_github/regretful-agent/tasks/R2R-pano/agents/pano_agent.py", line 404, in rollout_regret
is_training=self.encoder.training)
File "/home/inceptio/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/inceptio/Documents/VLN_github/regretful-agent/tasks/R2R-pano/models/policy_model.py", line 113, in forward
proj_navigable_feat = proj_masking(navigable_feat_no_visited, self.proj_navigable_mlp, navigable_mask)
File "/home/inceptio/Documents/VLN_github/regretful-agent/tasks/R2R-pano/models/modules.py", line 95, in proj_masking
proj_feat = projector(feat.view(-1, feat.size(2)))
File "/home/inceptio/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/inceptio/.local/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/home/inceptio/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/inceptio/.local/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 83, in forward
exponential_average_factor, self.eps)
File "/home/inceptio/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1697, in batch_norm
training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED