CUDA Error : status execution failed when calling cublasLTMatmul

[CRS1601]

I am trying to run the provided notbook on the sample data using GPU Tesla V4 on GCP and getting this error. I searched the internet which say there is a mismatch in shape of the embedding layers.
But the model runs perfectly fine while using CPU.

Please anyone can help me resolve the issue.

[anwai98]

Hi @CRS1601,

Thanks for your interest in micro-sam.

Could you help us with a few details to figure out the issue better:

• Can you tell us which notebook are you trying to run, and in which part of the code are you getting the error?
• Can you run the following script in your GCP GPU kernel for us and share the outputs to have a better idea of your PyTorch setup:

  import torch
  print(torch.__version__)
  print(torch.version.cuda)
  print(torch.cuda.is_available())
  print(torch.cuda.get_device_name())

[CRS1601]

Hi @anwai98,

I am using the "automatic_segmentation.ipynb" notbook.

Please find the below outputs
import torch
print(torch.version)
1.3.1+cu117

print(torch.version.cuda)
11.7

print(torch.cuda.is_available())
True

print(torch.cuda.get_device_name())
'Tesla T4'

[anwai98]

Hi @CRS1601,

Most of the articles I found over the internet hint around updating the pytorch versions to the latest build. I also spotted in the Segment Anything repo that they recommend using pytorch>=1.7.

Could you try to update your pytorch version and let us know if the error is still there?