Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory #483

Open
3 tasks done
Amit30swgoh opened this issue Jul 18, 2024 · 4 comments
Open
3 tasks done

[ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory #483

Amit30swgoh opened this issue Jul 18, 2024 · 4 comments
Labels
bug Something isn't working new

Comments

@Amit30swgoh
Copy link

First, confirm

  • I have read the instruction carefully
  • I have searched the existing issues
  • I have updated the extension to the latest version

What happened?

image
image
[ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

please help me because i want to run it faster useing GPU

Steps to reproduce the problem

when running Reactor fast face getting ONNXRuntimeError Error :( please help friend.

Sysinfo

google colab pro

Relevant console log

FETCH DATA from: /content/drive/MyDrive/ComfyUI/custom_nodes/ComfyUI-Manager/extension-node-map.json [DONE]
#clip_model_path chatml None
#model 
start_local_llm error
#read_workflow_json_files_all /content/drive/MyDrive/ComfyUI/custom_nodes/comfyui-mixlab-nodes/app/
got prompt
[rgthree] Using rgthree's optimized recursive execution.
[rgthree] First run patching recursive_output_delete_if_changed and recursive_will_execute.
[rgthree] Note: If execution seems broken due to forward ComfyUI changes, you can disable the optimization from rgthree settings in ComfyUI.
[ReActor] 12:36:01 - STATUS - Working: source face index [0], target face index [0]
[ReActor] 12:36:02 - STATUS - Analyzing Source Image...
2024-07-18 12:36:02.316082451 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

2024-07-18 12:36:03.004577726 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

2024-07-18 12:36:03.059246241 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

2024-07-18 12:36:03.100719245 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

2024-07-18 12:36:03.411743966 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

[ReActor] 12:36:04 - STATUS - Analyzing Target Image 0...
[ReActor] 12:36:04 - STATUS - Analyzing Target Image 1...
[ReActor] 12:36:05 - STATUS - Analyzing Target Image 2...
[ReActor] 12:36:05 - STATUS - Analyzing Target Image 3...
[ReActor] 12:36:05 - STATUS - Analyzing Target Image 4...
[ReActor] 12:36:06 - STATUS - Analyzing Target Image 5...
[ReActor] 12:36:06 - STATUS - Analyzing Target Image 6...
[ReActor] 12:36:06 - STATUS - Analyzing Target Image 7...
[ReActor] 12:36:07 - STATUS - Analyzing Target Image 8...
[ReActor] 12:36:07 - STATUS - Analyzing Target Image 9...
[ReActor] 12:36:07 - STATUS - Analyzing Target Image 10...
[ReActor] 12:36:08 - STATUS - Analyzing Target Image 11...
[ReActor] 12:36:08 - STATUS - Analyzing Target Image 12...
[ReActor] 12:36:08 - STATUS - Analyzing Target Image 13...
[ReActor] 12:36:09 - STATUS - Analyzing Target Image 14...
[ReActor] 12:36:09 - STATUS - Analyzing Target Image 15...
[ReActor] 12:36:09 - STATUS - Analyzing Target Image 16...
[ReActor] 12:36:10 - STATUS - Analyzing Target Image 17...
[ReActor] 12:36:10 - STATUS - Analyzing Target Image 18...
[ReActor] 12:36:10 - STATUS - Analyzing Target Image 19...
[ReActor] 12:36:11 - STATUS - Analyzing Target Image 20...
[ReActor] 12:36:11 - STATUS - Analyzing Target Image 21...
[ReActor] 12:36:11 - STATUS - Analyzing Target Image 22...
[ReActor] 12:36:12 - STATUS - Analyzing Target Image 23...
[ReActor] 12:36:12 - STATUS - Analyzing Target Image 24...
[ReActor] 12:36:13 - STATUS - Analyzing Target Image 25...
[ReActor] 12:36:13 - STATUS - Analyzing Target Image 26...
[ReActor] 12:36:13 - STATUS - Analyzing Target Image 27...
[ReActor] 12:36:14 - STATUS - Analyzing Target Image 28...
[ReActor] 12:36:14 - STATUS - Analyzing Target Image 29...
[ReActor] 12:36:15 - STATUS - Analyzing Target Image 30...
[ReActor] 12:36:15 - STATUS - Analyzing Target Image 31...
[ReActor] 12:36:15 - STATUS - Analyzing Target Image 32...
[ReActor] 12:36:16 - STATUS - Analyzing Target Image 33...
[ReActor] 12:36:16 - STATUS - Analyzing Target Image 34...
[ReActor] 12:36:17 - STATUS - Analyzing Target Image 35...
[ReActor] 12:36:17 - STATUS - Analyzing Target Image 36...
[ReActor] 12:36:17 - STATUS - Analyzing Target Image 37...
[ReActor] 12:36:18 - STATUS - Analyzing Target Image 38...
[ReActor] 12:36:18 - STATUS - Analyzing Target Image 39...
[ReActor] 12:36:18 - STATUS - Analyzing Target Image 40...
[ReActor] 12:36:19 - STATUS - Analyzing Target Image 41...
[ReActor] 12:36:19 - STATUS - Analyzing Target Image 42...
[ReActor] 12:36:19 - STATUS - Analyzing Target Image 43...
[ReActor] 12:36:20 - STATUS - Analyzing Target Image 44...
[ReActor] 12:36:20 - STATUS - Analyzing Target Image 45...
[ReActor] 12:36:20 - STATUS - Analyzing Target Image 46...
[ReActor] 12:36:21 - STATUS - Source Face 0: Detected Gender -F-
[ReActor] 12:36:21 - STATUS - OK - Detected Gender matches Condition
2024-07-18 12:36:22.077235867 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

[ReActor] 12:36:25 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:25 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:25 - STATUS - Swapping 0...
[ReActor] 12:36:26 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:26 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:26 - STATUS - Swapping 1...
[ReActor] 12:36:28 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:28 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:28 - STATUS - Swapping 2...
[ReActor] 12:36:30 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:30 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:30 - STATUS - Swapping 3...
[ReActor] 12:36:31 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:31 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:31 - STATUS - Swapping 4...
[ReActor] 12:36:33 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:33 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:33 - STATUS - Swapping 5...
[ReActor] 12:36:34 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:34 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:34 - STATUS - Swapping 6...
[ReActor] 12:36:36 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:36 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:36 - STATUS - Swapping 7...
[ReActor] 12:36:38 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:38 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:38 - STATUS - Swapping 8...
[ReActor] 12:36:39 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:39 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:39 - STATUS - Swapping 9...
[ReActor] 12:36:39 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:39 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:39 - STATUS - Swapping 10...
[ReActor] 12:36:40 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:40 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:40 - STATUS - Swapping 11...
[ReActor] 12:36:41 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:41 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:41 - STATUS - Swapping 12...
[ReActor] 12:36:42 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:42 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:42 - STATUS - Swapping 13...
[ReActor] 12:36:44 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:44 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:44 - STATUS - Swapping 14...
[ReActor] 12:36:45 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:45 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:45 - STATUS - Swapping 15...
[ReActor] 12:36:47 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:47 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:47 - STATUS - Swapping 16...
[ReActor] 12:36:48 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:48 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:48 - STATUS - Swapping 17...
[ReActor] 12:36:49 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:49 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:49 - STATUS - Swapping 18...
[ReActor] 12:36:50 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:50 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:50 - STATUS - Swapping 19...
[ReActor] 12:36:50 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:50 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:50 - STATUS - Swapping 20...
[ReActor] 12:36:51 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:51 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:51 - STATUS - Swapping 21...
[ReActor] 12:36:52 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:52 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:52 - STATUS - Swapping 22...
[ReActor] 12:36:53 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:53 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:53 - STATUS - Swapping 23...
[ReActor] 12:36:54 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:54 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:54 - STATUS - Swapping 24...
[ReActor] 12:36:55 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:55 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:55 - STATUS - Swapping 25...
[ReActor] 12:36:56 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:56 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:56 - STATUS - Swapping 26...
[ReActor] 12:36:56 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:56 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:56 - STATUS - Swapping 27...
[ReActor] 12:36:58 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:58 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:58 - STATUS - Swapping 28...
[ReActor] 12:36:59 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:36:59 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:36:59 - STATUS - Swapping 29...
[ReActor] 12:37:01 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:01 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:01 - STATUS - Swapping 30...
[ReActor] 12:37:03 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:03 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:03 - STATUS - Swapping 31...
[ReActor] 12:37:04 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:04 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:04 - STATUS - Swapping 32...
[ReActor] 12:37:06 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:06 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:06 - STATUS - Swapping 33...
[ReActor] 12:37:07 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:07 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:07 - STATUS - Swapping 34...
[ReActor] 12:37:08 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:08 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:08 - STATUS - Swapping 35...
[ReActor] 12:37:09 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:09 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:09 - STATUS - Swapping 36...
[ReActor] 12:37:10 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:10 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:10 - STATUS - Swapping 37...
[ReActor] 12:37:11 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:11 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:11 - STATUS - Swapping 38...
[ReActor] 12:37:12 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:12 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:12 - STATUS - Swapping 39...
[ReActor] 12:37:14 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:14 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:14 - STATUS - Swapping 40...
[ReActor] 12:37:15 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:15 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:15 - STATUS - Swapping 41...
[ReActor] 12:37:17 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:17 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:17 - STATUS - Swapping 42...
[ReActor] 12:37:19 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:19 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:19 - STATUS - Swapping 43...
[ReActor] 12:37:20 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:20 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:20 - STATUS - Swapping 44...
[ReActor] 12:37:21 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:21 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:21 - STATUS - Swapping 45...
[ReActor] 12:37:22 - STATUS - Target Face 0: Detected Gender -F-
[ReActor] 12:37:22 - STATUS - OK - Detected Gender matches Condition
[ReActor] 12:37:22 - STATUS - Swapping 46...
[ReActor] 12:37:23 - STATUS - --Done!--
Comfy-VFI: Clearing cache... Done cache clearing
Comfy-VFI: Clearing cache... Done cache clearing
Comfy-VFI: Clearing cache... Done cache clearing
Comfy-VFI: Clearing cache... Done cache clearing
Comfy-VFI: Final clearing cache... Done cache clearing

Additional information

No response
@Amit30swgoh Amit30swgoh added bug Something isn't working new labels Jul 18, 2024
@Amit30swgoh
Copy link
Author

Amit30swgoh commented Jul 18, 2024 •
edited
Loading

please help @Gourieff perhaps its because of onnxruntime-gpu 1.18.1? perhaps need to change to 1.15? or move back to "insightface==0.7.3
onnx>=1.14.0
opencv-python>=4.7.0.72
numpy
segment_anything" ?

@Amit30swgoh
Copy link
Author

managed to fix but it doesnt utilize vram so 10 sec vid takes 40 min :(
image

@Amit30swgoh Amit30swgoh reopened this Jul 19, 2024
@andreclaudino
Copy link

I have this exactly same problem on ComfyUI start. Tyried to downgrade packages as you recommended but didn't work.

@aifantasyglam
Copy link

same problem using Colab Pro, A1111, SD1.5, but just with the first image

image

@aifantasyglam aifantasyglam mentioned this issue Aug 27, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working new
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@andreclaudino @Amit30swgoh @aifantasyglam