fix(cuda): downgrade to 12.0 to increase compatibility range

[mudler]

Description

This PR is an attempt to fix #2394

The PR needs more testing too - especially on newer CUDA/drivers combo to see if it's compatible with new driver versions.

Relevant CUDA docs: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id5

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	f2e7057
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/669feef5024dd500083755d6
😎 Deploy Preview	https://deploy-preview-2994--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dave-gray101]

@mudler - specifically on master only what do you think about trying to build an image against both 12.0 (instead of 12.4 apparently) and the latest 12.5? Is there any value in adding an extra matrix entry in that situation so we can be aware of upcoming incompatibilities / errors before they hit us severely?

[mudler]

@mudler - specifically on master only what do you think about trying to build an image against both 12.0 (instead of 12.4 apparently) and the latest 12.5? Is there any value in adding an extra matrix entry in that situation so we can be aware of upcoming incompatibilities / errors before they hit us severely?

What I'd like to try first if 12.0 works as well on hosts that have 12.4 and 12.5 - I have access to a machine with 12.4, and runpod has 12.2 - I miss only 12.5 to double check.

From the CUDA docs it looks like it does, but needs some cross-check first to see if that's really the case.

Adding another host to the matrix would be my last resort - It really puts too much into CI/docs/installation scripts, but we can do it if really needed

[mudler]

waiting for https://github.com/mudler/LocalAI/actions/runs/10060851192/job/27809452863?pr=2994 to push a test image so can give that a shot, and will report later the hosts I could test on.

[dave-gray101]

@mudler - specifically on master only what do you think about trying to build an image against both 12.0 (instead of 12.4 apparently) and the latest 12.5? Is there any value in adding an extra matrix entry in that situation so we can be aware of upcoming incompatibilities / errors before they hit us severely?

What I'd like to try first if 12.0 works as well on hosts that have 12.4 and 12.5 - I have access to a machine with 12.4, and rupod has 12.2 - I miss only 12.4 to double check.

Adding another host to the matrix would be my last resort - It really puts too much into CI/docs/installation scripts, but we can do it if really needed

Works for me - I'll admit I'm a bit confused with all the different versions of CUDA 12 floating around - I have only have a 12.5 CUDA dev laptop, but the docker host environment is temporarily snared up in an WSL/Docker incompatibility. I'll try to downgrade things and get it up and running again if we need to run more tests there locally.

[mudler]

@mudler - specifically on master only what do you think about trying to build an image against both 12.0 (instead of 12.4 apparently) and the latest 12.5? Is there any value in adding an extra matrix entry in that situation so we can be aware of upcoming incompatibilities / errors before they hit us severely?

What I'd like to try first if 12.0 works as well on hosts that have 12.4 and 12.5 - I have access to a machine with 12.4, and rupod has 12.2 - I miss only 12.4 to double check.
Adding another host to the matrix would be my last resort - It really puts too much into CI/docs/installation scripts, but we can do it if really needed

Works for me - I'll admit I'm a bit confused with all the different versions of CUDA 12 floating around - I have only have a 12.5 CUDA dev laptop, but the docker host environment is temporarily snared up in an WSL/Docker incompatibility. I'll try to downgrade things and get it up and running again if we need to run more tests there locally.

12.5 sounds good - sorry I did a mistake in writing, I miss 12.5 in my test matrix

[mudler]

Works with host with CUDA 12.2 (runpod.io)

logs.txt

[mudler]

Works as well on 12.4:

Tue Jul 23 16:39:52 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    On  |   00000000:01:00.0 Off |                  Off |
| N/A   44C    P0            103W /  350W |    5202MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

@dave-gray101 can you test on 12.5? container image: ttl.sh/localai-ci-pr-2994:sha-1a89570-cublas-cuda12-ffmpeg