You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Running lastest AnythingLLM on Docker in Linux Ubuntu 20.4 Server
Needed to use Lance_revert image though to prevent crashes
NVIDIA Driver Version: 470.256.02
CUDA Version: 11.4
nvidia-smi gives no errors but no processes are found = AnythingLLM is not using the GPU :-(
I know it´s an outdated GPU but maybe there is a workaround?
This would be entirely dependent on the LLM provider you are using - are you loading a model manually into /models or using something external like Ollama/LMstudio/LocalAI?
There is really nothing in AnythingLLM that would use the GPU unless you are using the "native" llama-cpp LLM, which is going to be deprecated soon because of these kinds of issues with CUDA detection and needing to rebuild the binary for it
Thank you @timothycarambat. I tried both, native and external LLMs. But good to know that if I have an external LLM running, AnythingLLM does not need GPU support...
How are you running AnythingLLM?
Docker (local)
What happened?
Running lastest AnythingLLM on Docker in Linux Ubuntu 20.4 Server
Needed to use Lance_revert image though to prevent crashes
NVIDIA Driver Version: 470.256.02
CUDA Version: 11.4
nvidia-smi gives no errors but no processes are found = AnythingLLM is not using the GPU :-(
I know it´s an outdated GPU but maybe there is a workaround?
Are there known steps to reproduce?
docker pull mintplexlabs/anythingllm:lancedb_revert
export STORAGE_LOCATION=/mnt &&
mkdir -p $STORAGE_LOCATION &&
touch "$STORAGE_LOCATION/.env" &&
docker run -d -p 3001:3001
--cap-add SYS_ADMIN
--gpus all
--name anythingllm
-v ${STORAGE_LOCATION}:/app/server/storage
-v ${STORAGE_LOCATION}/.env:/app/server/.env
-e STORAGE_DIR="/app/server/storage"
mintplexlabs/anythingllm:lancedb_revert
The text was updated successfully, but these errors were encountered: