You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There was a noticeable slowdown of doing inference on LLMs. Something like 30-40% less tokens / second.
This change affected both CPU, Cuda and Vulkan backends.
This regression still has not been fixed GPT4All-Chat 3.2.1
Steps to Reproduce
Upgrade from GPT4All-Chat 3.0 to GPT4All-Chat 3.1
Expected Behavior
No slowdown.
Your Environment
GPT4All version: 3.2.1
Operating System: Windows 10
Chat model used (if applicable): For example llama-3-8b-instruct, but happens to all models, e.g. mistral-7b or phi-3-mini-instruct
I was using the known good model, Llama 3 8b instruct. I used all layers on GPU. I used 8k context since that is the limit of the model. I don't like to offload, for me using CPU will usually work better if the model is too big for the GPU.
Bug Report
There was a noticeable slowdown of doing inference on LLMs. Something like 30-40% less tokens / second.
This change affected both CPU, Cuda and Vulkan backends.
This regression still has not been fixed GPT4All-Chat 3.2.1
Steps to Reproduce
Upgrade from GPT4All-Chat 3.0 to GPT4All-Chat 3.1
Expected Behavior
No slowdown.
Your Environment
Hypothesis for root cause
Here is the changelog for version 3.1: https://github.com/nomic-ai/gpt4all/releases/tag/v3.1.0
I highly suspect updating llama.cpp at #2694 introduced this regression, but who knows. Would need to do a git bisect.
The text was updated successfully, but these errors were encountered: