-
-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support llama-3 #789
Comments
h i tried llama-3 and may be you can use the setup. first add template for llama3 in file.
then add option for choosing the llama3 in localGPT
you can run now with here is the model i used for tesitng.
|
@toomy0toons did you upgrade the llama cpp or transformers version to make this work with llama-3? |
I did install i have cuda GPU so i installed the cublas version.
I did not install anything or upgrade anything besides official insturctions. It works out of the box. but since |
I think since llama2 is probably not going to be used anymore, I will update the prompt template for llama3 as default template. |
@toomy0toons I tried with another version( QuantFactory/Meta-Llama-3-8B-GGUF) and it did't work. |
hi i have downloaded llama3 70b model . can some one provide me steps to convert into hugging face model and then run in the localGPT as currently i have done the same for llama 70b i am able to perform but i am not able to convert the full model files to .hf format files. so i would request for an proper steps in how i can perform. please let me know guys any steps please let me know. thank you |
Hi @toomy0toons , trying to do the same but having some issues as per this #793 |
my understanding is that the instruct model (8b) has extra set of tokens or has diffenrent prompt template. try 7b models? |
No 7B models for llama3 (https://adithyask.medium.com/from-7b-to-8b-parameters-understanding-weight-matrix-changes-in-llama-transformer-models-31ea7ed5fd88) Do you mean none of the embedding models in constants.py are ok to run any of the llama-3 8b models? |
@toomy0toons found out the answer here https://youtu.be/S6PdFPoteBU?si=pSsxCNFJsz_dxn8b&t=551 |
@PromtEngineer For info, the video you posted: https://www.youtube.com/watch?v=S6PdFPoteBU&t=549s that mentioned
Let's resolve the mystery here. The reason was caused by the underlying library of llama-cpp (EOS for instruct models). Some of the related discussion can be seen here: ggerganov/llama.cpp#6745 (comment) and there's some latter changes to support llama3 EOS e.g. https://github.com/ggerganov/llama.cpp/pull/6751/files In particular, this commit from llama.cpp is what we are looking for: ggerganov/llama.cpp@7370d66 It was released on April 21: https://github.com/ggerganov/llama.cpp/tree/b2707 but for llama-cpp-python, we need to find the version at least greater than the patch for llama.cpp which is this one:
We can see the patch llama-cpp-python was included for version greater than 0.2.62 (can be verified using In summary, we only need to be sure the llama-cpp-python version is greater than 0.2.62 to run the quantized llama3 model. |
Created associated pull request to strength the docs: #823. No particular code changes needed from our end in this case but we need to clarify the llama-cpp-python version to support llama3. |
Fix #789: Update README with instructions for running the quantized L…
I wanna deploy the application how to do it so I have the infrastructure
but deployment of llm and multiple user to access provide me the steps to
do it
…On Sat, 21 Sep 2024 at 4:56 AM, PromptEngineer ***@***.***> wrote:
Closed #789 <#789> as
completed via b4322d4
<b4322d4>
.
—
Reply to this email directly, view it on GitHub
<#789 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APQASI6XXJFNJ66JMD5Y7DDZXSVIVAVCNFSM6AAAAABGS6M2AGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGM2DMNZRG43DENI>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Hi
Please add support for llama-3
Currently the prompt template is not compatible since llama-3 uses different style.
Ref: https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3
Currently as is I was unable to use the llama-3 model.
Thanks in advance!
The text was updated successfully, but these errors were encountered: