-
Notifications
You must be signed in to change notification settings - Fork 399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add LlamaCpp integration #486
Conversation
Great work! I still need to review the PR, but one remark for now: can we name the model |
Sure! |
Is there any guidance on how to handle artifacts (the GGUF model used in the test) in the CI/CD? Just check it in using LFS? |
So far we've been downloading models from the hub each time the tests are run. Since they're tiny models it doesn't affect the runtime much, but it would be better to cache them for sure. |
Ok. The llama test model is around 2meg so it should be fine. Another open question is what the current intention behind the outer dimension of the tokenizer encode method is. Is it supposed to be a batch dimension? If so, I have to implement a padding meachnism, because the llama cpp tokenizer doesn't support that out of the box. |
We won't add it as a dependency by default but will document it in the And yes |
It does have a batch-based API for inference (which is also recommended) but not for tokenization. This creates a little bit of overhead within the outlines sided implementation of the tokenizer, but this should be ok. I will incorporate the batch API in the coming days... |
66de1fe
to
7e2799e
Compare
@dtiarks is this ready for review? |
Not yet. I'm still trying to figure out why I get so many whitespaces in the output JSON and if it has something to do with my implementation. Will ping you once I have news. |
It might not be your implementation: #484 (comment) |
Ok. I have a few smaller issues on my list. But those should be done today or tomorrow. |
01bfac9
to
824cc48
Compare
Thank you so much for this addition! I'm sure it will be greatly appreciated by the community :) |
fixes #422
This is a draft PR for a LlamaCpp integration using the low-level API.