Add LlamaCpp integration

[dtiarks]

fixes #422

This is a draft PR for a LlamaCpp integration using the low-level API.

[rlouf]

Great work! I still need to review the PR, but one remark for now: can we name the model llamacpp instead of llama to avoid confusion?

[dtiarks]

Sure!

[dtiarks]

Is there any guidance on how to handle artifacts (the GGUF model used in the test) in the CI/CD? Just check it in using LFS?

[rlouf]

So far we've been downloading models from the hub each time the tests are run. Since they're tiny models it doesn't affect the runtime much, but it would be better to cache them for sure.

[dtiarks]

Ok. The llama test model is around 2meg so it should be fine.
In order to have the llama tests running in CI/CD I added llama-cpp-python as test dependency. Question is if we should at it as project dependency as well.

Another open question is what the current intention behind the outer dimension of the tokenizer encode method is. Is it supposed to be a batch dimension? If so, I have to implement a padding meachnism, because the llama cpp tokenizer doesn't support that out of the box.

[rlouf]

We won't add it as a dependency by default but will document it in the installation section of the documentation.

And yes token_ids tensors are of shape n_batch x n_tokens and we keep the outer dimension throughout even for batch size of 1. Does llamacpp handle batch inference?

[dtiarks]

It does have a batch-based API for inference (which is also recommended) but not for tokenization. This creates a little bit of overhead within the outlines sided implementation of the tokenizer, but this should be ok.

I will incorporate the batch API in the coming days...

[rlouf]

@dtiarks is this ready for review?

[dtiarks]

Not yet. I'm still trying to figure out why I get so many whitespaces in the output JSON and if it has something to do with my implementation.

Will ping you once I have news.

[rlouf]

It might not be your implementation: #484 (comment)

[dtiarks]

Ok. I have a few smaller issues on my list. But those should be done today or tomorrow.

[rlouf]

Thank you so much for this addition! I'm sure it will be greatly appreciated by the community :)