Feat/add transformers integration

[saattrupdan]

This adds integration with the transformers package, by supplying a prefix_allowed_tokens_fn that can be supplied to both transformers pipelines and directly to generate methods of generative models from the transformers package.

The transformers integration has been put in an integrations top-level directory, and the existing vLLM integration has also been moved into that directory. We keep a reference in the previous serve/vllm.py module, to preserve backwards compatibility.

One notable difference between vLLM and transformers is that transformers include the input in the generated sequences. Since this can mess up the FSMs, we keep track of the input tokens, and use these both to identify when we change to new samples (resetting the FSM) as well as to know which token ID prefix to remove, before getting the next state from the FSM.

Also added an example with the transformers integration.

Closes #713

[rlouf]

Could you rebase your branch on main to remove the merge commits? I haven't had time to test it locally yet, but will get around to it soon. It would be great to also move the integration logic that's in outlines.models.llamacpp.py in outlines.integrations, if you have time to do it.

[saattrupdan]

It would be great to also move the integration logic that's in outlines.models.llamacpp.py in outlines.integrations, if you have time to do it.

Sure thing. Is it correctly understood that this concerns LogitsProcessor, RegexLogitsProcessor and CFGLogitsProcessor in the models.llamacpp module that should be moved to integrations.llamacpp?

Could you rebase your branch on main to remove the merge commits?

Sure thing, I'll do that when the llamacpp integration is set up.

[saattrupdan]

@rlouf The LlamaCpp integration has been moved now, and have also adapted the imports in the models.llamacpp module to point to integrations.llamacpp now. Did a rebase as well. Hopefully it's ready for a merge now!

[rlouf]

Will take a look early next week!

[rlouf]

I just fixed merged conflicts due to a big refactor of the FSM interface. I will take a closer look at the PR tomorrow.

[Kamakshi8104]

Can this be used with multimodal models too?

[rlouf]

Can this be used with multimodal models too?

It should!

[Kamakshi8104]

thanks!! Can't wait to try it out!!

[rlouf]

I was able to run the example locally, and tried the vLLM integration as well; llama.cpp is covered by the integration tests. Great job!

@Kamakshi8104 would you like to add an example using a multimodal model to the docs?

[Kamakshi8104]

Yes I can add an example🙂. I am in the middle of exams so will get started on it this coming week👍

[jeremyallenjacobson]

Yes I can add an example🙂. I am in the middle of exams so will get started on it this coming week👍

Any update on this? Very interested in a seeing a multimodal example, thanks!