Skip to content

Commit

Permalink
Bump vLLM version
Browse files Browse the repository at this point in the history
  • Loading branch information
Andrew Lapp authored and rlouf committed Feb 5, 2024
1 parent cb7f747 commit 80c0957
Show file tree
Hide file tree
Showing 3 changed files with 2 additions and 7 deletions.
5 changes: 0 additions & 5 deletions docs/reference/vllm.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,8 @@ Outlines can be deployed as an LLM service using the vLLM inference engine and a

```bash
pip install outlines[serve]
pip install "pydantic>=2.0"
```

!!! Warning

Updating Pydantic to v2 after the installation is necessary.

You can then start the server with:

```bash
Expand Down
2 changes: 1 addition & 1 deletion outlines/serve/vllm.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ def __init__(self, regex_string, llm):
An instance of `vllm.LLM`
"""
tokenizer = self.adapt_tokenizer(llm.tokenizer)
tokenizer = self.adapt_tokenizer(llm.tokenizer.tokenizer)

fsm = RegexFSM(regex_string, tokenizer)
self.fsm = fsm
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ test = [
"huggingface_hub"
]
serve = [
"vllm>=0.2.6",
"vllm>=0.3.0",
"ray==2.9.0",
"uvicorn",
"fastapi"
Expand Down

1 comment on commit 80c0957

@felixzhu555
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can maybe replace llm.tokenizer.tokenizer with llm.get_tokenizer()

Please sign in to comment.