-
Notifications
You must be signed in to change notification settings - Fork 399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
update vllm patch to v0.2.6 and add regex support for /generate
#481
Conversation
problem with `nest_asyncio` and uvloop
|
some success:
At least it doesn't crash anymore. |
/generate
everything works |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we default to an empty list of logits processors in case neither a schema nor a regex string are provided?
We also need to update the documentation to reflect the changes before merging.
Besides this everything looks good.
@rlouf thanks for the review! I made some changes based on your feedback |
Great contribution, thank you! |
closes #478
outlines.serve
to vllm-v0.2.6 and ray-2.9.0nest_asyncio
withuvloop.Loop
(see Can't patch loop of type <class 'uvloop.Loop'> erdewit/nest_asyncio#49)/generate
outlines.serve.vllm
codeHere's a
Dockerfile
that I used for testing:and the accompanying
Makefile
: