Accelerate not installed in Docker Image #1106
Labels
Comments
|
Thank you, happy to review a PR! |
|
Additionally, |
|
@rlouf I would be happy to create a PR for the docker setup, first I want to fully answer @lapp0's question, as this might be important for why I would like to use accelerate. I would prefer to use vLLM. I created a simple pydantic use case for vllm, transformers and serve, this is what the code was for each, and what was output. Since running these, I added params to the vLLM example and it started returning valid json, but I was expecting it to work out of the box as vLLM has default parameters + outlines should be restricting to correct schema(?) Server Call Code:import requests
from pydantic import BaseModel
# Define the Book model
class Book(BaseModel):
title: str
author: str
year: int
# Define the request parameters
ip_address = "localhost"
port = "8000"
prompt = "Create a book entry with the fields title, author, and year"
schema = Book.model_json_schema()
# Create the request body
outlines_request = {
"prompt": prompt,
"schema": schema
}
print("Prompt: ", prompt)
# Make the API call
response = requests.post(f"http://{ip_address}:{port}/generate/", json=outlines_request)
# Check if the request was successful
if response.status_code == 200:
result = response.json()
print("Result:", result["text"])
else:
print(f"Error: {response.status_code}, {response.text}")Server command: Output: VLLM Code:from outlines import models, generate
from pydantic import BaseModel
from vllm import SamplingParams
class Book(BaseModel):
title: str
author: str
year: int
print("\n\npydantic_vllm_example\n\n")
model = models.vllm("microsoft/Phi-3-mini-128k-instruct", max_model_len= 25000)
params = SamplingParams(temperature=0, top_k=-1)
generator = generate.json(model, Book)
prompt = "Create a book entry with the fields title, author, and year"
result = generator(prompt)
print("Prompt:",prompt)
print("Result:",result)Output: Transformers Code:from outlines import models, generate
from pydantic import BaseModel
from outlines.samplers import greedy
class Book(BaseModel):
title: str
author: str
year: int
model = models.transformers("microsoft/Phi-3-mini-128k-instruct", device="cuda:0")
print("\n\npydantic_transformers_example\n\n")
generator = generate.json(model, Book)
prompt = "Create a book entry with the fields title, author, and year"
result = generator(prompt)
print("Prompt:",prompt)
print("Result:",result)Output: |
|
I'll look into the bug with json handling in vLLM. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the issue as clearly as possible:
Not sure if this is a bug or a feature request. But
accelerateapparently isn't installed in the docker image. Which means that one can either use transformers with no GPU acceleration, or vLLM. vLLM currently doesn't have feature parity with transformers from what I can tell (likegenerate.json()).Running the code outside of the image with the library +
accelerateworks.pip install acceleratein the container works to solve the issue as well, and it seems like the marginal download was very small.Steps/code to reproduce the bug:
Expected result:
Error message:
Outlines/Python version information:
Docker Image Version Hash:
98c8512bd46fVersion information
Context for the issue:
I'm trying to write a post on using Outlines with Vast, and Vast needs everything to be based in a docker container to run, it would be great if users could start their workloads in the container without needing to install accelerate first.
The text was updated successfully, but these errors were encountered: