-
Notifications
You must be signed in to change notification settings - Fork 401
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
syntax error when using Optional[Tuple[int,int]]] #905
Comments
outlines
Library and google/gemma-1.1-2b-it
/solidrust/Hermes-2-Pro-Llama-3-8B-AWQ
Modelsptive title>google/gemma-1.1-2b-it
/solidrust/Hermes-2-Pro-Llama-3-8B-AWQ
Models
google/gemma-1.1-2b-it
/solidrust/Hermes-2-Pro-Llama-3-8B-AWQ
Modelsgoogle/gemma-1.1-2b-it
&solidrust/Hermes-2-Pro-Llama-3-8B-AWQ
Models
I've encountered an interesting and perplexing behavior while working with the ObservationWhen loading the
Steps/code to reproduce the OOM error:from vllm import LLM
from vllm.sampling_params import SamplingParams
from outlines.transformers.vllm import VLLM
import pandas as pd
# Initialize the model
model_name = "solidrust/Hermes-2-Pro-Llama-3-8B-AWQ"
llm = LLM(
model=model_name,
tokenizer=model_name,
trust_remote_code=True,
dtype="auto",
quantization="awq"
)
# Set the sampling parameters for the model
samplimg_params = SamplingParams(max_tokens=2048)
# Wrap the model with VLLM
model = VLLM(llm)
# Prepare the prompts
prompts = [extract_job_description_summary(job['title'], job['description']) for job in df[['title', 'description']].to_dict(orient='records')]
# Generate JSON
generator = generate.json(model, JobDescriptionSummary, whitespace_pattern="[ \n\t]?")
results = generator(prompts, sampling_params=samplimg_params)
# Extract the results into a DataFrame
data = [model.dict() for model in results]
extracted_texts_df = pd.DataFrame(data) result:The OOM error occurs during the model loading phase when loading directly from the Hugging Face Hub, whereas a syntax error occurs when loading the model from disk. |
okay, after a lot of testing the issue boils down to the Tuple contraint in the Basemodel. from datetime import datetime
import json
from enum import Enum
from typing import List,Optional
from typing_extensions import List
from pydantic import BaseModel, constr
import interegular
import outlines.models as models
from outlines.fsm.json_schema import build_regex_from_schema
from outlines.integrations.utils import adapt_tokenizer, convert_json_schema_to_str
import pandas as pd
from pydantic import BaseModel, Field, conlist, constr
from outlines import models, prompt, generate
from typing import Annotated, Tuple, List, Optional
from pydantic import BaseModel, StringConstraints
class JobDescriptionSummary(BaseModel):
salary_range: Optional[Tuple[int]] = Field(
default=None,
description="Salary range for the job, represented as a tuple of (min_salary, max_salary) in integers."
regex_str = build_regex_from_schema(json.dumps(JobDescriptionSummary.model_json_schema()))
regex_pattern = interegular.parse_pattern(regex_str) returns:
|
google/gemma-1.1-2b-it
&solidrust/Hermes-2-Pro-Llama-3-8B-AWQ
Models
Pinging @lapp0 |
Thanks for the great reproduction scripts and isolation of the problem @hugocool! You can verify the fix with
Issue details The issue is that Your json schema:
Smoke test:
Output:
|
Describe the issue as clearly as possible:
When using the
outlines
library to generate JSON structures with thegoogle/gemma-1.1-2b-it
andsolidrust/Hermes-2-Pro-Llama-3-8B-AWQ
models, the JSON output is often invalid, leading to failures in downstream processes. This issue is frequently associated withInvalidSyntax
errors during regex pattern parsing.Steps/code to reproduce the bug:
Initialize the Sampling Parameters and Load the Model:
Note: This code downloads the model to a local temporary directory and loads it. This simulates uploading to S3 and then downloading it, as the end result is the same.
Prepare the Prompts:
Generate JSON:
Expected result:
The model should generate valid JSON objects conforming to the schema defined by
JobDescriptionSummary
.Error message:
The output JSON is often invalid, containing syntax errors that prevent proper parsing and downstream processing.
Error Details
For the
google/gemma-1.1-2b-it
model:For the
solidrust/Hermes-2-Pro-Llama-3-8B-AWQ
model:Outlines/Python version information:
Version information
Context for the issue:
The text was updated successfully, but these errors were encountered: