-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dynamic generation with outlines #14
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Start client from langchain.prompts import PromptTemplate
from langchain.prompts.chat import (
ChatPromptTemplate,
SystemMessagePromptTemplate,
AIMessagePromptTemplate,
HumanMessagePromptTemplate,
)
from langchain.schema import HumanMessage, SystemMessage, AIMessage
import json
from langchain.chat_models import ChatOpenAI
llm = ChatOpenAI(temperature=1.0,
openai_api_base="http://localhost:5000/v1",
openai_api_key="Test",
streaming=True,
max_tokens=1024) Test choices messages = [
SystemMessage(
content="You are a helpful assistant."
),
HumanMessage(
content="Who is better bob or fred?"
)
]
for chunk in llm.stream(messages, extra_body={"outlines_type": "choices", "choices": ["bob", "fred"]}):
print(chunk.content, end="", flush=True) Output:
Test JSON from enum import Enum
from pydantic import BaseModel, constr
import json
class Weapon(str, Enum):
sword = "sword"
axe = "axe"
mace = "mace"
spear = "spear"
bow = "bow"
crossbow = "crossbow"
class Armor(str, Enum):
leather = "leather"
chainmail = "chainmail"
plate = "plate"
class Character(BaseModel):
name: constr(max_length=10)
age: int
armor: Armor
weapon: Weapon
strength: int
messages = [
SystemMessage(
content="You are a helpful assistant."
),
HumanMessage(
content=f"Give me an interesting character description based on the following schema: {json.dumps(Character.schema())}"
)
]
for chunk in llm.stream(messages, extra_body={"outlines_type": "json", "json": json.dumps(Character.schema())}):
print(chunk.content, end="", flush=True) Output
Test Regex messages = [
SystemMessage(
content="You are a helpful assistant."
),
HumanMessage(
content=f"Choose between bob and fred."
)
]
for chunk in llm.stream(messages, extra_body={"outlines_type": "regex", "regex": "bob|fred"}):
print(chunk.content, end="", flush=True) Output
Test stop_at keyword
For testing keyboard interrupt, I ran the below code, interrupted during generation, and then ran it again. And it seemed to work messages = [
SystemMessage(
content="You are a helpful assistant."
),
HumanMessage(
content="What is your name?"
)
]
for chunk in llm.stream(messages):
print(chunk.content, end="", flush=True) |
Let me know if there should be more tests! |
…OS compatible with Llama3 and 'stop_at' string from outlines
…_enhance_stop_condition Enhance ExllamaV2Sampler with Temperature Parameter and Update EOS Token for Llama3 Compatibility
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a pr to adapt outlines for exllamav2 dynamic generation. I think this will remove the need for this pr as exllamav2 should do this under the hood.
To run this, you need to install outlines from my branch using
This is currently a PR in outlines too here so in the future
might be enough.
I started server with
and tested code for
I'll add the code for these tests in the comments