-
Notifications
You must be signed in to change notification settings - Fork 431
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error during constrained JSON generation. #612
Comments
Thank you for your well written and easy to reproduce issue! Here is the raw output:
Here is the regex pattern which is generated from your schema:
It looks like the Similar to #580 (comment) I recommend using @rlouf I'm wondering if something changed in the sampler that result in these repetition issues? I'm noticing this issue popping up in a few places. @willkurt mentioned he doesn't see this issue with greedy, so maybe the multinomial sampler has trouble generating EOS? Just giving you a heads up that something may have changed. I'm considering writing a logits observation utility to help us with these kinds of issues. |
Not that I know of. A tool to be able to quickly look at the logits would be super useful! |
@lapp0 Thanks for looking into this and the quick response! To be clear, I don't always see the issue with greedy so long as the constraints are set higher. I definitely do see this issue with greedy in some cases, so it's not just a problem with multinomial. Additionally the problem doesn't seem to be with not honoring the constraints, but that the integer there is unbounded. Technically Python integers have no max value so this is technically not incorrect behavior, so certainly undesirable. |
Seems Logits debug log via #616
after the Compared to
A 61% chance of Here are some thoughts on how to ensure quality generations for Outlines users. I can create issues and tackle some of these if you agree @rlouf
|
Describe the issue as clearly as possible:
I'm currently working on attempting to generate JSON using a JSON prompt and this is leading to strange behavior (and most importantly an error) when attempting to generate.
The model seems to be unable to generate valid json within the token limit, even though the Pydantic model I'm using is constrained to only 10 characters in the character field and an int. This issue also happened when I attempted unconstrained generation, which is why I added the constraint.
notes:
max_length=100
and sampler togreedy
I don't get the error, but if I change the sampler tomultinomial
I get it again. Both error atmax_length=10
.greedy
andmultinomial
samplers.mps
but I was also able to reproduce this error on a Linux box with a 4090 using thecuda
/torch.float16
configuration as well.Steps/code to reproduce the bug:
Expected result:
Outlines/Python version information:
Version information
The text was updated successfully, but these errors were encountered: