-
Notifications
You must be signed in to change notification settings - Fork 399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[vLLM integration bug] Generated output is stopped for all samples in batch #757
Comments
This error can also be reproduced with |
@rlouf After extensive debugging, I've identified that this happens here, where we reset the If we thus simply replace the following code block: if len(input_ids) == 0:
self._fsm_state = defaultdict(int)
else:
last_token = input_ids[-1]
last_seq_id = hash(tuple(input_ids[:-1]))
self._fsm_state[seq_id] = self.fsm.get_next_state(
state=self._fsm_state[last_seq_id], token_id=last_token
) with the following: if len(input_ids) > 0:
last_token = input_ids[-1]
last_seq_id = hash(tuple(input_ids[:-1]))
self._fsm_state[seq_id] = self.fsm.get_next_state(
state=self._fsm_state[last_seq_id], token_id=last_token
) then it works just fine in my tests. Is there a reason for the resetting of |
PR open now (#760), feel free to close if you see an issue with not resetting the FSM state. |
Good find, I think this condition was left from a previous version. There is no good reason to reset |
Perfect. The PR is ready when you've got time 🙂 |
Describe the issue as clearly as possible:
When using vLLM with the
JSONLogitsProcessor
, it stops the generation prematurely for the entire batch when the batch is sufficiently large, making the majority of the outputs non-JSON.Steps/code to reproduce the bug:
The above script generates correct JSON output when
num
is less than 129, but the first output is truncated whenevernum
is 130 or above. When we do not include logits processors in the sampling params then it is always correct JSON.Here is the full output:
Expected result:
The output should remain the same for any batch size.
Outlines/Python version information:
Version information
Context for the issue:
This causes all models to get random performance on the NER task in the ScandEval benchmark, which uses
outlines
under the hood for structured generation.The text was updated successfully, but these errors were encountered: