-
Notifications
You must be signed in to change notification settings - Fork 26.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate: add min_p
sampling
#30639
Generate: add min_p
sampling
#30639
Conversation
def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor: | ||
# Convert logits to probabilities | ||
probs = torch.softmax(scores, dim=-1) | ||
# Get the probability of the top token for each sequence in the batch | ||
top_probs, _ = probs.max(dim=-1, keepdim=True) | ||
# Calculate the actual min_p threshold by scaling min_p with the top token's probability | ||
scaled_min_p = self.min_p * top_probs | ||
# Create a mask for tokens that have a probability less than the scaled min_p | ||
tokens_to_remove = probs < scaled_min_p | ||
|
||
sorted_indices = torch.argsort(scores, descending=True, dim=-1) | ||
sorted_indices_to_remove = torch.gather(tokens_to_remove, dim=-1, index=sorted_indices) | ||
# Keep at least min_tokens_to_keep | ||
sorted_indices_to_remove[..., : self.min_tokens_to_keep] = False | ||
|
||
indices_to_remove = sorted_indices_to_remove.scatter(1, sorted_indices, sorted_indices_to_remove) | ||
scores_processed = scores.masked_fill(indices_to_remove, self.filter_value) | ||
return scores_processed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
copy/paste from the implementation authors are using as reference: https://github.com/oobabooga/text-generation-webui/pull/4449/files#diff-51532f26c3cdbf11129835e199b8f4f621d2ec5968ce71760d149e251ff51526
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems mostly fine, since it references the oogabooga core implementation + changes other relevant HF transformers files.
Comments mainly about min_p values are the opposite of comparable top_p values (min_p = 0.1 is like top_p = 0.9), which may lead to some issues/confusion since a lot of users are used to top_p values. Also optional is to mention that min_p = 0.05 - 0.1 is what most people use, but maybe that's too specific so heh.
That said, the main functionality is there, so this seems good to go atm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm unfamiliar with what this does at the moment 💀 but I would caution that since min_p values are the opposite of comparable top_p values (min_p = 0.1 is comparably selective as top_p = 0.9), copying the same values might cause issues.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No copy here, I wrote the tests with values that make sense to be filtered/not filtered according to the min_p
sampling strategy. If you look at the first test case, it's actually the example provided in the original issue :D
Co-authored-by: menhguin <minh1228@gmail.com>
Co-authored-by: menhguin <minh1228@gmail.com>
@menhguin the "it's the opposite of top_p" is a good rule of thumb! 👍 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! Thanks for adding
|
||
def __init__(self, min_p: float, filter_value: float = -float("Inf"), min_tokens_to_keep: int = 1): | ||
if min_p < 0 or min_p > 1.0: | ||
raise ValueError(f"`min_p` has to be a float >= 0 and <= 1, but is {min_p}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Either bounds in the comment or bounds in the check should be updated to be consistent
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The bounds are consistent :D Both mean that we accept [0, 1]
I'll convert the condition to not (0 <= min_p <= 1.0)
, which should be more readable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
d'oh 🤦♀️
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* min_p * more relaxed test to avoid numerical issues * Update src/transformers/generation/logits_process.py Co-authored-by: menhguin <minh1228@gmail.com> * Update src/transformers/generation/configuration_utils.py Co-authored-by: menhguin <minh1228@gmail.com> * docstring clarifications * PR comments * Update tests/generation/test_logits_process.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * make fixup --------- Co-authored-by: menhguin <minh1228@gmail.com> Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* min_p * more relaxed test to avoid numerical issues * Update src/transformers/generation/logits_process.py Co-authored-by: menhguin <minh1228@gmail.com> * Update src/transformers/generation/configuration_utils.py Co-authored-by: menhguin <minh1228@gmail.com> * docstring clarifications * PR comments * Update tests/generation/test_logits_process.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * make fixup --------- Co-authored-by: menhguin <minh1228@gmail.com> Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
What does this PR do?
Adds
min_p
, a dynamic sampling strategy. Instead of picking a fixed number of tokens (top_k
) or a fixed cumulative probability (top_p
), it sets a minimum probability filter that scales with the most likely token in that round:Fixes #27670, opened by the authors of the technique. Opening the PR now because we now have empirical checks on the technique (see the original issue), in addition to the vibe checks.
Cool example:
with
min_p
removing
min_p
(doesn't follow the instruction as closely)