split completion text to sequence_len #616
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
NanoCode012
reviewed
Sep 21, 2023
| @@ -77,7 +82,7 @@ def _tokenize( | |||
| result = self.tokenizer( | |||
| prompt, | |||
| truncation=True, | |||
| max_length=self.sequence_len, | |||
| max_length=self.max_length, | |||
There was a problem hiding this comment.
What is max_length for? Since you're splitting long texts in the prompter, I don't quite understand what this is for.
There was a problem hiding this comment.
the tokenizer no longer emits the entire tokenized string (it used to emit a warning), but now truncates it to max_length.
There was a problem hiding this comment.
without this, we would end up with truncated completion data, and we wouldn't be able to split it later on.
There was a problem hiding this comment.
ah I see. You split after tokenize, not before.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds support for splitting completion text to the sequence_len rather than dropping or truncating.