-
Notifications
You must be signed in to change notification settings - Fork 26.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate: handle text conditioning with multimodal encoder-decoder models #22748
Conversation
cc @younesbelkada @NielsRogge FYI -- this PR consolidates your recent changes regarding text conditioning on multimodal models. The next models should be easier to add :) |
The documentation is not available anymore as the PR was closed or merged. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this!
Thanks a lot @gante! 🙏 |
What does this PR do?
Consolidates
decoder_input_ids
preparation changes in a single place, for all future multimodal encoder-decoder models on PT and TF.In a nutshell, this PR generalizes the following use cases:
decoder_input_ids
, but it is missing the BOS token (some tokenizers, like the T5 tokenizer, do not prepend a BOS token). In this case, a BOS token is prepended.input_ids
, but the encoder has noinput_ids
input. In this case,input_ids
is handled just likedecoder_input_ids
.Slow tests were run on T5, Pix2Struct, BLIP, and BLIP2.