-
Notifications
You must be signed in to change notification settings - Fork 26.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[LLaVa] Some improvements #27895
[LLaVa] Some improvements #27895
Conversation
@@ -270,22 +270,22 @@ def resize_token_embeddings(self, new_num_tokens: Optional[int] = None, pad_to_m | |||
def _merge_input_ids_with_image_features( | |||
self, image_features, inputs_embeds, input_ids, attention_mask, position_ids | |||
): | |||
nb_images, image_hidden_dim, embed_dim = image_features.shape | |||
num_images, num_image_patches, embed_dim = image_features.shape |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is not really a "hidden dim" but rather the number of image patches. As below "num_image_tokens" is already used, this could be confusing with "num_image_patches". The former actually refers to the number of special image tokens, hence I've renamed that.
This model was contributed by [ArthurZ](https://huggingface.co/ArthurZ) and [ybelkada](https://huggingface.co/ybelkada). | ||
The original code can be found [here](https://github.com/haotian-liu/LLaVA/tree/main/llava). | ||
|
||
## Usage tips |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @younesbelkada just an FYI, for new docs we always have a ## Usage tips
section and a ## Resources
section
I'm not sure the CookieCutter creates those sections by default
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice cleanup
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the cleanup, can you confirm the slow tests pass with these changes?
They are failing on my setup:
Could possibly also be the case on main, will check |
Yeah it still fails for me both on main and my branch even after #27909. So @ArthurZucker could you perhaps run the slow tests from my branch before merging? But I'm pretty sure it doesn't affect integration tests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! can confirm this PR does not affect slow tests
* More improvements * Improve variable names * Update READMEs, improve docs
* More improvements * Improve variable names * Update READMEs, improve docs
What does this PR do?
Some minor improvements when going over the LLaVa code.