Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Mistral] Mistral-7B-v0.1 support #26447

Merged
merged 7 commits into from
Sep 27, 2023
Merged

Conversation

Bam4d
Copy link
Contributor

@Bam4d Bam4d commented Sep 27, 2023

What does this PR do?

Support for Mistral 7B models

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just left 2 nits, thanks a lot for opening the PR and adding this model to transformers!
Regarding the CIs, pip install -e ".[quality" should make sure you have the correct versioning to run make style and make fixup. The hub test is flaky

Comment on lines +25 to +30
Mistral-7B-v0.1 is a decoder-based LM with the following architectural choices:
* Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens
* GQA (Grouped Query Attention) - allowing faster inference and lower cache size.
* Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.

We also provide an instruction fine-tuned model: `Mistral-7B-Instruct-v0.1` which can be used for chat-based inference.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be awesome to have more details about the model directly here like the data it was trained on, but it's a nit!


class MistralAttention(nn.Module):
"""
Multi-headed attention from 'Attention Is All You Need' paper. Modified to use sliding window attention
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last nit, is there a link to a doc/article/blog where the window attention is introduced? 🤗

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done :)

@ArthurZucker
Copy link
Collaborator

Failing test is expected, you can add the configuration_mistral file to the not_doctested.txt file.
The error:

failed on setup with "worker 'gw0' crashed while running 'src/transformers/models/mistral/configuration_mistral.py::transformers.models.mistral.configuration_mistral.MistralConfig'"
worker 'gw0' crashed while running 'src/transformers/models/mistral/configuration_mistral.py::transformers.models.mistral.configuration_mistral.MistralConfig'

is just from trying to init a too big model for the runner! (Llama also had this issue)

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@LysandreJik LysandreJik merged commit 72958fc into huggingface:main Sep 27, 2023
18 of 21 checks passed
blbadger pushed a commit to blbadger/transformers that referenced this pull request Nov 8, 2023
* [Mistral] Mistral-7B-v0.1 support

* fixing names

* slightly longer test

* fixups

* not_doctested

* wrongly formatted references

* make fixuped

---------

Co-authored-by: Timothee Lacroix <t@eugen.ai>
Co-authored-by: timlacroix <t@mistral.ai>
EduardoPach pushed a commit to EduardoPach/transformers that referenced this pull request Nov 18, 2023
* [Mistral] Mistral-7B-v0.1 support

* fixing names

* slightly longer test

* fixups

* not_doctested

* wrongly formatted references

* make fixuped

---------

Co-authored-by: Timothee Lacroix <t@eugen.ai>
Co-authored-by: timlacroix <t@mistral.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants