Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactoring tying mechanism + small fixes #62

Merged
merged 9 commits into from
Feb 12, 2024
Merged

Refactoring tying mechanism + small fixes #62

merged 9 commits into from
Feb 12, 2024

Commits on Feb 11, 2024

  1. normalized_model -> unwrapped_model

    @NouamaneTazi
    NouamaneTazi committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    bd82174 View commit details
    Browse the repository at this point in the history

  2. we need to refactor module_id_to_prefix

    @NouamaneTazi
    NouamaneTazi committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    275ddff View commit details
    Browse the repository at this point in the history

  3. fix tied embeddings for llama

    @NouamaneTazi
    NouamaneTazi committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    d8c41d5 View commit details
    Browse the repository at this point in the history

  4. add logging to sync_tied_weights_gradients

    @NouamaneTazi
    NouamaneTazi committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    d442211 View commit details
    Browse the repository at this point in the history

  5. refactor kv weights tying

    @NouamaneTazi
    NouamaneTazi committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    01de8d9 View commit details
    Browse the repository at this point in the history

  6. some pp blocks don't have kv weights

    @NouamaneTazi
    NouamaneTazi committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    da4aa25 View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2024

  1. set log to debug

    @NouamaneTazi
    NouamaneTazi committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    e9ae35e View commit details
    Browse the repository at this point in the history

  2. .

    @NouamaneTazi
    NouamaneTazi committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    5de8a8e View commit details
    Browse the repository at this point in the history

  3. we can allow for non nanotronmodel to be tied as long as name is defi… 

    …ned correctly
    @NouamaneTazi
    NouamaneTazi committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    bae29be View commit details
    Browse the repository at this point in the history