Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Mamba PR #83

Merged
merged 60 commits into from
Mar 4, 2024
Merged

Add Mamba PR #83

merged 60 commits into from
Mar 4, 2024

Commits on Jan 24, 2024

  1. fix: llama import

    @3outeille
    3outeille committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    abafcb9 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2024

  1. feat: at leading loading mamba properly

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    e8b8314 View commit details
    Browse the repository at this point in the history

  2. loss going down

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    d9513b0 View commit details
    Browse the repository at this point in the history

  3. refacto: init mamba weights

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    147fd25 View commit details
    Browse the repository at this point in the history

  4. fix: dtype for brrr compatibility

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    3e4dda1 View commit details
    Browse the repository at this point in the history

  5. fix: bring back strict union to fix NanotronConfigs + use torch.dtype… 

    … in config
    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    bebbf6f View commit details
    Browse the repository at this point in the history

  6. fix: revert back to casting str to torch dtype

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    ad384b9 View commit details
    Browse the repository at this point in the history

  7. fix

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    02d21de View commit details
    Browse the repository at this point in the history

  8. fix: mismatch params counting due to no tie embedding weight

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    1168bfe View commit details
    Browse the repository at this point in the history

  9. fix: run_generate is now compatible with Brrr

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    7bfc56c View commit details
    Browse the repository at this point in the history

  10. chore: single file mamba slow (for residual fix)

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    1fe1cd4 View commit details
    Browse the repository at this point in the history

  11. fix: residual for slow mamba

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    720df8e View commit details
    Browse the repository at this point in the history

  12. fix(mamba-slow): transpose embedding

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    7902959 View commit details
    Browse the repository at this point in the history

  13. update(mamba-slow)

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    b2ca553 View commit details
    Browse the repository at this point in the history

  14. fix: unified random unit

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    3c4ef42 View commit details
    Browse the repository at this point in the history

  15. feat(optimizer): can now apply weight decay to specifcs parameters

    @3outeille
    3outeille committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    afcbb10 View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2024

  1. Merge branch 'mamba' into mamba-update

    @3outeille
    3outeille committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    7216b11 View commit details
    Browse the repository at this point in the history

  2. refacto: mamba1

    @3outeille
    3outeille committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    a6a163f View commit details
    Browse the repository at this point in the history

  3. feat: add selective scan interface

    @3outeille
    3outeille committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    ad17887 View commit details
    Browse the repository at this point in the history

  4. feat: make fast path compatible with TP

    @3outeille
    3outeille committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    6bfd6aa View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2024

  1. refacto: no split weight

    @3outeille
    3outeille committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    f0e219b View commit details
    Browse the repository at this point in the history

  2. refacto: cleaner init weights

    @3outeille
    3outeille committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    ae312fe View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2024

  1. refacto: init weights

    @3outeille
    3outeille committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    ff26ff5 View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. feat: save/load _no_weight_decay attribute for Mamba model

    @3outeille
    3outeille committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    0ab7c9b View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. clean import

    @3outeille
    3outeille committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    3caf7fe View commit details
    Browse the repository at this point in the history

  2. clean utils in mamba

    @3outeille
    3outeille committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    ec6f954 View commit details
    Browse the repository at this point in the history

  3. cleaning initialization for mamba

    @3outeille
    3outeille committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    1773420 View commit details
    Browse the repository at this point in the history

  4. no sync conv and rmsnorm for mamba

    @3outeille
    3outeille committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    6fc0d9a View commit details
    Browse the repository at this point in the history

  5. Merge branch 'mamba-update' into mamba-pr

    @3outeille
    3outeille committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    6e8fd1f View commit details
    Browse the repository at this point in the history

  6. clean run_train

    @3outeille
    3outeille committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    fa6953d View commit details
    Browse the repository at this point in the history

  7. clean run_train

    @3outeille
    3outeille committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    20dc9ec View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. rename folder

    @3outeille
    3outeille committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    beb73ee View commit details
    Browse the repository at this point in the history

  2. add assert only ALL_REDUCE mode

    @3outeille
    3outeille committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    d6ae604 View commit details
    Browse the repository at this point in the history

  3. add typing to mamba

    @3outeille
    3outeille committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    5d161b1 View commit details
    Browse the repository at this point in the history

  4. remove init method in favor of new initialization method

    @3outeille
    3outeille committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    befb66d View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2024

  1. fix test serialize

    @3outeille
    3outeille committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    35370b7 View commit details
    Browse the repository at this point in the history

  2. small fix

    @3outeille
    3outeille committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    790eddc View commit details
    Browse the repository at this point in the history

  3. add mamba example

    @3outeille
    3outeille committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    37867d1 View commit details
    Browse the repository at this point in the history

  4. fix logger

    @3outeille
    3outeille committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    54a9c63 View commit details
    Browse the repository at this point in the history

  5. update README

    @3outeille
    3outeille committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    3953a5c View commit details
    Browse the repository at this point in the history

  6. update yaml

    @3outeille
    3outeille committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    3fa2194 View commit details
    Browse the repository at this point in the history

  7. move to examples

    @3outeille
    3outeille committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    d4bba7c View commit details
    Browse the repository at this point in the history

  8. update readme

    @3outeille
    3outeille committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    a52c37f View commit details
    Browse the repository at this point in the history

  9. discard run_generate for now

    @3outeille
    3outeille committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    1f54ae2 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. revert dynamic weight decay for now

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    7616ce7 View commit details
    Browse the repository at this point in the history

  2. unifying init_model_randomly for other models

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    e944f63 View commit details
    Browse the repository at this point in the history

  3. decouple mamba logic from core

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    8224221 View commit details
    Browse the repository at this point in the history

  4. fix logging + init_method

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    d28c968 View commit details
    Browse the repository at this point in the history

  5. delete yaml file

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    576eb56 View commit details
    Browse the repository at this point in the history

  6. small fix

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    ea2fa96 View commit details
    Browse the repository at this point in the history

  7. fix tp assert

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    a8448bb View commit details
    Browse the repository at this point in the history

  8. separate config mamba

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    106d59c View commit details
    Browse the repository at this point in the history

  9. update requirements readme

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    db889b9 View commit details
    Browse the repository at this point in the history

  10. various fix

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    f4816ee View commit details
    Browse the repository at this point in the history

  11. Merge branch 'main' into main

    @3outeille
    3outeille authored Mar 4, 2024
    Configuration menu
    Copy the full SHA
    d243b70 View commit details
    Browse the repository at this point in the history

  12. fix import

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    1098e19 View commit details
    Browse the repository at this point in the history

  13. change directory level

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    ae6af18 View commit details
    Browse the repository at this point in the history

  14. decouple Mamba logic from sync weight

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    baba00e View commit details
    Browse the repository at this point in the history

  15. fix decorator

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    7b513e8 View commit details
    Browse the repository at this point in the history

  16. small fixes

    @3outeille
    3outeille committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    dbefcc7 View commit details
    Browse the repository at this point in the history