Add Mamba PR #83

3outeille · 2024-02-28T16:54:10Z

No description provided.

… in config

src/nanotron/trainer.py

examples/mamba/mamba/config_mamba.py

examples/mamba/mamba/trainer.py

src/nanotron/optim/named_optimizer.py

examples/mamba/README.md

src/nanotron/trainer.py

xrsrke

LGTM. some last changes before merging

examples/mamba/mamba/config.py

examples/mamba/README.md

examples/mamba/mamba/trainer.py

examples/mamba/mamba.py

src/nanotron/helpers.py

src/nanotron/trainer.py

examples/mamba/mamba.py

NouamaneTazi

LGTM! Very cleaaaan 🐍🚀

3outeille · 2024-03-04T16:51:37Z

thank you @NouamaneTazi and @xrsrke for taking the time to review the PR ❤️

3outeille added 30 commits January 24, 2024 10:22

fix: llama import

abafcb9

feat: at leading loading mamba properly

e8b8314

loss going down

d9513b0

refacto: init mamba weights

147fd25

fix: dtype for brrr compatibility

3e4dda1

fix: bring back strict union to fix NanotronConfigs + use torch.dtype…

bebbf6f

… in config

fix: revert back to casting str to torch dtype

ad384b9

fix

02d21de

fix: mismatch params counting due to no tie embedding weight

1168bfe

fix: run_generate is now compatible with Brrr

7bfc56c

chore: single file mamba slow (for residual fix)

1fe1cd4

fix: residual for slow mamba

720df8e

fix(mamba-slow): transpose embedding

7902959

update(mamba-slow)

b2ca553

fix: unified random unit

3c4ef42

feat(optimizer): can now apply weight decay to specifcs parameters

afcbb10

Merge branch 'mamba' into mamba-update

7216b11

refacto: mamba1

a6a163f

feat: add selective scan interface

ad17887

feat: make fast path compatible with TP

6bfd6aa

refacto: no split weight

f0e219b

refacto: cleaner init weights

ae312fe

refacto: init weights

ff26ff5

feat: save/load _no_weight_decay attribute for Mamba model

0ab7c9b

clean import

3caf7fe

clean utils in mamba

ec6f954

cleaning initialization for mamba

1773420

no sync conv and rmsnorm for mamba

6fc0d9a

Merge branch 'mamba-update' into mamba-pr

6e8fd1f

clean run_train

fa6953d

3outeille force-pushed the main branch from 74efde5 to fb4b996 Compare March 1, 2024 19:12

xrsrke requested changes Mar 3, 2024

View reviewed changes

NouamaneTazi reviewed Mar 4, 2024

View reviewed changes

src/nanotron/trainer.py Outdated Show resolved Hide resolved

src/nanotron/trainer.py Show resolved Hide resolved

3outeille added 5 commits March 4, 2024 10:26

revert dynamic weight decay for now

7616ce7

unifying init_model_randomly for other models

e944f63

decouple mamba logic from core

8224221

fix logging + init_method

d28c968

delete yaml file

576eb56

3outeille force-pushed the main branch from 7caf66c to 576eb56 Compare March 4, 2024 10:27

3outeille added 2 commits March 4, 2024 10:37

small fix

ea2fa96

fix tp assert

a8448bb

3outeille marked this pull request as ready for review March 4, 2024 12:52

separate config mamba

106d59c

3outeille force-pushed the main branch from 0a4d0c3 to 106d59c Compare March 4, 2024 13:17

xrsrke requested changes Mar 4, 2024

View reviewed changes

examples/mamba/mamba/config.py Outdated Show resolved Hide resolved

examples/mamba/README.md Show resolved Hide resolved

examples/mamba/mamba/trainer.py Outdated Show resolved Hide resolved

examples/mamba/mamba/trainer.py Outdated Show resolved Hide resolved

3outeille and others added 5 commits March 4, 2024 14:22

update requirements readme

db889b9

various fix

f4816ee

Merge branch 'main' into main

d243b70

fix import

1098e19

change directory level

ae6af18

NouamaneTazi reviewed Mar 4, 2024

View reviewed changes

examples/mamba/mamba.py Outdated Show resolved Hide resolved

3outeille added 2 commits March 4, 2024 16:33

decouple Mamba logic from sync weight

baba00e

fix decorator

7b513e8

NouamaneTazi reviewed Mar 4, 2024

View reviewed changes

src/nanotron/helpers.py Show resolved Hide resolved

NouamaneTazi reviewed Mar 4, 2024

View reviewed changes

src/nanotron/trainer.py Outdated Show resolved Hide resolved

NouamaneTazi reviewed Mar 4, 2024

View reviewed changes

examples/mamba/mamba.py Show resolved Hide resolved

NouamaneTazi approved these changes Mar 4, 2024

View reviewed changes

small fixes

dbefcc7

NouamaneTazi merged commit 5e128fc into huggingface:main Mar 4, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Mamba PR #83

Add Mamba PR #83

3outeille commented Feb 28, 2024

xrsrke left a comment

NouamaneTazi left a comment

3outeille commented Mar 4, 2024

Add Mamba PR #83

Add Mamba PR #83

Conversation

3outeille commented Feb 28, 2024

xrsrke left a comment

Choose a reason for hiding this comment

NouamaneTazi left a comment

Choose a reason for hiding this comment

3outeille commented Mar 4, 2024