Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LAMB Optimizer #1613

Merged
merged 42 commits into from
Oct 11, 2022
Merged

Add LAMB Optimizer #1613

merged 42 commits into from
Oct 11, 2022

Conversation

hanlint
Copy link
Contributor

@hanlint hanlint commented Oct 11, 2022

This PR adds the LAMB Optimizer and makes some minor changes to the Enwiki streaming dataset. cc: @nik-mosaic

nik-mosaic and others added 30 commits September 28, 2022 13:04
@nik-mosaic
Add unpadded bert model
1bad01a
@nik-mosaic
Fix pyright issues
56f78cf
@nik-mosaic
Add bert unpadded model to trainer
e1c616c
update trainer hparams
34846f4
@nik-mosaic
Merge branch 'mosaicml:dev' into nikhil/bert-unpad
a0464f3
Update trainer and model
a84861b
Merge branch 'nikhil/bert-unpad' of github.com:nik-mosaic/composer in…
000f54a 
…to nikhil/bert-unpad
@nik-mosaic
Add bert fusions
e8919bf
Add custom model and ops
518fa99
@nik-mosaic
Update custom BertForMaskedLM
2e15ad4
@nik-mosaic
Update bias in PredictionHead decoder
ed84ee4
@nik-mosaic
add dense_seq_output logic
6b9f4e5
@nik-mosaic
Add apex cross entropy loss
d69023e
Comment out Apex cross entropy loss
691892e
@nik-mosaic
Add masked tokens mask
7e771e0
Remove pooled_output
df4fe5b
@nik-mosaic
Move fusions out of subfolder
b4022a9
@nik-mosaic
Remove model surgery
bb2b526
@nik-mosaic
Merge branch 'mosaicml:dev' into nikhil/bert-unpad
821067a
@nik-mosaic
Add FusedLamb optimizer
9b64b60
@nik-mosaic
add optimizer to registry and test
301f5bc
@nik-mosaic
Use list instead of tuple for betas
1c13e22
@nik-mosaic
Merge branch 'mosaicml:dev' into nikhil/bert-unpad
f85a347
@nik-mosaic
UFix output shape
1404244
@nik-mosaic
Merge branch 'nikhil/bert-unpad' of https://github.com/nik-mosaic/com…
3e67059 
…poser into nikhil/bert-unpad
@nik-mosaic
Try removing unneeeded DatasetHparams
bb1d293
@nik-mosaic
Add back collation
a653a05
@nik-mosaic
Change input name
bbaf807
@nik-mosaic
Replace attention_mask with input_mask
92cafd4
@nik-mosaic
Change token_type_ids to segment ids
33b29d3
@hanlint hanlint requested review from a team and knighton as code owners October 11, 2022 22:29
nik-mosaic
nik-mosaic approved these changes Oct 11, 2022
View reviewed changes
Copy link
Contributor

@nik-mosaic nik-mosaic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks.

@hanlint hanlint enabled auto-merge (squash) October 11, 2022 23:07
@hanlint hanlint disabled auto-merge October 11, 2022 23:07
@hanlint hanlint merged commit b55764a into mosaicml:dev Oct 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mvpatel2000 mvpatel2000 mvpatel2000 left review comments

@nik-mosaic nik-mosaic nik-mosaic approved these changes

@knighton knighton Awaiting requested review from knighton

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@hanlint @mvpatel2000 @nik-mosaic