Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactoring attention #1182

Merged
merged 16 commits into from
May 8, 2024
Merged

Refactoring attention #1182

merged 16 commits into from
May 8, 2024

Conversation

ShashankMosaicML
Copy link
Contributor

@ShashankMosaicML ShashankMosaicML commented May 8, 2024 •
edited
Loading

The following runs show that these changes do not affect the functionality of training:

  1. Text dataset:
    a. Loss curves:
Screenshot 2024-05-08 at 3 04 49 PM b. mfu: Screenshot 2024-05-08 at 3 05 15 PM c. memory Screenshot 2024-05-08 at 3 05 39 PM
  1. Finetuning hf dataset:
    a. Loss curves:
Screenshot 2024-05-08 at 4 11 09 PM b. mfu: Screenshot 2024-05-08 at 4 11 30 PM c. memory: Screenshot 2024-05-08 at 4 11 54 PM
  1. Finetuning streaming dataset:
    a. Loss curves: Screenshot 2024-05-08 at 4 17 56 PM
    b. mfu:
Screenshot 2024-05-08 at 4 18 24 PM c. memory: Screenshot 2024-05-08 at 4 18 45 PM

@ShashankMosaicML ShashankMosaicML requested review from dakinggg, mvpatel2000 and vchiley and removed request for dakinggg May 8, 2024 01:05
@ShashankMosaicML
adding co-authors
83623b2 
Co-Authored-By: Vitaliy Chiley <vitaliy.chiley@databricks.com>
Co-Authored-By: Cheng Li <cheng.li@databricks.com>
@ShashankMosaicML ShashankMosaicML removed the request for review from vchiley May 8, 2024 01:19
ShashankMosaicML and others added 6 commits May 7, 2024 18:21
@ShashankMosaicML
adding co-authors
ccb76cc 
Co-Authored-By: Vitaliy Chiley <vchiley@me.com>
@ShashankMosaicML
adding co-authors
85d27f8 
Co-authored-by: Vitaliy Chiley <vitaliy.chiley@databricks.com>
Co-authored-by: Vitaliy Chiley <vchiley@me.com>
@ShashankMosaicML
Update config_utils.py
9a251ec 
adding co-authors
Co-authored-by: Vitaliy Chiley <vitaliy.chiley@databricks.com>
Co-authored-by: Vitaliy Chiley <vchiley@me.com>
Co-authored-by: Cheng Li <cheng.li@databricks.com>
Co-authored-by: Cheng Li <@cli99>
@ShashankMosaicML @vchiley
lint
ae3bb9b 
Co-authored-by: Vitaliy Chiley <vitaliy.chiley@databricks.com>
Co-authored-by: Vitaliy Chiley <vchiley@me.com>
@ShashankMosaicML
Merge branch 'refactor_attention' of github.com:ShashankMosaicML/llm-…
76a42ae 
…foundry into refactor_attention
@ShashankMosaicML @vchiley @cli99
Adding_co_authors
8479b96 
Co-authored-by: Vitaliy Chiley <vitaliy.chiley@databricks.com>
Co-authored-by: Vitaliy Chiley <vchiley@me.com>
Co-authored-by: Cheng Li <cheng.li@databricks.com>
@ShashankMosaicML ShashankMosaicML marked this pull request as ready for review May 8, 2024 01:45
dakinggg
dakinggg reviewed May 8, 2024
View reviewed changes
llmfoundry/models/mpt/modeling_mpt.py Outdated Show resolved Hide resolved
llmfoundry/models/layers/blocks.py Outdated Show resolved Hide resolved
ShashankMosaicML and others added 3 commits May 8, 2024 10:03
@ShashankMosaicML @dakinggg
Update llmfoundry/models/mpt/modeling_mpt.py
eeff23d 
Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
@ShashankMosaicML
addressing comments
8ee7409
@ShashankMosaicML @cli99
adding_co_authors
122862d 
Co-authored-by: Cheng Li <pistasable@gmail.com>
@ShashankMosaicML ShashankMosaicML merged commit ac563e6 into mosaicml:main May 8, 2024
9 checks passed
@ShashankMosaicML ShashankMosaicML deleted the refactor_attention branch May 8, 2024 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dakinggg dakinggg dakinggg approved these changes

@mvpatel2000 mvpatel2000 Awaiting requested review from mvpatel2000

@b-chu b-chu Awaiting requested review from b-chu

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@ShashankMosaicML @dakinggg