New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compile GLU #1049

Merged

josejg merged 1 commit into mosaicml:main from josejg:compiled_glu

Mar 22, 2024

Contributor

josejg commented Mar 21, 2024

This PR adds torch.compile to the MPTGLU layer forward.

Under a LLaMa-2-7b-like workload (displayed below) adding torch.compile produces an additional ~100 tok/s/gpu with no changes in the loss


          Compile GLU

a633fa7

dakinggg approved these changes

View reviewed changes

josejg merged commit 26a5fd4 into mosaicml:main

10 checks passed

jjanezhang pushed a commit that referenced this pull request


          Compile GLU (#1049)

7ae473b

KuuCi pushed a commit that referenced this pull request


          Compile GLU (#1049)

86053af

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment