Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add shifted sparse attention #973

Merged
merged 17 commits into from
Jan 18, 2024
Merged

Add shifted sparse attention #973

merged 17 commits into from
Jan 18, 2024

Commits on Jan 12, 2024

  1. Add s2_attn to hijack flash code

    @joecummings
    joecummings committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    2b2fd52 View commit details
    Browse the repository at this point in the history

  2. Refactor code to account for s2_attn

    @joecummings
    joecummings committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    1450af9 View commit details
    Browse the repository at this point in the history

  3. Add test for models utils

    @joecummings
    joecummings committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    60126bf View commit details
    Browse the repository at this point in the history

  4. Add s2_attention option to llama configs

    @joecummings
    joecummings committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    5e66cb4 View commit details
    Browse the repository at this point in the history

  5. Add s2_attention option to README config

    @joecummings
    joecummings committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    0f57f30 View commit details
    Browse the repository at this point in the history

  6. Format code to appease linter

    @joecummings
    joecummings authored and joecummings committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    cb335d8 View commit details
    Browse the repository at this point in the history

  7. chore: lint

    @winglian @joecummings
    winglian authored and joecummings committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    dcb5694 View commit details
    Browse the repository at this point in the history

  8. Remove xpos and llama-landmark [bad merge]

    @joecummings
    joecummings committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    4135039 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2024

  1. add e2e smoke tests for shifted sparse attention

    @winglian
    winglian committed Jan 14, 2024
    Configuration menu
    Copy the full SHA
    34c62fb View commit details
    Browse the repository at this point in the history

  2. remove stray patch from merge

    @winglian
    winglian committed Jan 14, 2024
    Configuration menu
    Copy the full SHA
    cb899d9 View commit details
    Browse the repository at this point in the history

  3. update yml with link to paper for s2_attention/longlora

    @winglian
    winglian committed Jan 14, 2024
    Configuration menu
    Copy the full SHA
    02d1e90 View commit details
    Browse the repository at this point in the history

  4. fix assertion check for full fine tune

    @winglian
    winglian committed Jan 14, 2024
    Configuration menu
    Copy the full SHA
    e8ba3fe View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2024

  1. increase sequence len for tests and PR feedback updates

    @winglian
    winglian committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    9292665 View commit details
    Browse the repository at this point in the history

  2. reduce context len to 16k for tests

    @winglian
    winglian committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    5e0890d View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2024

  1. reduce context len to 16k for tests

    @winglian
    winglian committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    bee8f8c View commit details
    Browse the repository at this point in the history

  2. reduce batch size for larger context len and udpate test to check mes… 

    …sage
    @winglian
    winglian committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    e6e67dd View commit details
    Browse the repository at this point in the history

  3. fix test for message

    @winglian
    winglian committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    4f09ef4 View commit details
    Browse the repository at this point in the history