Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bruce/vllm eval #1084

Draft
wants to merge 20 commits into
base: openai_compatible_gauntlet
Choose a base branch
from
Draft

Bruce/vllm eval #1084

wants to merge 20 commits into from

Commits on Mar 21, 2024

  1. use context logprobs

    @bfontain
    bfontain committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    b9d84b5 View commit details
    Browse the repository at this point in the history

  2. first pass

    @bfontain
    bfontain committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    0c4e994 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. fix indent

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    a646ad6 View commit details
    Browse the repository at this point in the history

  2. fix init

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    9836653 View commit details
    Browse the repository at this point in the history

  3. support for generations

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    abb065f View commit details
    Browse the repository at this point in the history

  4. Disable tqdm

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    ed92f31 View commit details
    Browse the repository at this point in the history

  5. turn off metric sync

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    efb3bf4 View commit details
    Browse the repository at this point in the history

  6. turn off autocast

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    f7abd43 View commit details
    Browse the repository at this point in the history

  7. fix device

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    0cabf72 View commit details
    Browse the repository at this point in the history

  8. really fix device

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    259b930 View commit details
    Browse the repository at this point in the history

  9. really really fix device

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    f4b4a56 View commit details
    Browse the repository at this point in the history

  10. fix shapes (tokenizer and model vocab size may not be equal)

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    ec04877 View commit details
    Browse the repository at this point in the history

  11. really fix shapes

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    ba644fb View commit details
    Browse the repository at this point in the history

  12. switch to using tokens to avoid (de)tokenization problems

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    218c169 View commit details
    Browse the repository at this point in the history

  13. swap to nnl_loss with logprobs

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    cb96406 View commit details
    Browse the repository at this point in the history

  14. remove nnl_loss

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    39327ff View commit details
    Browse the repository at this point in the history

  15. final fix

    @bfontain
    bfontain committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    d7875a3 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. fix deps

    @bfontain
    bfontain committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    645b9ab View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. use new faster logits path

    @bfontain
    bfontain committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    e433084 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. make generative evals stable

    @bfontain
    bfontain committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    0fcbd83 View commit details
    Browse the repository at this point in the history