Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add inference api eval wrapper #494

Merged
merged 75 commits into from
Sep 18, 2023
Merged

Add inference api eval wrapper #494

merged 75 commits into from
Sep 18, 2023

Commits on Jul 14, 2023

  1. add subset num batches

    @bmosaicml
    bmosaicml committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    7e8511b View commit details
    Browse the repository at this point in the history

  2. add subset num batches

    @bmosaicml
    bmosaicml committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    059c43e View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2023

  1. remove tiktoken

    @bmosaicml
    bmosaicml committed Jul 16, 2023
    Configuration menu
    Copy the full SHA
    75c455c View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2023

  1. remove openai import

    @bmosaicml
    bmosaicml committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    f028ad8 View commit details
    Browse the repository at this point in the history

  2. remove bad line

    @bmosaicml
    bmosaicml committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    06fa54a View commit details
    Browse the repository at this point in the history

  3. foo

    @bmosaicml
    bmosaicml committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    3a139b2 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2023

  1. add training callback

    @bmosaicml
    bmosaicml committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    56a2c88 View commit details
    Browse the repository at this point in the history

  2. modify yamls

    @bmosaicml
    bmosaicml committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    e16e86b View commit details
    Browse the repository at this point in the history

  3. implement train

    @bmosaicml
    bmosaicml committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    8341a76 View commit details
    Browse the repository at this point in the history

  4. fix indexing to get most recent eval result

    @bmosaicml
    bmosaicml committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    6ff5cc5 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2023

  1. finish

    @bmosaicml
    bmosaicml committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    06560d5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into enable_gauntlet_training

    @bmosaicml
    bmosaicml authored Aug 2, 2023
    Configuration menu
    Copy the full SHA
    9e07ece View commit details
    Browse the repository at this point in the history

  3. finish

    @bmosaicml
    bmosaicml committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    989f61a View commit details
    Browse the repository at this point in the history

  4. finish

    @bmosaicml
    bmosaicml committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    4c316f1 View commit details
    Browse the repository at this point in the history

  5. finish

    @bmosaicml
    bmosaicml committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    7de1b8c View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2023

  1. finish

    @bmosaicml
    bmosaicml committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    8a77e88 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into enable_gauntlet_training

    @bmosaicml
    bmosaicml authored Aug 9, 2023
    Configuration menu
    Copy the full SHA
    61d682a View commit details
    Browse the repository at this point in the history

  3. foo

    @bmosaicml
    bmosaicml committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    6b2116d View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2023

  1. foo

    @bmosaicml
    bmosaicml committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    33d3165 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2023

  1. working on debugging changeS

    @bmosaicml
    bmosaicml committed Aug 12, 2023
    Configuration menu
    Copy the full SHA
    85c2641 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2023

  1. [wip] removing logger dependency from model gauntlet

    @bmosaicml
    bmosaicml committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    1b3944f View commit details
    Browse the repository at this point in the history

  2. remove logger from eval

    @bmosaicml
    bmosaicml committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    309570d View commit details
    Browse the repository at this point in the history

  3. remove logger from eval

    @bmosaicml
    bmosaicml committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    850bc8e View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2023

  1. remove logger from eval

    @bmosaicml
    bmosaicml committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    82cee97 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into enable_gauntlet_training

    @bmosaicml
    bmosaicml committed Aug 15, 2023
    Configuration menu
    Copy the full SHA
    fe2c141 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2023

  1. debug

    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    df170de View commit details
    Browse the repository at this point in the history

  2. debug

    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    c20ee09 View commit details
    Browse the repository at this point in the history

  3. debug

    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    f23a1ad View commit details
    Browse the repository at this point in the history

  4. debug

    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    7865e83 View commit details
    Browse the repository at this point in the history

  5. fix

    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    96210f0 View commit details
    Browse the repository at this point in the history

  6. finish?

    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    669a770 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into enable_gauntlet_training

    @bmosaicml
    bmosaicml authored Aug 16, 2023
    Configuration menu
    Copy the full SHA
    b269552 View commit details
    Browse the repository at this point in the history

  8. fix bug

    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    6819b43 View commit details
    Browse the repository at this point in the history

  9. merge main

    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    03f80d9 View commit details
    Browse the repository at this point in the history

  10. fix bug

    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    a4f981a View commit details
    Browse the repository at this point in the history

  11. Revert "ignore empty outputs" 

    This reverts commit e0d77bb282c82daa2db686450a551d671d715f27.
    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    d18cef8 View commit details
    Browse the repository at this point in the history

  12. fix pyright

    @bmosaicml
    bmosaicml committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    c882f01 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2023

  1. fix pyright

    @bmosaicml
    bmosaicml committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    9fd4ae1 View commit details
    Browse the repository at this point in the history

  2. update versions

    @bmosaicml
    bmosaicml committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    f174bb6 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2023

  1. merge

    @bmosaicml
    bmosaicml committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    c54509b View commit details
    Browse the repository at this point in the history

  2. fix

    @bmosaicml
    bmosaicml committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    1b80362 View commit details
    Browse the repository at this point in the history

  3. merge updates

    @bmosaicml
    bmosaicml committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    c89c118 View commit details
    Browse the repository at this point in the history

  4. remove info from yamls

    @bmosaicml
    bmosaicml committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    e286dbd View commit details
    Browse the repository at this point in the history

  5. remove load in 8bit

    @bmosaicml
    bmosaicml committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    79ca900 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2023

  1. Merge branch 'main' into add_openai_wrapper

    @bmosaicml
    bmosaicml authored Sep 13, 2023
    Configuration menu
    Copy the full SHA
    f613b3b View commit details
    Browse the repository at this point in the history

  2. address comments

    @bmosaicml
    bmosaicml committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    2157759 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2023

  1. address comments

    @bmosaicml
    bmosaicml committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    4bc1534 View commit details
    Browse the repository at this point in the history

  2. address comments

    @bmosaicml
    bmosaicml committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    b1bc1e3 View commit details
    Browse the repository at this point in the history

  3. add monkeypatch

    @bmosaicml
    bmosaicml committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    984cdbd View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into add_openai_wrapper

    @bmosaicml
    bmosaicml authored Sep 14, 2023
    Configuration menu
    Copy the full SHA
    c7daee0 View commit details
    Browse the repository at this point in the history

  5. add back in bsz

    @bmosaicml
    bmosaicml committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    3108ea5 View commit details
    Browse the repository at this point in the history

  6. add back in bsz

    @bmosaicml
    bmosaicml committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    b5e5e2f View commit details
    Browse the repository at this point in the history

  7. add openai reqs

    @bmosaicml
    bmosaicml committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    0cb29ab View commit details
    Browse the repository at this point in the history

  8. remove branch

    @bmosaicml
    bmosaicml committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    57b684f View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2023

  1. fix conditional import

    @bmosaicml
    bmosaicml committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    29da49d View commit details
    Browse the repository at this point in the history

  2. Update llmfoundry/models/inference_api_wrapper/interface.py 

    Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
    @bmosaicml @dakinggg
    bmosaicml and dakinggg authored Sep 15, 2023
    Configuration menu
    Copy the full SHA
    1a9a77e View commit details
    Browse the repository at this point in the history

  3. Update llmfoundry/models/inference_api_wrapper/openai_causal_lm.py 

    Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
    @bmosaicml @dakinggg
    bmosaicml and dakinggg authored Sep 15, 2023
    Configuration menu
    Copy the full SHA
    2bd933a View commit details
    Browse the repository at this point in the history

  4. Update llmfoundry/models/inference_api_wrapper/interface.py 

    Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
    @bmosaicml @dakinggg
    bmosaicml and dakinggg authored Sep 15, 2023
    Configuration menu
    Copy the full SHA
    77f5279 View commit details
    Browse the repository at this point in the history

  5. fix comments

    @bmosaicml
    bmosaicml committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    359c893 View commit details
    Browse the repository at this point in the history

  6. fix comments

    @bmosaicml
    bmosaicml committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    4a41efd View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into add_openai_wrapper

    @bmosaicml
    bmosaicml authored Sep 15, 2023
    Configuration menu
    Copy the full SHA
    618ec6f View commit details
    Browse the repository at this point in the history

  8. fix comments

    @bmosaicml
    bmosaicml committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    8f155fa View commit details
    Browse the repository at this point in the history

  9. Merge branch 'add_openai_wrapper' of github.com:mosaicml/llm-foundry … 

    …into add_openai_wrapper
    @bmosaicml
    bmosaicml committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    3d4d0da View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2023

  1. Update tests/test_inference_api_eval_wrapper.py

    @dakinggg
    dakinggg authored Sep 16, 2023
    Configuration menu
    Copy the full SHA
    6cff092 View commit details
    Browse the repository at this point in the history

  2. Update tests/test_inference_api_eval_wrapper.py

    @dakinggg
    dakinggg authored Sep 16, 2023
    Configuration menu
    Copy the full SHA
    e7df76f View commit details
    Browse the repository at this point in the history

  3. Update tests/test_inference_api_eval_wrapper.py

    @dakinggg
    dakinggg authored Sep 16, 2023
    Configuration menu
    Copy the full SHA
    dc7cef5 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into add_openai_wrapper

    @dakinggg
    dakinggg authored Sep 16, 2023
    Configuration menu
    Copy the full SHA
    02187d0 View commit details
    Browse the repository at this point in the history

  5. Update tests/test_inference_api_eval_wrapper.py

    @dakinggg
    dakinggg authored Sep 16, 2023
    Configuration menu
    Copy the full SHA
    7638244 View commit details
    Browse the repository at this point in the history

  6. Update tests/test_inference_api_eval_wrapper.py

    @dakinggg
    dakinggg authored Sep 16, 2023
    Configuration menu
    Copy the full SHA
    e11d316 View commit details
    Browse the repository at this point in the history

  7. pyright ignore

    @bmosaicml
    bmosaicml committed Sep 16, 2023
    Configuration menu
    Copy the full SHA
    735f719 View commit details
    Browse the repository at this point in the history

  8. Update tests/test_inference_api_eval_wrapper.py

    @dakinggg
    dakinggg authored Sep 16, 2023
    Configuration menu
    Copy the full SHA
    f2b02dd View commit details
    Browse the repository at this point in the history

  9. Update tests/test_inference_api_eval_wrapper.py

    @dakinggg
    dakinggg authored Sep 16, 2023
    Configuration menu
    Copy the full SHA
    9801af9 View commit details
    Browse the repository at this point in the history

  10. Update tests/test_inference_api_eval_wrapper.py

    @dakinggg
    dakinggg authored Sep 16, 2023
    Configuration menu
    Copy the full SHA
    e809971 View commit details
    Browse the repository at this point in the history

  11. Update tests/test_inference_api_eval_wrapper.py

    @dakinggg
    dakinggg authored Sep 16, 2023
    Configuration menu
    Copy the full SHA
    2bf35e5 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. Merge branch 'main' into add_openai_wrapper

    @bmosaicml
    bmosaicml authored Sep 18, 2023
    Configuration menu
    Copy the full SHA
    eaaebf2 View commit details
    Browse the repository at this point in the history