Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add inference api eval wrapper #494

Merged
merged 75 commits into from
Sep 18, 2023
Merged

Add inference api eval wrapper #494

merged 75 commits into from
Sep 18, 2023

Conversation

bmosaicml
Copy link
Contributor

@bmosaicml bmosaicml commented Jul 31, 2023 •
edited
Loading

This PR creates an inference API wrapper interface for eval. It also provides two concrete instantiations of the interface for both OpenAI chat models and regular OpenAI models. The current version works only for LM tasks. In the future we can support QA tasks and CodeGen. We CANNOT support multiple choice tasks, because the models don't return the full logits.

This interface is also the basis for Nikhil's Mosaic faster transformer inference eval PR #558

Test OpenAI APIs: openai-eval-leLuzi openai-eval-qUWZVI
Test HF models not broken: all-eval-WxT9fe

| model_name               |   average |   world_knowledge |   commonsense_reasoning |   language_understanding |   symbolic_problem_solving |   reading_comprehension |   world_knowledge_lm_task_subscore |   language_understanding_lm_task_subscore |   symbolic_problem_solving_lm_task_subscore |   reading_comprehension_lm_task_subscore |   world_knowledge_lite |   commonsense_reasoning_lite |   language_understanding_lite |   symbolic_problem_solving_lite |   reading_comprehension_lite |
|:-------------------------|----------:|------------------:|------------------------:|-------------------------:|---------------------------:|------------------------:|-----------------------------------:|------------------------------------------:|--------------------------------------------:|-----------------------------------------:|-----------------------:|-----------------------------:|------------------------------:|--------------------------------:|-----------------------------:|
| mosaicml/mpt-7b-instruct |  0.434033 |          0.398764 |                0.415097 |                 0.371509 |                   0.171216 |                0.414691 |                           0.576308 |                                  0.378966 |                                    0.273182 |                                 0.588472 |               0.376383 |                     0.636311 |                      0.692063 |                        0.195028 |                     0.588472 |

| model_name           |   average | world_knowledge   | commonsense_reasoning   | language_understanding   | symbolic_problem_solving   | reading_comprehension   |   world_knowledge_lm_task_subscore |   language_understanding_lm_task_subscore |   symbolic_problem_solving_lm_task_subscore |   reading_comprehension_lm_task_subscore | world_knowledge_lite   | commonsense_reasoning_lite   | language_understanding_lite   | symbolic_problem_solving_lite   |   reading_comprehension_lite |
|:---------------------|----------:|:------------------|:------------------------|:-------------------------|:---------------------------|:------------------------|-----------------------------------:|------------------------------------------:|--------------------------------------------:|-----------------------------------------:|:-----------------------|:-----------------------------|:------------------------------|:--------------------------------|-----------------------------:|
| openai/gpt-4         |  0.683983 |                   |                         |                          |                            |                         |                           0.656461 |                                  0.577393 |                                    0.843193 |                                 0.671434 |                        |                              |                               |                                 |                     0.671434 |
| openai/gpt-3.5-turbo |  0.529258 |                   |                         |                          |                            |                         |                           0.630127 |                                  0.456013 |                                    0.701307 |                                 0.429421 |                        |                              |                               |                                 |                     0.429421 |
| openai/davinci       |  0.475937 |                   |                         |                          |                            |                         |                           0.625064 |                                  0.439049 |                                    0.276832 |                                 0.519369 |                        |                              |                               |                                 |                     0.519369 |
| openai/ada           |  0.281576 |                   |                         |                          |                            |                         |                           0.283446 |                                  0.272376 |                                    0.156899 |                                 0.347579 |                        |                              |                               |                                 |                     0.347579 |

bmosaicml and others added 30 commits July 14, 2023 16:41
@bmosaicml
add subset num batches
7e8511b
@bmosaicml
add subset num batches
059c43e
@bmosaicml
remove tiktoken
75c455c
@bmosaicml
remove openai import
f028ad8
@bmosaicml
remove bad line
06fa54a
@bmosaicml
foo
3a139b2
@bmosaicml
add training callback
56a2c88
@bmosaicml
modify yamls
e16e86b
@bmosaicml
implement train
8341a76
@bmosaicml
fix indexing to get most recent eval result
6ff5cc5
@bmosaicml
finish
06560d5
@bmosaicml
Merge branch 'main' into enable_gauntlet_training
9e07ece
@bmosaicml
finish
989f61a
@bmosaicml
finish
4c316f1
@bmosaicml
finish
7de1b8c
@bmosaicml
finish
8a77e88
@bmosaicml
Merge branch 'main' into enable_gauntlet_training
61d682a
@bmosaicml
foo
6b2116d
@bmosaicml
foo
33d3165
@bmosaicml
working on debugging changeS
85c2641
@bmosaicml
[wip] removing logger dependency from model gauntlet
1b3944f
@bmosaicml
remove logger from eval
309570d
@bmosaicml
remove logger from eval
850bc8e
@bmosaicml
remove logger from eval
82cee97
@bmosaicml
Merge branch 'main' into enable_gauntlet_training
fe2c141
@bmosaicml
debug
df170de
@bmosaicml
debug
c20ee09
@bmosaicml
debug
f23a1ad
@bmosaicml
debug
7865e83
@bmosaicml
fix
96210f0
dakinggg
dakinggg reviewed Sep 15, 2023
View reviewed changes
llmfoundry/models/inference_api_wrapper/openai_causal_lm.py Outdated Show resolved Hide resolved
llmfoundry/models/inference_api_wrapper/openai_causal_lm.py Outdated Show resolved Hide resolved
llmfoundry/models/inference_api_wrapper/openai_causal_lm.py Show resolved Hide resolved
llmfoundry/models/inference_api_wrapper/openai_causal_lm.py Outdated Show resolved Hide resolved
llmfoundry/models/inference_api_wrapper/openai_causal_lm.py Outdated Show resolved Hide resolved
llmfoundry/models/inference_api_wrapper/openai_causal_lm.py Outdated Show resolved Hide resolved
tests/test_inference_api_eval_wrapper.py Outdated Show resolved Hide resolved
llmfoundry/models/inference_api_wrapper/interface.py Outdated Show resolved Hide resolved
llmfoundry/models/inference_api_wrapper/interface.py Outdated Show resolved Hide resolved
llmfoundry/models/inference_api_wrapper/interface.py Outdated Show resolved Hide resolved
bmosaicml and others added 8 commits September 15, 2023 15:19
@bmosaicml @dakinggg
Update llmfoundry/models/inference_api_wrapper/interface.py
1a9a77e 
Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
@bmosaicml @dakinggg
Update llmfoundry/models/inference_api_wrapper/openai_causal_lm.py
2bd933a 
Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
@bmosaicml @dakinggg
Update llmfoundry/models/inference_api_wrapper/interface.py
77f5279 
Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
@bmosaicml
fix comments
359c893
@bmosaicml
fix comments
4a41efd
@bmosaicml
Merge branch 'main' into add_openai_wrapper
618ec6f
@bmosaicml
fix comments
8f155fa
@bmosaicml
Merge branch 'add_openai_wrapper' of github.com:mosaicml/llm-foundry …
3d4d0da 
…into add_openai_wrapper
dakinggg
dakinggg approved these changes Sep 16, 2023
View reviewed changes
tests/test_inference_api_eval_wrapper.py Show resolved Hide resolved
tests/test_inference_api_eval_wrapper.py Show resolved Hide resolved
tests/test_inference_api_eval_wrapper.py Show resolved Hide resolved
@bmosaicml bmosaicml merged commit c369a68 into main Sep 18, 2023
7 of 8 checks passed
@dakinggg dakinggg deleted the add_openai_wrapper branch October 11, 2023 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dakinggg dakinggg dakinggg approved these changes

@abhi-mosaic abhi-mosaic Awaiting requested review from abhi-mosaic

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@bmosaicml @dakinggg