Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add benchmarks to inference README #393

Merged
merged 28 commits into from
Jun 29, 2023
Merged

Add benchmarks to inference README #393

merged 28 commits into from
Jun 29, 2023

Conversation

sashaDoubov
Copy link
Contributor

Include benchmark info

@sashaDoubov sashaDoubov changed the title Inference readme Add benchmarks to inference README Jun 29, 2023
sashaDoubov and others added 2 commits June 29, 2023 11:08
@sashaDoubov
pass linter
b35921d
@sashaDoubov @vchiley
Update scripts/inference/benchmarking/README.md
7d9202a 
Co-authored-by: Vitaliy Chiley <6439018+vchiley@users.noreply.github.com>
sashaDoubov and others added 8 commits June 29, 2023 11:08
@sashaDoubov @vchiley
Update scripts/inference/benchmarking/README.md
00315a5 
Co-authored-by: Vitaliy Chiley <6439018+vchiley@users.noreply.github.com>
@sashaDoubov @vchiley
Update scripts/inference/benchmarking/README.md
cab755c 
Co-authored-by: Vitaliy Chiley <6439018+vchiley@users.noreply.github.com>
@sashaDoubov
Merge branch 'inference_readme' of github.com:sashaDoubov/llm-foundry…
1a767de 
… into inference_readme
@sashaDoubov
address review
de1fc7c
@sashaDoubov @vchiley
Update scripts/inference/benchmarking/README.md
5ffaa4a 
Co-authored-by: Vitaliy Chiley <6439018+vchiley@users.noreply.github.com>
@sashaDoubov @vchiley
Update scripts/inference/benchmarking/README.md
21e7b47 
Co-authored-by: Vitaliy Chiley <6439018+vchiley@users.noreply.github.com>
@sashaDoubov
falcon add
0b23e93
@sashaDoubov
Merge branch 'inference_readme' of github.com:sashaDoubov/llm-foundry…
3d8db3b 
… into inference_readme
abhi-mosaic
abhi-mosaic reviewed Jun 29, 2023
View reviewed changes
Copy link
Member

@abhi-mosaic abhi-mosaic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The colors/linestyles of the MPT-30B latency-throughput curve need to be fixed a bit:

  • 80GB = blue, 40GB = green
  • 1 GPU = dash, 2GPU = dots, 4GPu = solid

this way it will match the 7B plot which looks excellent!

@abhi-mosaic
Copy link
Member

If possible I would also recommend standardizing the colors of the plots with latency @ token length. Sometihng like

  • bs=1 (red)
  • bs = 2 (blue)
  • bs = 4 (yellow)
  • bs = 8 (green)
  • bs = 16 (orange)
  • bs = 64 (purple)

so people can quickly look from one plot to the next and compare

@sashaDoubov
Copy link
Contributor Author

@abhi-mosaic fixing plots now!

@sashaDoubov sashaDoubov merged commit cc84569 into mosaicml:main Jun 29, 2023
10 checks passed
@sashaDoubov sashaDoubov mentioned this pull request Jun 29, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@abhi-mosaic abhi-mosaic abhi-mosaic left review comments

@vchiley vchiley vchiley approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sashaDoubov @abhi-mosaic @vchiley