Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TextGeneration][Timer] text gen specific timings + improved timing tooling #1121

Merged
merged 5 commits into from
Jul 24, 2023

Conversation

bfineran
Copy link
Member

@bfineran bfineran commented Jul 13, 2023

adds:

  • bugfix for TimerManager workflow for engine_forward subtimings
    • since engine_forward is always called in a child thread for batch splitting, it will not have access to the current timer contextvar
    • see changes in timer.py new_timer_context for this
  • adds timer.time helper contextmanager; see docstring for usage
  • adds subtimings for prefill, generation and the individual autoregressive passes for each in the text generation pipeline
    • see example in test plan for what these timings may look like

test_plan:
Manually for now - once we have more text gen pipeline tests, we can add assertions that the proper stages are generated in the timer manager:

# running text gen / opt pipeline
from deepsparse import Pipeline
opt = Pipeline.create(task="opt", model_path=MODEL_PATH, engine_type="onnxruntime", max_generated_tokens=128)
output = opt(sequences="Who is the president of the United States?", return_logits=True)

print(opt.timer_manger)
print(opt.timer_manager.stages)

Manager and stages:

TimerManager({'engine_token_generation': 4.507111581042409, 'engine_prompt_prefill_single': 0.03916543936356902, 'engine_prompt_prefill': 0.3927434356883168, 'pre_process': 0.001995994709432125, 'engine_token_generation_single': 0.035451228459050334, 'engine_forward': 4.909945313818753, 'total_inference': 4.912277169525623, 'post_process': 0.00030374620109796524})
['engine_token_generation', 'engine_prompt_prefill_single', 'engine_prompt_prefill', 'pre_process', 'engine_token_generation_single', 'engine_forward', 'total_inference', 'post_process']

@bfineran bfineran self-assigned this Jul 13, 2023
dbogunowicz
dbogunowicz previously approved these changes Jul 14, 2023
dbogunowicz
dbogunowicz previously approved these changes Jul 18, 2023
Copy link
Member

@rahul-tuli rahul-tuli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending response to comments!

src/deepsparse/transformers/pipelines/text_generation.py Outdated Show resolved Hide resolved
src/deepsparse/utils/timer.py Outdated Show resolved Hide resolved
src/deepsparse/utils/timer.py Outdated Show resolved Hide resolved
src/deepsparse/utils/timer.py Outdated Show resolved Hide resolved
src/deepsparse/transformers/pipelines/text_generation.py Outdated Show resolved Hide resolved
@bfineran
Copy link
Member Author

review suggestions LGTM

bfineran and others added 3 commits July 21, 2023 09:47
Revert to using timer.time for `TOKEN_GENERATION`
Remove finally clause from `contextmanagers`
Address review comments from @rahul-tuli
@dbogunowicz dbogunowicz merged commit 29a8f68 into main Jul 24, 2023
7 checks passed
@dbogunowicz dbogunowicz deleted the text-gen-timings branch July 24, 2023 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants