[Pipeline Refactor][Text-Generation] Refactor `transformers` helpers functions #1394

dbogunowicz · 2023-11-08T13:44:03Z

Feature Description

A small refactor that:

factors out the transformers-specific functionalities away from v2.text_generation.pipeline
allows specifying onnx_model_name in get_deployment_path

Manual Testing

from deepsparse.v2.text_generation.pipeline import TextGenerationPipeline
model_path = "hf:mgoin/TinyStories-1M-deepsparse"
pipeline = TextGenerationPipeline(model_path)
print(pipeline(prompt="Cool prompt bro"))

created=datetime.datetime(2023, 11, 8, 13, 51, 1, 19125) prompts='Cool prompt bro' generations=[GeneratedText(text='och.\n\nThe little girl was so excited. She ran to the store and bought a big bag of candy. She was so happy! She had a great time at the store and she was so happy.\n', score=None, finished=True, finished_reason='stop')]

…e code to remove repeat code, update map function

…eature/damian/v2/factor_out_transformation_utils

…scope of fixtures module to help with speed

dsikka

once rebased, I'll rereview but split/join landed so you could test with multiple prompts and also possibly pull in my unit test PR to see if all the unit tests still pass with the changes.

…eature/damian/v2/factor_out_transformation_utils

dbogunowicz · 2023-11-10T07:20:43Z

@dsikka @bfineran Rebased this branch on top of features/v2/unit_testing and ran the unittests:

 pytest tests/deepsparse/v2/unit

Result

tests/deepsparse/v2/unit/text_generation/test_kv_cache.py .                                                                                                                                                                                                                                                      [ 11%]
tests/deepsparse/v2/unit/text_generation/test_msic.py .                                                                                                                                                                                                                                                          [ 22%]
tests/deepsparse/v2/unit/text_generation/test_process_inputs.py .                                                                                                                                                                                                                                                [ 33%]
tests/deepsparse/v2/unit/text_generation/test_single_token_engine.py ....                                                                                                                                                                                                                                        [ 77%]
tests/deepsparse/v2/unit/text_generation/test_token_generation.py ..                                                                                                                                                                                                                                             [100%]

Also tested manually:

from deepsparse.v2.text_generation.pipeline import TextGenerationPipeline
model_path = "hf:mgoin/TinyStories-1M-deepsparse"
pipeline = TextGenerationPipeline(model_path)
out = pipeline(prompt=["MF Doom was a great rapper.])

src/deepsparse/v2/text_generation/nl_engine_operator.py

bfineran

LGTM

bfineran · 2023-11-13T16:02:23Z

tests/deepsparse/v2/unit/text_generation/test_msic.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from deepsparse.v2.text_generation import CompilePromptLogits


note rebase

dsikka

LGTM pending a few comments and quick tests for the initialization, which can check ORT vs Deepsparse (i.e. make sure both get initialized properly) with and without internal kv_cache. Just want to make sure all the cases get set-up correctly before calling the first operator

src/deepsparse/transformers/helpers.py

dsikka · 2023-11-13T20:05:35Z

src/deepsparse/transformers/helpers.py

+    return onnx_path, config, tokenizer
+
+
+def get_deployment_path(


We talked about a bug/fix for this in stand-up: #1396
Have we considered this fix as part of this function?

Let's not make this PR take on too many hats. The problem that @bfineran was fixing was particular to the few transformer tasks. More of those are bound to happen once we swap v1 transformers pipelines in favor of v2 transformers pipelines. We will deal with them in their own time, this PR seems fairly orthogonal.

dsikka · 2023-11-13T20:07:51Z

src/deepsparse/transformers/helpers.py

@@ -44,7 +46,84 @@
 _LOGGER = get_main_logger()


-def get_deployment_path(model_path: str) -> Tuple[str, str]:
+def setup_transformers_pipeline(


Could we also get the v1 pipeline to use these? Fear is that we'll have two places to update going forward.

Sure, good idea.

This reverts commit 51c4ee6.

dbogunowicz · 2023-11-20T11:40:43Z

@dsikka Introduced corrections according to your comments and ran pytest tests/deepsparse/transformers/ to make sure I didnt break anything.

* Pipelines Refactor - Initial Impl (#1287) * [Pipeline Refactor] Additional functionality, engine operator, linear router and image classification pipeline/operators/example (#1325) * initial functionality and working example with image classification * remove testing image * update args * initial functionality and working example with image classification * remove testing image * pr comments * defines schemas for operators and test * add image classification test, PR comments * fix input/output handling in pipeline and operator base classes to be more generic; remove context * add additional operator input message * typo fix * [v2] EngineOperator updates to make continuous batching easier (#1371) * [v2] EngineOperator updates to make continuous batching easier * test fixes * [Pipeline Refactor] Update routes, text generation initial functionality (#1348) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * [Pipeline Refactor] Additional Operators, Route update and completed generation functionality (#1356) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * initial functionality and working example with image classification * updates func * prompt inference, initial functionality * finish generation operators and update routes * further breakdown operators * add operators * fix can_operate condition * update can_operate to not rely on the inference_state * rebase + update * fix condition * fix capacity settting again * typo fixes * [Pipeline Refactor] Split/Join Functionality for multiple prompts (#1384) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * move map to base class * [Pipeline Refactor] Unit Testing for Text Generation Operators (#1392) * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * fix name * [Continuous Batching] Queue Implementation to support batching grouping and prioritization (#1373) * [Continuous Batching] Queue Implementation to support batching grouping and prioritization * has_key method * thread safety * add blocking option for pop_batch * update docstring * allow mutex to be shared across continuous batching objects * revert last commit * [Continuous Batching] Executor thread for running continuous batching (#1374) * [Continuous Batching] Executor thread for running continuous batching * quality * ensure that executor stops when main thread does - clean up test hack * [ContinuousBatching] ContinuousBatchingScheduler Implementation (#1375) * [ContinuousBatching] ContinuousBatchingScheduler Implementation * cleanup unnecessary stop condition * [continuous batching] singleton pattern for scheduler (#1391) * [continuous batching] singleton pattern for scheduler * catch from review * [Pipeline Refactor][Text-Generation] Create a helper function for creating engine_inputs (#1364) * rebasing off my initial commit * cleanups * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py --------- Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> * [Pipeline Refactor][Text-Generation] Refactor `transformers` helpers functions (#1394) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * initial commit * fix error * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py * pipeline runs, but incorrectly * Revert "pipeline runs, but incorrectly" This reverts commit 51c4ee6. * PR review comments --------- Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> * [Text Generation][V2] End-to-end tests (#1402) * initial commit * initial commit * its working now * beautification * thank you Dipika <3 * ready to review * [Pipeline Refactor][Text Generation][Continuous Batching] Integration (#1409) * update split/join * use map * update * run end-to-end * clean-up * fix bug with batch size, introduce SplitRoute dataclass * update tests to use new inputs/outputs * use the normal scheduler for internal kv_cache * add pipeline inpuits * clean-up * change engine type, update docstrings, update override function to be more generic * move subgraph functionality to its own function; clean-up cont batching in text gen pipeline * update linear pathway to also use subgraph execution * rebase fix * fix tests * [Pipeline Refactor] Operator Registry (#1420) * initial registry functionality * use sparsezoo mixin * [Pipeline Refactor] Fix Operator scheduling to fix issue with slow execution (#1453) * fix scheduling to fix issue with engine running very slowly; introduce new completed attribute for Subgraph instead of checking instance type * fix warning message * [Pipeline Refactor] Add `Pipeline.create` method to initialize pipelines (#1457) * add pipeline create method for pipeline creation using the operator registry * add instance check * [Pipeline Refactor] async (#1380) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * initial functionality and working example with image classification * updates func * prompt inference, initial functionality * finish generation operators and update routes * further breakdown operators * add operators * fix can_operate condition * update can_operate to not rely on the inference_state * rebase + update * fix condition * async initial functionality * fix capacity settting again * add blocking * more testing * update to use split/join * fix * rebase fix * remove index * change event loop * rebase fix * update async run to use new operator scheduling properly * rebase fixes (#1458) * more fixes (#1459) --------- Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com>

* Pipelines Refactor - Initial Impl (#1287) * [Pipeline Refactor] Additional functionality, engine operator, linear router and image classification pipeline/operators/example (#1325) * initial functionality and working example with image classification * remove testing image * update args * initial functionality and working example with image classification * remove testing image * pr comments * defines schemas for operators and test * add image classification test, PR comments * fix input/output handling in pipeline and operator base classes to be more generic; remove context * add additional operator input message * typo fix * [v2] EngineOperator updates to make continuous batching easier (#1371) * [v2] EngineOperator updates to make continuous batching easier * test fixes * [Pipeline Refactor] Update routes, text generation initial functionality (#1348) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * [Pipeline Refactor] Additional Operators, Route update and completed generation functionality (#1356) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * initial functionality and working example with image classification * updates func * prompt inference, initial functionality * finish generation operators and update routes * further breakdown operators * add operators * fix can_operate condition * update can_operate to not rely on the inference_state * rebase + update * fix condition * fix capacity settting again * typo fixes * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * initial commit * fix error * [Pipeline Refactor] Split/Join Functionality for multiple prompts (#1384) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * move map to base class * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * [Pipeline Refactor] Unit Testing for Text Generation Operators (#1392) * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * fix name * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py * [Continuous Batching] Queue Implementation to support batching grouping and prioritization (#1373) * [Continuous Batching] Queue Implementation to support batching grouping and prioritization * has_key method * thread safety * add blocking option for pop_batch * update docstring * allow mutex to be shared across continuous batching objects * revert last commit * [Continuous Batching] Executor thread for running continuous batching (#1374) * [Continuous Batching] Executor thread for running continuous batching * quality * ensure that executor stops when main thread does - clean up test hack * [ContinuousBatching] ContinuousBatchingScheduler Implementation (#1375) * [ContinuousBatching] ContinuousBatchingScheduler Implementation * cleanup unnecessary stop condition * [continuous batching] singleton pattern for scheduler (#1391) * [continuous batching] singleton pattern for scheduler * catch from review * [Pipeline Refactor][Text-Generation] Create a helper function for creating engine_inputs (#1364) * rebasing off my initial commit * cleanups * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py --------- Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> * pipeline runs, but incorrectly * it works for a single sequence * cleanup. now lets figure out how to run multiple sequences * [Pipeline Refactor][Text-Generation] Refactor `transformers` helpers functions (#1394) * add split/join functionality * update router to include split/join in parent class, refactor pipeline code to remove repeat code, update map function * process multiple generations * initial commit * fix error * unit testing for text generation operators * additional changes * unit testing completion * remove debug * fix * add todo * more clean-up * fix test * add docstrings/comments * break out tests to individual unit test files; add conftest and make scope of fixtures module to help with speed * Delete tests/deepsparse/v2/unit/text_generation/test_msic.py * pipeline runs, but incorrectly * Revert "pipeline runs, but incorrectly" This reverts commit 51c4ee6. * PR review comments --------- Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> * [Text Generation][V2] End-to-end tests (#1402) * initial commit * initial commit * its working now * beautification * thank you Dipika <3 * ready to review * integration tests pass * [Pipeline Refactor][Text Generation][Continuous Batching] Integration (#1409) * update split/join * use map * update * run end-to-end * clean-up * fix bug with batch size, introduce SplitRoute dataclass * update tests to use new inputs/outputs * use the normal scheduler for internal kv_cache * add pipeline inpuits * clean-up * change engine type, update docstrings, update override function to be more generic * move subgraph functionality to its own function; clean-up cont batching in text gen pipeline * update linear pathway to also use subgraph execution * rebase fix * fix tests * [Pipeline Refactor] Operator Registry (#1420) * initial registry functionality * use sparsezoo mixin * fix tricky rebase * one more cleanup * got tests to work after rebase. implementing SPLIT and JOIN in linearouter now * pipeline working, with GraphRouter. Needs some more testing * ready for review * cleanup * simplify after PR review round * [Pipeline Refactor] Fix Operator scheduling to fix issue with slow execution (#1453) * fix scheduling to fix issue with engine running very slowly; introduce new completed attribute for Subgraph instead of checking instance type * fix warning message * [Pipeline Refactor] Add `Pipeline.create` method to initialize pipelines (#1457) * add pipeline create method for pipeline creation using the operator registry * add instance check * [Pipeline Refactor] async (#1380) * initial functionality and working example with image classification * remove testing image * rebase fixes * initial functionality and working example with image classification * text gen * updates func * prompt inference, initial functionality * remove image; update state docstring * Fix typo * add todo for split/join * remove context, clean-up args, remove prefill_preprocess_operaator * fix docstrings * initial functionality and working example with image classification * updates func * prompt inference, initial functionality * finish generation operators and update routes * further breakdown operators * add operators * fix can_operate condition * update can_operate to not rely on the inference_state * rebase + update * fix condition * async initial functionality * fix capacity settting again * add blocking * more testing * update to use split/join * fix * rebase fix * remove index * change event loop * rebase fix * update async run to use new operator scheduling properly * rebase fixes (#1458) * more fixes (#1459) * bring back functionalities that were lost in v2 during rebasing * Update src/deepsparse/transformers/helpers.py * ready for review * bring tests back" * quality * original readme * addressing Dipikas comments * Update src/deepsparse/transformers/pipelines/text_generation/pipeline_no_kv_cache.py * addressing PR review --------- Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com>

dsikka and others added 4 commits November 3, 2023 15:24

add split/join functionality

f18d5f3

update router to include split/join in parent class, refactor pipelin…

2c4d231

…e code to remove repeat code, update map function

process multiple generations

672ca20

initial commit

304eb35

dbogunowicz changed the base branch from main to v2 November 8, 2023 13:44

fix error

71515ac

dbogunowicz requested review from dsikka and bfineran November 8, 2023 13:51

dbogunowicz mentioned this pull request Nov 8, 2023

[Pipeline Refactor] Unit Testing for Text Generation Operators #1392

Merged

Merge remote-tracking branch 'origin/features/v2/run_multiple' into f…

6f1b175

…eature/damian/v2/factor_out_transformation_utils

dbogunowicz changed the base branch from v2 to features/v2/run_multiple November 9, 2023 13:39

Base automatically changed from features/v2/run_multiple to v2 November 9, 2023 14:49

dsikka added 10 commits November 9, 2023 16:12

unit testing for text generation operators

a508342

additional changes

cbb0e86

unit testing completion

2541581

remove debug

8c8989d

fix

f8d75e3

add todo

fd1e466

more clean-up

64c0552

fix test

913665a

add docstrings/comments

e15521f

break out tests to individual unit test files; add conftest and make …

379481e

…scope of fixtures module to help with speed

dsikka reviewed Nov 9, 2023

View reviewed changes

Merge remote-tracking branch 'origin/features/v2/unit_testing' into f…

a90a20a

…eature/damian/v2/factor_out_transformation_utils

dbogunowicz requested a review from dsikka November 10, 2023 07:21

mgoin reviewed Nov 10, 2023

View reviewed changes

src/deepsparse/v2/text_generation/nl_engine_operator.py Outdated Show resolved Hide resolved

Merge branch 'v2' into feature/damian/v2/factor_out_transformation_utils

c0c4240

dbogunowicz requested a review from mgoin November 13, 2023 08:27

bfineran approved these changes Nov 13, 2023

View reviewed changes

Delete tests/deepsparse/v2/unit/text_generation/test_msic.py

4f248dd

dsikka reviewed Nov 13, 2023

View reviewed changes

dsikka requested changes Nov 13, 2023

View reviewed changes

dbogunowicz and others added 5 commits November 14, 2023 10:46

Merge branch 'v2' into feature/damian/v2/factor_out_transformation_utils

98f7a6d

Merge branch 'v2' into feature/damian/v2/factor_out_transformation_utils

d1683b4

pipeline runs, but incorrectly

51c4ee6

Revert "pipeline runs, but incorrectly"

a4f6f19

This reverts commit 51c4ee6.

PR review comments

71f4c6d

dbogunowicz requested review from dsikka and bfineran November 20, 2023 11:40

dsikka approved these changes Nov 20, 2023

View reviewed changes

bfineran approved these changes Nov 20, 2023

View reviewed changes

dbogunowicz merged commit b80a417 into v2 Nov 20, 2023

dbogunowicz deleted the feature/damian/v2/factor_out_transformation_utils branch November 20, 2023 20:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Pipeline Refactor][Text-Generation] Refactor `transformers` helpers functions #1394

[Pipeline Refactor][Text-Generation] Refactor `transformers` helpers functions #1394

dbogunowicz commented Nov 8, 2023 •

edited

Loading

dsikka left a comment

dbogunowicz commented Nov 10, 2023 •

edited

Loading

bfineran left a comment

bfineran Nov 13, 2023

dsikka left a comment •

edited

Loading

dsikka Nov 13, 2023

dbogunowicz Nov 20, 2023

dsikka Nov 13, 2023

dbogunowicz Nov 20, 2023

dbogunowicz commented Nov 20, 2023

[Pipeline Refactor][Text-Generation] Refactor transformers helpers functions #1394

[Pipeline Refactor][Text-Generation] Refactor transformers helpers functions #1394

Conversation

dbogunowicz commented Nov 8, 2023 • edited Loading

Feature Description

Manual Testing

dsikka left a comment

Choose a reason for hiding this comment

dbogunowicz commented Nov 10, 2023 • edited Loading

bfineran left a comment

Choose a reason for hiding this comment

bfineran Nov 13, 2023

Choose a reason for hiding this comment

dsikka left a comment • edited Loading

Choose a reason for hiding this comment

dsikka Nov 13, 2023

Choose a reason for hiding this comment

dbogunowicz Nov 20, 2023

Choose a reason for hiding this comment

dsikka Nov 13, 2023

Choose a reason for hiding this comment

dbogunowicz Nov 20, 2023

Choose a reason for hiding this comment

dbogunowicz commented Nov 20, 2023

[Pipeline Refactor][Text-Generation] Refactor `transformers` helpers functions #1394

[Pipeline Refactor][Text-Generation] Refactor `transformers` helpers functions #1394

dbogunowicz commented Nov 8, 2023 •

edited

Loading

dbogunowicz commented Nov 10, 2023 •

edited

Loading

dsikka left a comment •

edited

Loading