Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KV Cache Interface] Text Generation & Decoder Engine Implementation #1089

Merged
merged 101 commits into from
Jun 28, 2023

Commits on Jun 5, 2023

  1. initial commit

    dbogunowicz committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    48ac0ac View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    cf7f2b9 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2023

  1. Configuration menu
    Copy the full SHA
    832630a View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2023

  1. Configuration menu
    Copy the full SHA
    9958c83 View commit details
    Browse the repository at this point in the history
  2. limit to 150mb

    dbogunowicz committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    e6d2b03 View commit details
    Browse the repository at this point in the history
  3. ready to review

    dbogunowicz committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    7f9935b View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2023

  1. initial commit

    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    b1cf01b View commit details
    Browse the repository at this point in the history
  2. [Codegen][ORT][Static Seq Length] TextGenerationPipeline (#946)

    * initial commit
    
    * coreys simplifications
    
    * finishing the second model static
    
    * ready, time for beautification
    
    * ready for review
    
    * moved the code to examples
    
    * fix eos logic
    
    * add argument num_tokens_to_generate
    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    0a3f48d View commit details
    Browse the repository at this point in the history
  3. [CodeGen][Documentation] (#956)

    * initial commit
    
    * coreys simplifications
    
    * finishing the second model static
    
    * ready, time for beautification
    
    * ready for review
    
    * moved the code to examples
    
    * fix eos logic
    
    * add argument num_tokens_to_generate
    
    * initial commit
    
    * change order
    
    * Update examples/codegen/README.md
    
    Co-authored-by: corey-nm <109536191+corey-nm@users.noreply.github.com>
    
    ---------
    
    Co-authored-by: corey-nm <109536191+corey-nm@users.noreply.github.com>
    2 people authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    add4625 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    22d2746 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    7f1651d View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b85746d View commit details
    Browse the repository at this point in the history
  7. refactor sucessfull

    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    aadc608 View commit details
    Browse the repository at this point in the history
  8. Pipeline fully refactored, time to test engine support. Note: Sliding…

    … window not yet implemented!
    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    58bc2b0 View commit details
    Browse the repository at this point in the history
  9. First iteration with Sage

    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    d538444 View commit details
    Browse the repository at this point in the history
  10. Apply suggestions from code review

    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    e19676b View commit details
    Browse the repository at this point in the history
  11. ORT agrees with the Engine. But they both give not entirely correct r…

    …esult. Hey, this is good news still
    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    7908b74 View commit details
    Browse the repository at this point in the history
  12. dynamic ORT vs static DS

    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    4bc3472 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    c07f7ed View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    fb77838 View commit details
    Browse the repository at this point in the history
  15. adjust shapes and slicing to enable static autoregressive pass - ISSU…

    …E: tokens past the base seq len are repeated
    bfineran authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    2097463 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    5eb10a9 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    9213f29 View commit details
    Browse the repository at this point in the history
  18. cleanup the pipeline

    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    d9af004 View commit details
    Browse the repository at this point in the history
  19. further cleanup post merge

    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    476f25d View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    fab44e4 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    d454e2f View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    1613e25 View commit details
    Browse the repository at this point in the history
  23. Stop saving tmp files, otherwise the engine looks for external files …

    …in the wrong place
    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    b61055c View commit details
    Browse the repository at this point in the history
  24. Left pad support

    bfineran authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    6ee25fc View commit details
    Browse the repository at this point in the history
  25. cleanup

    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    5d3004b View commit details
    Browse the repository at this point in the history
  26. cleanup2

    dbogunowicz authored and markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    ace6fa5 View commit details
    Browse the repository at this point in the history
  27. Add in pipeline timing

    markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    388586d View commit details
    Browse the repository at this point in the history
  28. add in force tokens logic

    markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    afd0139 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    30eeda7 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    5882b56 View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    4bbe33d View commit details
    Browse the repository at this point in the history
  32. nest input shape override

    markurtz committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    afa5746 View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    e2bb78c View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    2299009 View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2023

  1. Configuration menu
    Copy the full SHA
    2935b77 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2023

  1. Configuration menu
    Copy the full SHA
    b89b156 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2023

  1. initial commit

    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    dc3d61b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a294265 View commit details
    Browse the repository at this point in the history
  3. limit to 150mb

    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    af97f2b View commit details
    Browse the repository at this point in the history
  4. ready to review

    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    c117788 View commit details
    Browse the repository at this point in the history
  5. fix the erronous Makefile

    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    4ad5f49 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    9e816bb View commit details
    Browse the repository at this point in the history
  7. perhaps fixed GHA

    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    f97467f View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    6be8d87 View commit details
    Browse the repository at this point in the history
  9. initial commit

    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    e2f088d View commit details
    Browse the repository at this point in the history
  10. Merge remote-tracking branch 'origin/feature/damian/do_not_save_to_tm…

    …p' into feature/damian/codegen_pipeline_clean
    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    9fc6c64 View commit details
    Browse the repository at this point in the history
  11. tested with actual model

    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    a610faf View commit details
    Browse the repository at this point in the history
  12. remove val_inp argument

    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    347d1fb View commit details
    Browse the repository at this point in the history
  13. Update README.md

    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    e11027c View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    a950910 View commit details
    Browse the repository at this point in the history
  15. Update README.md

    dbogunowicz committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    c1d02dc View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    711cdfb View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2023

  1. Configuration menu
    Copy the full SHA
    e602662 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. Configuration menu
    Copy the full SHA
    06b5246 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2023

  1. initial implementation

    dbogunowicz committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    5d59d23 View commit details
    Browse the repository at this point in the history
  2. initial implementation

    dbogunowicz committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    765a5f7 View commit details
    Browse the repository at this point in the history
  3. Revert "initial implementation"

    This reverts commit 765a5f7.
    dbogunowicz committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    15586a4 View commit details
    Browse the repository at this point in the history
  4. rebase

    dbogunowicz committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    4d35779 View commit details
    Browse the repository at this point in the history
  5. add tests

    dbogunowicz committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    775c648 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    54aec69 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    25cdd38 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2023

  1. Configuration menu
    Copy the full SHA
    830a85e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    388e7ab View commit details
    Browse the repository at this point in the history
  3. initial implementation

    dbogunowicz committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    3970a7a View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7cdf939 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    950c653 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    ea82e99 View commit details
    Browse the repository at this point in the history
  7. Revert "remove files to make review easier"

    This reverts commit ea82e99.
    dbogunowicz committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    016cac1 View commit details
    Browse the repository at this point in the history
  8. Merge DecoderKVCache with KVCacheORT (KVCacheORT will not exist, it i…

    …s just an abstraction)
    
    * rebase
    
    * add tests
    dbogunowicz committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    c6efccd View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    a19cf2e View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    c59da37 View commit details
    Browse the repository at this point in the history
  11. DecoderKVCache that manipulates cache state and additionally passes i…

    …nfo to the engine via KVCache object
    dbogunowicz committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    6d40c03 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2023

  1. Configuration menu
    Copy the full SHA
    741f452 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7b27abe View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2023

  1. Configuration menu
    Copy the full SHA
    db6b54b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b3fb3b8 View commit details
    Browse the repository at this point in the history
  3. Merge remote-tracking branch 'origin/feature/damian/kv_cache_ort' int…

    …o feature/damian/decoder_engine
    dbogunowicz committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    47c0c4b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    76e332d View commit details
    Browse the repository at this point in the history
  5. Merge remote-tracking branch 'origin/feature/damian/kv_cache_ort' int…

    …o feature/damian/decoder_engine
    dbogunowicz committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    4791ed3 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    8c5734b View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    6c5daab View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2023

  1. Configuration menu
    Copy the full SHA
    812408c View commit details
    Browse the repository at this point in the history
  2. fix no kv cache inference

    dbogunowicz committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    952abda View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2ff4987 View commit details
    Browse the repository at this point in the history
  4. remove the prefill option

    dbogunowicz committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    725a210 View commit details
    Browse the repository at this point in the history
  5. fix docstring

    dbogunowicz committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    108596e View commit details
    Browse the repository at this point in the history
  6. remove prefill

    dbogunowicz committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    f6a9baf View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    b25886a View commit details
    Browse the repository at this point in the history
  8. Merge remote-tracking branch 'origin/feature/damian/kv_cache_ort' int…

    …o feature/damian/decoder_engine
    dbogunowicz committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    53d7b70 View commit details
    Browse the repository at this point in the history
  9. merge

    dbogunowicz committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    0b0f74a View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    4d6860a View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    d68f045 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2023

  1. addressed PR comments

    dbogunowicz committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    759dc93 View commit details
    Browse the repository at this point in the history
  2. merge

    dbogunowicz committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    3e1d32f View commit details
    Browse the repository at this point in the history
  3. quality

    dbogunowicz committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    4c39d7f View commit details
    Browse the repository at this point in the history