Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Brandon/flux model loading #6739

Merged
merged 113 commits into from
Aug 27, 2024
Merged

Brandon/flux model loading #6739

merged 113 commits into from
Aug 27, 2024

Commits on Aug 26, 2024

  1. Configuration menu
    Copy the full SHA
    5256f26 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    562c2cc View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b617631 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6a068cc View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    5149a3e View commit details
    Browse the repository at this point in the history
  6. Use the FluxPipeline.encode_prompt() api rather than trying to run th…

    …e two text encoders separately.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    761e49f View commit details
    Browse the repository at this point in the history
  7. Got FLUX schnell working with 8-bit quantization. Still lots of rough…

    … edges to clean up.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    5e1b3e9 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    e71c7d0 View commit details
    Browse the repository at this point in the history
  9. Make 8-bit quantization save/reload work for the FLUX transformer. Re…

    …load is still very slow with the current optimum.quanto implementation.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    f7753be View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    28654ec View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    40e9a4e View commit details
    Browse the repository at this point in the history
  12. WIP - experimentation

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    df1ac07 View commit details
    Browse the repository at this point in the history
  13. Make quantized loading fast.

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    11279ab View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    bcb7f8e View commit details
    Browse the repository at this point in the history
  15. Split a FluxTextEncoderInvocation out from the FluxTextToImageInvocat…

    …ion. This has the advantage that we benfit from automatic caching when the prompt isn't changed.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    3f1fbc6 View commit details
    Browse the repository at this point in the history
  16. wip

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    06d35c3 View commit details
    Browse the repository at this point in the history
  17. NF4 loading working... I think.

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    b1cf2f5 View commit details
    Browse the repository at this point in the history
  18. NF4 inference working

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    31c8d76 View commit details
    Browse the repository at this point in the history
  19. Clean up NF4 implementation.

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    17f5952 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    52ff3c7 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    307a130 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    27f33bb View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    e157ff3 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    6779e03 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    4bb3438 View commit details
    Browse the repository at this point in the history
  26. Run Ruff

    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    6b0f5f4 View commit details
    Browse the repository at this point in the history
  27. Run Ruff

    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    3814cd7 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    a6ad70e View commit details
    Browse the repository at this point in the history
  29. Add backend functions and classes for Flux implementation, Update the…

    … way flux encoders/tokenizers are loaded for prompt encoding, Update way flux vae is loaded
    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    f3096a8 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    5fc6c28 View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    68d28db View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    f3ebbe1 View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    efab4a3 View commit details
    Browse the repository at this point in the history
  34. Install sub directories with folders correctly, ensure consistent dty…

    …pe of tensors in flux pipeline and vae
    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    95a2d97 View commit details
    Browse the repository at this point in the history
  35. Select dev/schnell based on state dict, use correct max seq len based…

    … on dev/schnell, and shift in inference, separate vae flux params into separate config
    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    98151ce View commit details
    Browse the repository at this point in the history
  36. Fix FLUX output image clamping. And a few other minor fixes to make i…

    …nference work with the full bfloat16 FLUX transformer model.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    5d7e154 View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    870ecd3 View commit details
    Browse the repository at this point in the history
  38. Configuration menu
    Copy the full SHA
    24829b9 View commit details
    Browse the repository at this point in the history
  39. Fix styling/lint

    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    f36c6d0 View commit details
    Browse the repository at this point in the history
  40. Configuration menu
    Copy the full SHA
    bebc6d3 View commit details
    Browse the repository at this point in the history
  41. Configuration menu
    Copy the full SHA
    3f845d9 View commit details
    Browse the repository at this point in the history
  42. exclude flux models from main model dropdown

    Mary Hipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    8b3e386 View commit details
    Browse the repository at this point in the history
  43. add default workflow for flux t2i

    Mary Hipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    35c263a View commit details
    Browse the repository at this point in the history
  44. Rename t5Encoder -> t5_encoder.

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    ec360ee View commit details
    Browse the repository at this point in the history
  45. Address minor review comments.

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    c822c3d View commit details
    Browse the repository at this point in the history
  46. Update doc string for import_local_model and remove access_token sinc…

    …e it's only usable for local file paths
    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    19238ed View commit details
    Browse the repository at this point in the history
  47. Configuration menu
    Copy the full SHA
    ede26a7 View commit details
    Browse the repository at this point in the history
  48. Configuration menu
    Copy the full SHA
    f0408bb View commit details
    Browse the repository at this point in the history
  49. Configuration menu
    Copy the full SHA
    9e888b1 View commit details
    Browse the repository at this point in the history
  50. Configuration menu
    Copy the full SHA
    6afb113 View commit details
    Browse the repository at this point in the history
  51. Add docs to the requantize(...) function explaining why it was copied…

    … from optimum-quanto.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    519bf71 View commit details
    Browse the repository at this point in the history
  52. Configuration menu
    Copy the full SHA
    c549a49 View commit details
    Browse the repository at this point in the history
  53. update flux_model_loader node to take a T5 encoder from node field in…

    …stead of hardcoded list, assume all models have been downloaded
    maryhipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    41fb09b View commit details
    Browse the repository at this point in the history
  54. Configuration menu
    Copy the full SHA
    c66ccad View commit details
    Browse the repository at this point in the history
  55. add FLUX schnell starter models and submodels as dependenices or adho…

    …c download options
    maryhipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    9020a8a View commit details
    Browse the repository at this point in the history
  56. fix(ui): only exclude flux main models from linear UI dropdown, not m…

    …odel manager list
    Mary Hipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    7264920 View commit details
    Browse the repository at this point in the history
  57. fix(ui): pass base/type when installing models, add flux formats to M…

    …M badges
    Mary Hipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    3fe9582 View commit details
    Browse the repository at this point in the history
  58. feat(ui): create new field for t5 encoder models in nodes

    Mary Hipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    24831d4 View commit details
    Browse the repository at this point in the history
  59. tsc and lint fix

    Mary Hipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    6899762 View commit details
    Browse the repository at this point in the history
  60. fix schema

    Mary Hipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    f67c4da View commit details
    Browse the repository at this point in the history
  61. update default workflow

    maryhipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    a04d479 View commit details
    Browse the repository at this point in the history
  62. fix(worker) fix T5 type

    maryhipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    192eda7 View commit details
    Browse the repository at this point in the history
  63. add better workflow description

    maryhipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    3c861fd View commit details
    Browse the repository at this point in the history
  64. add better workflow name

    maryhipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    dcfdc00 View commit details
    Browse the repository at this point in the history
  65. Configuration menu
    Copy the full SHA
    f51dd36 View commit details
    Browse the repository at this point in the history
  66. Update load_flux_model_bnb_llm_int8.py to work with a single-file FLU…

    …X transformer checkpoint.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    3c9811f View commit details
    Browse the repository at this point in the history
  67. Configuration menu
    Copy the full SHA
    9982bc2 View commit details
    Browse the repository at this point in the history
  68. Configuration menu
    Copy the full SHA
    c5c60f5 View commit details
    Browse the repository at this point in the history
  69. Remove automatic install of models during flux model loader, remove n…

    …o longer used import function on context
    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    5f3e325 View commit details
    Browse the repository at this point in the history
  70. Run ruff

    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    bc6e1ba View commit details
    Browse the repository at this point in the history
  71. Configuration menu
    Copy the full SHA
    407796c View commit details
    Browse the repository at this point in the history
  72. added FLUX dev to starter models

    maryhipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    d4ec434 View commit details
    Browse the repository at this point in the history
  73. Configuration menu
    Copy the full SHA
    374dc82 View commit details
    Browse the repository at this point in the history
  74. Configuration menu
    Copy the full SHA
    80a46d2 View commit details
    Browse the repository at this point in the history
  75. Mark FLUX nodes as prototypes.

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    5406a2f View commit details
    Browse the repository at this point in the history
  76. Configuration menu
    Copy the full SHA
    c9c4e47 View commit details
    Browse the repository at this point in the history
  77. Tidy is_schnell detection logic.

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    22a3b3d View commit details
    Browse the repository at this point in the history
  78. Configuration menu
    Copy the full SHA
    5307a6f View commit details
    Browse the repository at this point in the history
  79. Configuration menu
    Copy the full SHA
    5e9ef4b View commit details
    Browse the repository at this point in the history
  80. Move prepare_latent_image_patches(...) to sampling.py with all of the…

    … related FLUX inference code.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    0e9f6f7 View commit details
    Browse the repository at this point in the history
  81. Run FLUX VAE decoding in the user's preferred dtype rather than float…

    …32. Tested, and seems to work well at float16.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    b5c937e View commit details
    Browse the repository at this point in the history
  82. Configuration menu
    Copy the full SHA
    f34a923 View commit details
    Browse the repository at this point in the history
  83. Configuration menu
    Copy the full SHA
    b8d4630 View commit details
    Browse the repository at this point in the history
  84. Configuration menu
    Copy the full SHA
    a31c02b View commit details
    Browse the repository at this point in the history
  85. Configuration menu
    Copy the full SHA
    9899e42 View commit details
    Browse the repository at this point in the history
  86. Configuration menu
    Copy the full SHA
    cfcd860 View commit details
    Browse the repository at this point in the history
  87. Configuration menu
    Copy the full SHA
    4089ff2 View commit details
    Browse the repository at this point in the history
  88. Configuration menu
    Copy the full SHA
    54c48c3 View commit details
    Browse the repository at this point in the history
  89. Configuration menu
    Copy the full SHA
    5af214b View commit details
    Browse the repository at this point in the history
  90. Configuration menu
    Copy the full SHA
    f4612a9 View commit details
    Browse the repository at this point in the history
  91. Configuration menu
    Copy the full SHA
    18c0ec3 View commit details
    Browse the repository at this point in the history
  92. Downgrade revert torch version after removing optimum-qanto, and othe…

    …r minor version-related fixes.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    098db5c View commit details
    Browse the repository at this point in the history
  93. Configuration menu
    Copy the full SHA
    dbdd851 View commit details
    Browse the repository at this point in the history
  94. Configuration menu
    Copy the full SHA
    1d6c83b View commit details
    Browse the repository at this point in the history
  95. Replace swish() with torch.nn.functional.silu(h). They are functional…

    …ly equivalent, but in my test VAE deconding was ~8% faster after the change.
    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    1413ff9 View commit details
    Browse the repository at this point in the history
  96. Configuration menu
    Copy the full SHA
    bd1b37d View commit details
    Browse the repository at this point in the history
  97. Configuration menu
    Copy the full SHA
    d159fe6 View commit details
    Browse the repository at this point in the history
  98. ruff

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    877b88e View commit details
    Browse the repository at this point in the history
  99. Remove flux repo dependency

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    ae94e48 View commit details
    Browse the repository at this point in the history
  100. Configuration menu
    Copy the full SHA
    f046a38 View commit details
    Browse the repository at this point in the history
  101. ruff format

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    9f6f404 View commit details
    Browse the repository at this point in the history
  102. Remove outdated TODO.

    RyanJDick authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    9a530c7 View commit details
    Browse the repository at this point in the history
  103. Configuration menu
    Copy the full SHA
    bb80697 View commit details
    Browse the repository at this point in the history
  104. Configuration menu
    Copy the full SHA
    642a953 View commit details
    Browse the repository at this point in the history
  105. Configuration menu
    Copy the full SHA
    a90d098 View commit details
    Browse the repository at this point in the history
  106. Fix type error in tsc

    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    40a3fa5 View commit details
    Browse the repository at this point in the history
  107. Run ruff

    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    b9238b6 View commit details
    Browse the repository at this point in the history
  108. Rename params for flux and flux vae, add comments explaining use of t…

    …he config_path in model config
    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    5a5ca10 View commit details
    Browse the repository at this point in the history
  109. update default workflow for flux

    Mary Hipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    bf59ab3 View commit details
    Browse the repository at this point in the history
  110. remove prompt

    Mary Hipp authored and brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    bd2692b View commit details
    Browse the repository at this point in the history
  111. Run ruff

    brandonrising committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    5d42e67 View commit details
    Browse the repository at this point in the history
  112. Configuration menu
    Copy the full SHA
    c510234 View commit details
    Browse the repository at this point in the history
  113. Configuration menu
    Copy the full SHA
    3b29bad View commit details
    Browse the repository at this point in the history