A CozyMantis experiment with the new IPAdapter Batch nodes for driving video
![image](https://private-user-images.githubusercontent.com/5381731/318035605-d90a2be4-e291-4292-949b-9957e94f6880.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1Nzk0NDcsIm5iZiI6MTcyNDU3OTE0NywicGF0aCI6Ii81MzgxNzMxLzMxODAzNTYwNS1kOTBhMmJlNC1lMjkxLTQyOTItOTQ5Yi05OTU3ZTk0ZjY4ODAucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjVUMDk0NTQ3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9N2Y4ZmY3YTA0MDU4NjY5OGYyMDM0NjdkNzNjMGQzMTRmNTNkZTQyNjcyMTdhZWYzODQxNzdlNzAyYjQ4YTk0OSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.pu6yqek19StSbVj-5rDI0PtuJH8KV0bbRINXjIfr5jQ)
demo.mp4
- Generate a 1024x1024 reference image with 2 general passes and a face detailing pass;
- Run it through SV3D to get a 576x576 turnaround;
demo1-sv3d.mp4
-
Run the frames through the IPAdapter Tiled Batch node;
-
Upscale each frame to 1024x1024 using different values for denoise and the batch IPAdapter output models.
Denoise 0.3
demo1-3.mp4
Denoise 0.5
demo1-5.mp4
Denoise 0.7
7.mp4
Denoise 0.9
demo1-9.mp4
This workflow depends on a few other great custom nodes that you should install first. The recommended path is to use ComfyUI Manager's "Install missing nodes" feature - see https://github.com/ltdrdata/ComfyUI-Manager.
- Open the ComfyUI Manager Menu
- Click on the "Install Missing Custom Nodes" button
- Restart ComfyUI and refresh the web app
Alternatively, for manual installs, here is the list of required custom nodes and packs:
- Our Cozy Utils pack (https://github.com/cozymantis/cozy-utils-comfyui-nodes)
- Our Cozy Pose Face&Body Generator node (https://cozymantis.gumroad.com/l/cozy-pose-face-body-openpose-generator-comfyui-node-character-reference)
- Impact Pack (https://github.com/ltdrdata/ComfyUI-Impact-Pack)
- IPAdapter Plus (https://github.com/cubiq/ComfyUI_IPAdapter_plus)
- FaceID requires insightface, you need to install it in your ComfyUI environment. Run
pip install insightface
inside your ComfyUI python env, or try the "Install pip package" options in the ComfyUI Manager menu. Check this issue for help. - Check the IPAdapter installation instructions in the repo above.
- FaceID requires insightface, you need to install it in your ComfyUI environment. Run
- WAS Node Suite (https://github.com/WASasquatch/was-node-suite-comfyui)
- Image Resize for ComfyUI (https://github.com/palant/image-resize-comfyui)
- ComfyUI's ControlNet Auxiliary Preprocessors (https://github.com/Fannovel16/comfyui_controlnet_aux)
- rgthree (https://github.com/rgthree/rgthree-comfy)
- ComfyUI Essentials (https://github.com/cubiq/ComfyUI_essentials)
- ComfyUI Logic (https://github.com/theUpsider/ComfyUI-Logic)
- FizzNodes (https://github.com/FizzleDorf/ComfyUI_FizzNodes)
- An SD 1.5 checkpoint of your choice => models/checkpoints
- https://huggingface.co/stabilityai/sv3d/blob/main/sv3d_p.safetensors => models/checkpoints
- https://huggingface.co/Bingsu/adetailer/blob/main/face_yolov8m.pt => models/ultralytics/bbox
- https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth => models/sams
- https://huggingface.co/h94/IP-Adapter/resolve/main/models/ip-adapter-plus_sd15.safetensors => models/ipadapter
- https://huggingface.co/webui/ControlNet-modules-safetensors/blob/main/t2iadapter_openpose-fp16.safetensors => models/controlnet
- https://huggingface.co/h94/IP-Adapter/blob/main/models/image_encoder/model.safetensors => models/clip_vision/CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors
Made with π by the CozyMantis squad. Check out our ComfyUI nodes and workflows!
Please check licenses and terms of use for each of the nodes and models required by this workflow.
The workflow should not be used to intentionally create or disseminate images that create hostile or alienating environments for people. This includes generating images that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.
Also check out our improved character rotation worflow for ComfyUI!