Experiencing issues with SD1.5 generation in newer versions of SD.Next... trying to find the reason

[anothertal3]

Hi,

I'm using two SD.Next instances right now. One has been used explicitly for SD1.5 and is a couple of months old:

Version: app=sd.next updated=2024-06-24 hash=94f6f0db branch=master 
             url=https://github.com/vladmandic/automatic/tree/master ui=main
Latest published version: bab17a0b4f91b41c885f10262ef8c8e70ba72faa 2024-08-31T20:57:34Z

The other is up-to-date:

Version: app=sd.next updated=2024-08-31 hash=bab17a0b branch=master
              url=https://github.com/vladmandic/automatic/tree/master ui=main

I would like to finally migrate everything over to the new version but I am unable to do so right now as I'm experiencing strange quality issues when using SD1.5 models.

For comparison I've create the following picture using Photon and the outdated instance:

The parameters according to SD.Next were:

reportage photo of woman in heavy plate armor, sitting in bistro in paris in the morning, coffee on table
Negative prompt: nsfw, cartoon, painting, illustration, (worst quality, low quality, normal quality:2)
Steps: 20, Seed: 1437661128, Sampler: DPM++ 2M, CFG scale: 5, Size: 512x768, Parser: Full parser, Model: photon_v1, Model hash: ec41bd2a82, VAE: vae-ft-mse-840000-ema-pruned2, Backend: Original, App: SD.Next, Version: 94f6f0d, Operations: hires; txt2img, Second pass: True, Hires force: True, Hires steps: 10, Hires upscaler: ESRGAN 4x NMKD Superscale, Hires upscale: 1.5, Hires size: 768x1152, Denoising strength: 0.3, Hires sampler: DPM++ 2M, Image CFG scale: 6, CFG rescale: 0.7, Sampler brownian: False, Sampler sigma: karras, Template: "reportage photo of woman in heavy plate armor, sitting in bistro in paris in the morning, coffee on table", Negative Template: "nsfw, cartoon, painting, illustration, (worst quality, low quality, normal quality:2)"

When I mimic this using the same "original" backend on the new version, I receive the following image:

During generation it appears to be completely fine until the very end. Then it suddenly displays as very faded and washed-out.

The parameters according to SD.Next are pretty much identical and according to them, VAE has been used.

reportage photo of woman in heavy plate armor, sitting in bistro in paris in the morning, coffee on table
Negative prompt: nsfw, cartoon, painting, illustration, (worst quality, low quality, normal quality:2)
Steps: 20, Seed: 1437661128, Sampler: DPM++ 2M, CFG scale: 5, Size: 512x768, Parser: Full parser, Model: photon_v1, Model hash: ec41bd2a82, VAE: vae-ft-mse-840000-ema-pruned2, Backend: Original, App: SD.Next, Version: bab17a0, Operations: txt2img; hires, Second pass: True, Hires force: True, Hires steps: 10, Hires upscaler: ESRGAN 4x NMKD Superscale, Hires upscale: 1.5, Hires size: 768x1152, Denoising strength: 0.3, Hires sampler: DPM++ 2M, Image CFG scale: 6, CFG rescale: 0.7, Sampler brownian: False, Sampler sigma: karras, Template: "reportage photo of woman in heavy plate armor, sitting in bistro in paris in the morning, coffee on table", Negative Template: "nsfw, cartoon, painting, illustration, (worst quality, low quality, normal quality:2)"

I then switched to "diffusers" backend, "Diffusers pipeline" = "Autodetect" and now I get the following image:

The contrast is now all over the place and there appear to be small artifacts.

Parameters:

reportage photo of woman in heavy plate armor, sitting in bistro in paris in the morning, coffee on table
Negative prompt: nsfw, cartoon, painting, illustration, (worst quality, low quality, normal quality:2)
Steps: 20, Seed: 1437661128, Sampler: DPM++ 2M, CFG scale: 5, Size: 512x768, Parser: Full parser, Model: photon_v1, Model hash: ec41bd2a82, VAE: vae-ft-mse-840000-ema-pruned2, Backend: Diffusers, App: SD.Next, Version: bab17a0, Operations: txt2img; hires; upscale, Pipeline: StableDiffusionPipeline, Second pass: True, Hires force: True, Hires steps: 10, Hires upscaler: ESRGAN 4x NMKD Superscale, Hires upscale: 1.5, Hires size: 768x1152, Denoising strength: 0.3, Hires sampler: DPM++ 2M, Image CFG scale: 6, CFG rescale: 0.7, Template: "reportage photo of woman in heavy plate armor, sitting in bistro in paris in the morning, coffee on table", Negative Template: "nsfw, cartoon, painting, illustration, (worst quality, low quality, normal quality:2)", Sampler options: karras/sde-dpmsolver++/order 2/low order

Additional information:

• There are no errors in the console during generation.
• All models/VAE/etc. are in a shared folder and used by both so that shouldn't be an issue
• I've tested a couple of SD1.5 models: all show that faded look when using "original" backend and all show more intense contrast than usual when using "diffusers" backend (albeit Photon appears to be the most horrible in that regard)
• Different samples do not seem to make that big of a difference (either the result is completely unusable or it's similar to the one described by me)
• I tested different VAEs to see if it makes any difference. Using incorrect ones will affect the image even more, so it's likely being used.
• Changing the pipelines doesn't appear to improve anything
• I added Hires for clarity but I will result in the same issues even without.

Some more details about the system and configurations:

Start-up log of current version:

11:27:04-227777 INFO     Starting SD.Next
11:27:04-229782 INFO     Logger: file="F:\stable-diffusion\sd.next2\sdnext.log" level=INFO size=19052982 mode=append
11:27:04-230782 INFO     Python version=3.10.11 platform=Windows
                         bin="F:\stable-diffusion\sd.next2\venv\Scripts\python.exe"
                         venv="F:\stable-diffusion\sd.next2\venv"
11:27:04-348662 INFO     Version: app=sd.next updated=2024-08-31 hash=bab17a0b branch=master
                         url=https://github.com/vladmandic/automatic/tree/master ui=main
11:27:04-729738 INFO     Platform: arch=AMD64 cpu=AMD64 Family 25 Model 97 Stepping 2, AuthenticAMD system=Windows
                         release=Windows-10-10.0.22631-SP0 python=3.10.11
11:27:04-747886 INFO     nVidia CUDA toolkit detected: nvidia-smi present
11:27:04-953932 INFO     Verifying requirements
11:27:04-960528 INFO     Verifying packages
11:27:04-960528 INFO     Startup: standard
11:27:04-961535 INFO     Verifying submodules
11:27:17-562384 INFO     Extensions enabled: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'stable-diffusion-webui-rembg', 'a1111-sd-webui-tagcomplete',
                         'adetailer', 'sd-dynamic-prompts', 'sd-webui-controlnet']
11:27:17-563387 INFO     Verifying requirements
11:27:17-564771 INFO     Updating Wiki
11:27:18-150108 INFO     Command line args: ['--upgrade', '--docs'] upgrade=True docs=True
11:27:27-670499 INFO     Load packages: {'torch': '2.3.1+cu121', 'diffusers': '0.30.2', 'gradio': '3.43.2'}
11:27:29-142568 INFO     VRAM: Detected=16.0 GB Optimization=none
11:27:29-146082 INFO     Engine: backend=Backend.ORIGINAL compute=cuda device=cuda attention="Scaled-Dot-Product"
                         mode=no_grad
11:27:29-235890 INFO     Device: device=NVIDIA GeForce RTX 4060 Ti n=1 arch=sm_90 cap=(8, 9) cuda=12.1 cudnn=8907
                         driver=560.70
11:27:31-191513 INFO     Available VAEs: path="F:\stable-diffusion\models\VAE" items=4
11:27:31-194071 INFO     Disabled extensions: ['sdnext-modernui', 'sd-webui-prevent-artifact']
11:27:31-251289 INFO     Available models: path="F:\stable-diffusion\models\Stable-diffusion" items=31 time=0.06
11:27:31-656125 INFO     Extension: script='extensions-builtin\Lora\scripts\lora_script.py'
                         [2;36m11:27:31-652829[0m[2;36m [0m[34mINFO    [0m LoRA networks: [33mavailable[0m=[1;36m3138[0m
                         [33mfolders[0m=[1;36m21[0m
11:27:32-322274 INFO     Extension: script='extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py' Using
                         sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_scheduler.sqlite3
11:27:33-260503 INFO     Extension: script='extensions\a1111-sd-webui-tagcomplete\scripts\tag_autocomplete_helper.py'
                         Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will
                         be limited to those added through the extra networks menu.
11:27:33-687919 INFO     Extension: script='extensions\adetailer\scripts\!adetailer.py' [-] ADetailer initialized.
                         version: 24.8.0, num models: 10
11:27:33-976863 INFO     Extension: script='extensions\sd-webui-controlnet\scripts\api.py' ControlNet preprocessor
                         location: F:\stable-diffusion\sd.next2\extensions\sd-webui-controlnet\annotator\downloads
11:27:34-087370 INFO     Extension: script='extensions\sd-webui-controlnet\scripts\controlnet.py' Warning: ControlNet
                         failed to load SGM - will use LDM instead.
11:27:34-097225 INFO     Extension: script='extensions\sd-webui-controlnet\scripts\hook.py' Warning: ControlNet failed
                         to load SGM - will use LDM instead.
11:27:34-148813 INFO     UI theme: type=Standard name="black-orange"
11:27:38-800796 INFO     Local URL: http://127.0.0.1:7860/
11:27:38-801883 INFO     API Docs: http://127.0.0.1:7860/docs
11:27:38-802887 INFO     API ReDocs: http://127.0.0.1:7860/redocs
11:27:38-940984 INFO     [AgentScheduler] Task queue is empty
11:27:38-942052 INFO     [AgentScheduler] Registering APIs
11:27:39-065692 INFO     Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16
                         context=no_grad fp16=True bf16=None optimization=Scaled-Dot-Product

Config.json for outdated system:

{
	"ad_bbox_sortby": "Area (large to small)",
	"ad_only_seleted_scripts": false,
	"ckpt_dir": "F:\\stable-diffusion\\models\\Stable-diffusion",
	"clip_skip": 1,
	"code_former_weight": 0.9,
	"control_net_allow_script_control": true,
	"cross_attention_options": [
		"SDP disable memory attention"
	],
	"disabled_extensions": [
		"sd-webui-regional-prompter"
	],
	"dp_ignore_whitespace": true,
	"dp_write_raw_template": true,
	"embeddings_dir": "F:\\stable-diffusion\\models\\embeddings",
	"extra_networks": [
		"Style",
		"Embedding",
		"Lora"
	],
	"extra_networks_sort": "Date [Newest]",
	"extra_networks_view": "list",
	"facehires_strength": 0.4,
	"image_browser_ranking_pnginfo": true,
	"interrogate_clip_skip_categories": [
		"artists",
		"movements",
		"flavors"
	],
	"keep_incomplete": false,
	"lora_dir": "F:\\stable-diffusion\\models\\Lora",
	"multiple_tqdm": true,
	"postprocessing_enable_in_main_ui": [],
	"postprocessing_operation_order": [],
	"queue_keyboard_shortcut": "Ctrl+Disabled+KeyE",
	"quicksettings_list": [
		"sd_model_checkpoint",
		"sd_vae",
		"directories_filename_pattern",
		"img2img_fix_steps",
		"upscaler_for_img2img"
	],
	"samples_filename_pattern": "[datetime]_[seed]_[prompt]",
	"save_images_add_number": false,
	"save_images_before_face_restoration": true,
	"save_to_dirs": true,
	"schedulers_brownian_noise": false,
	"schedulers_sigma": "karras",
	"sd_backend": "original",
	"sd_checkpoint_hash": "ec41bd2a8271acde4ae81cac004d9f3300e7fb0870eae8cbfe0bbc4ef8e27f91",
	"sd_disable_ckpt": true,
	"sd_model_checkpoint": "photon_v1 [ec41bd2a82]",
	"sd_vae": "vae-ft-mse-840000-ema-pruned2.safetensors",
	"send_size": false,
	"show_progress_type": "TAESD",
	"show_samplers": [
		"DPM++ 2M",
		"DPM++ 2M SDE",
		"Euler a",
		"DPM++ SDE",
		"DPM SDE"
	],
	"styles_dir": "models\\styles",
	"uni_pc_lower_order_final": true,
	"uni_pc_order": 2,
	"upscaler_for_img2img": "ESRGAN 4x NMKD Superscale",
	"vae_dir": "F:\\stable-diffusion\\models\\VAE",
	"wildcards_enabled": false
}

Config for current version:

{
	"ad_bbox_sortby": "Area (large to small)",
	"ckpt_dir": "F:\\stable-diffusion\\models\\Stable-diffusion",
	"clip_skip": 1,
	"control_net_allow_script_control": true,
	"cuda_compile": [],
	"diffusers_offload_max_gpu_memory": 13,
	"disabled_extensions": [
		"sdnext-modernui",
		"sd-webui-prevent-artifact"
	],
	"dp_ignore_whitespace": true,
	"dp_write_raw_template": true,
	"embeddings_dir": "F:\\stable-diffusion\\models\\embeddings",
	"extra_networks": [
		"Style",
		"Embedding",
		"Lora"
	],
	"extra_networks_sort": "Date [Newest]",
	"gradio_theme": "black-orange",
	"interrogate_clip_skip_categories": [
		"artists",
		"movements",
		"flavors"
	],
	"ipex_optimize": [],
	"keep_incomplete": false,
	"lora_dir": "F:\\stable-diffusion\\models\\Lora",
	"multiple_tqdm": false,
	"nncf_compress_weights": [],
	"nncf_quantize": [],
	"openvino_devices": [],
	"postprocessing_enable_in_main_ui": [],
	"postprocessing_operation_order": [],
	"quicksettings_list": [
		"sd_model_checkpoint",
		"diffusers_pipeline",
		"sd_backend",
		"sd_vae"
	],
	"samples_filename_pattern": "[datetime]_[seed]_[prompt]",
	"save_to_dirs": true,
	"schedulers_brownian_noise": false,
	"schedulers_sigma": "karras",
	"sd_checkpoint_hash": "ec41bd2a8271acde4ae81cac004d9f3300e7fb0870eae8cbfe0bbc4ef8e27f91",
	"sd_model_checkpoint": "photon_v1 [ec41bd2a82]",
	"sd_vae": "vae-ft-mse-840000-ema-pruned2.safetensors",
	"sdp_options": [
		"Flash attention",
		"Memory attention",
		"Math attention"
	],
	"show_samplers": [],
	"styles_dir": "models\\styles",
	"tac_maxResults": 10,
	"tac_showWikiLinks": true,
	"uni_pc_lower_order_final": true,
	"uni_pc_order": 2,
	"vae_dir": "F:\\stable-diffusion\\models\\VAE",
	"xformers_options": [
		"Flash attention"
	]
}

Any help would be much appreciated!

[vladmandic]

first, there was actually a problem with vae decode in backend original causing washed out output, i just fixed that in dev branch.
second, your prompt is extremely unbalanced and is causing burn. i spoke about that many times and just the face that "it works in a1111" doesn't mean thats how prompting is supposed to be done.
its ok to use prompt attention, but when you heavily skew to one side like you did, burn happens.

i just removed :2 attention from your negative prompt and image in both backends now looks good.
and yes, original and diffuser backends do produce different images - they are simply different.

anyhow, attached are images using your settings for both backends:

[anothertal3]

First of all...

You are as quick and helpful as always. Many thanks to you!

I also thank you for your examples.

Regarding the "original" backend

If I understand you correctly, "original" had an issue with VAE and will be working as expected when the change gets merged into master. That's great news!

Regarding the "diffusers" backend

Sorry, if this has been discussed someplace else. I wasn’t aware that my issue is related.

I tested your suggestion using an xyz-chart and can confirm that it breaks very quickly after a certain limit. From my observation, the attention is somewhat compressed in range in “diffusers” mode. The sweet spot seems to be ~ “:1.4” instead now, instead of “:2”.

To be honest: this really surprises me. I know that there are differences between the output of "original" and "diffusers" but I always expected it to only influence the resulting composition. I never expected the quality to deteriorate so greatly when using the same prompt.

My example was nothing special, but it did use the recommended "recipe" of the model author for the negative prompt. I'm worried that fiddling with new models will prove more complicated in “diffusers” mode now.

Summary

I think I will wait for the patch you’ve mentioned and then switch over the latest version. I'll probably keep using the “original” mode to keep my prompts comparable to online references until SD1.5 has become completely irrelevant. Everything can already use "diffusers" backend.

[vladmandic]

yes, you understood correctly.

regarding prompt attention range, you do have option in settings -> execution -> prompt attention normalization
its not enabled by default, but you can try enabling it