[Bug]: DAT upscaler is not working in hires fix.

[rltgjqmcpgjadyd]

Checklist

• The issue exists after disabling all extensions
• The issue exists on a clean installation of webui
• The issue is caused by an extension, but I believe it is caused by a bug in the webui
• The issue exists in the current version of the webui
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

DAT upscaler is not working in hires fix.
it is only result in a black square image

Steps to reproduce the problem

1. go to txt2img
2. generate any image & set hires fix. upscaler DAT
3. only result in a black square image

What should have happened?

is DAT upscaler not available in hires fix?

What browsers do you use to access the UI ?

Google Chrome

Sysinfo

sysinfo-2024-01-21-00-50.json

Console logs

WARNING:py.warnings:D:\SDWebUI\modules\upscaler_utils.py:33: RuntimeWarning: invalid value encountered in cast
  arr = arr.round().astype(np.uint8)

Additional information

No response

[Cyberbeing]

@rltgjqmcpgjadyd can you reproduce this with --no-half in COMMANDLINE_ARGS of webui-user.bat? If that avoids the problem, remove --no-half and then try what I've mentioned below.

I ran into a similar issue rarely with Hires upscale and certain prompts/settings (but not Extras upscale) months back, not related specifically to DAT, and was able to resolve it after encountering it more frequently recently with HAT models by disabling autocast when upscaling. For DAT, HAT, and a few others, this should have no negative effects, since Spandrel enforces Float32/BFloat16 only for those model types anyway (of which webui currently only allows Float32).

Assuming you've run into the same problem, you can try applying with devices.without_autocast(): to the line just before here if you want to be very broad (remember to also add from modules import devices at top):

stable-diffusion-webui/modules/images.py

Line 278 in 19c95de

im = upscaler.scaler.upscale(im, scale, upscaler.data_path)

            with devices.without_autocast():
                im = upscaler.scaler.upscale(im, scale, upscaler.data_path)

which may be a good idea just in case autocast is causing other rare minor floating-point errors with slim potential for NaNs in other parts of the upscaling code as well. Since I've encountered this issue rarely even with FP16 ESRGAN upscale models during Hires in webui since 1.5.x or even earlier, I'd say the potential for that is very high. To avoid this, upscaling could be made to use manual casts only.

or alternatively the line just before here:

stable-diffusion-webui/modules/upscaler_utils.py

Line 47 in 19c95de

return torch_bgr_to_pil_image(model(tensor))

but only if you don't care about autocast being partially enabled in other parts of the upscaling code.

The last choice would be to do this for each model format specifically on a case-by-case basis (hat_model.py, dat_model.py, etc), likely just before return upscale_with_model. Though I haven't personally tested this location like the other two.

From what I can tell, the autocast NaN bug occurs sometime after model(tensor) is called during Hires Upscale, only when Float16 autocast is enabled for Stable Diffusion. I assume since Extras Upscale is somewhat separated from from T2I/I2I, it doesn't run into this autocast bug.

It is not so obvious in the current code, but when I tested refactoring to use torchvision.transforms.v2 (ToPILImage, ToTorchTensor, ToDtype) instead of numpy directly and split up the code a bit, the NaN occurred after model(tensor).squeeze(0) if autocast was not disabled before then, which also happened to be the first instance of model(tensor) being called (similar to the line linked above).

[rltgjqmcpgjadyd]

fix after 652a7bb