Mesh compression breaks normal

[atirut-w]

Godot version

v4.2.beta.custom_build [ee118e7]

System information

Godot v4.2.beta (ee118e7) - Windows 10.0.22621 - Vulkan (Forward+) - dedicated AMD Radeon RX 6700 XT (Advanced Micro Devices, Inc.; 27.20.21034.37) - AMD Ryzen 7 5700X 8-Core Processor (16 Threads)

Issue description

When mesh compression is enabled (default), the normals of a mesh becomes corrupted. Most likely introduced by #81138 (needs verification). May be related to #82890. Seemed to start appearing after I overrode light materials.

With compression:

Without compression:

Steps to reproduce

(See MRP)

1. Check normal buffer.
2. Go to the import settings of dm3_open_sky.obj and disable Force Disable Compression.
3. Check normal buffer again.

Minimal reproduction project

UV2 Compression.zip

[atirut-w]

cc @clayjohn, I think?

[bitsawer]

Probably caused by the new mesh compresssion as suggested. Might be a duplicate of #83236, but I'll leave this open for now in case the issues have different causes. This issue also has a proper test project which can be useful.

[clayjohn]

I can't reproduce on master (51f81e1) or on beta1. So indeed, may be related to #82890. (as it seems I can't repro that one either).

Can you try importing your scene again but with "ensure tangents" disabled?

Also, can you please let me know what settings you used to build the engine?

[bitsawer]

Tested on current master: 51f81e1

edit: Disabling Ensure tangents also seems to fix this.

Godot built using llvm-mingw on Windows 10, command line scons -j7 platform=windows target=editor dev_build=yes dev_mode=yes use_mingw=yes use_llvm=yes linker=lld compiledb=yes fast_unsafe=yes optimize=none tests=yes verbose=no CCFLAGS=-Wno-ignored-attributes (generated command line, so there are some redundancies)

Not compressed:

Compressed (note the light differences on some 90-degree surfaces, especially the large building):

I can't really use normal debug view, for some reason that project makes it looks garbled. Seems to work okay on some other projects. Looks like the normal debug view issue is tracked in #62955, disabling MSAA fixes it.

[bitsawer]

Master build (51f81e1) with UBSAN enabled gives these warnings, which seem to match the tangent conversion in RenderingServer::_surface_set_data():

servers\rendering_server.cpp:501:21: runtime error: 3.34993e-312 is outside the range of representable values of type ''
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior servers\rendering_server.cpp:501:21 in
servers\rendering_server.cpp:502:21: runtime error: 3.34993e-312 is outside the range of representable values of type ''
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior servers\rendering_server.cpp:502:21 in
servers\rendering_server.cpp:512:21: runtime error: 3.34993e-312 is outside the range of representable values of type ''
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior servers\rendering_server.cpp:512:21 in
servers\rendering_server.cpp:514:21: runtime error: 3.34993e-312 is outside the range of representable values of type ''
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior servers\rendering_server.cpp:514:21 in
servers\rendering_server.cpp:513:21: runtime error: 3.34993e-312 is outside the range of representable values of type ''
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior servers\rendering_server.cpp:513:21 in

[clayjohn]

Perhaps we need to clamp res to the 0-1 range?

edit: nevermind, 3.34993e-312 is within the 0-1 range, but it is much too small for a float

Hmmm 3.34993e-312 is smaller than what even double precision can represent

[bitsawer]

Hard to say much about the UBSAN errors or the weird float value, could even be some clang runtime formatting issue.

I noticed another problem with the test project (again on 51f81e1)

Download and unzip the MRP, delete .godot folder if it exists. Then open the project and double click dm3_open_sky.obj in the FileSystem dock. Godot UI hangs and starts spamming this error repeatedly to console:

ERROR: Can't use compression flag 'ARRAY_FLAG_COMPRESS_ATTRIBUTES' while using normals without tangent array.
   at: (servers\rendering_server.cpp:1189)
ERROR: Condition "err != OK" is true.
   at: add_surface_from_arrays (scene\resources\mesh.cpp:1785)
ERROR: Index p_idx = -1 is out of bounds (surfaces.size() = 0).
   at: surface_set_material (scene\resources\mesh.cpp:1906)

[clayjohn]

Hmmm that's a good lead. Perhaps somehow we are getting a tangent array with uninitialized memory through

[clayjohn]

One more data point:

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior servers/rendering_server.cpp:512:21 in 
servers/rendering_server.cpp:332:39: runtime error: division by zero
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior servers/rendering_server.cpp:332:39 in 
servers/rendering_server.cpp:333:38: runtime error: division by zero
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior servers/rendering_server.cpp:333:38 in 
servers/rendering_server.cpp:334:38: runtime error: division by zero

godot/servers/rendering_server.cpp

Lines 332 to 334 in 51f81e1

	r_axis.x = (p_normal.y - binormal.z) / denom;
	r_axis.y = (tangent.z - p_normal.x) / denom;
	r_axis.z = (binormal.x - tangent.y) / denom;

Ah, looks like they are NaNs. That's why the value appeared random for you

[clayjohn]

This fixes the division by zero when denom is 0 (I just removed it entirely because we normalize right after anyway) and adds a few other things 4f3eddd

We can still get NaNs from the normalization for some reason. I still have more to investigate, so not opening a PR yet