GLES3 - Depth sorting problem with overlapping transparent textures

[kubajz22]

Issue description:
Hi, I'm using Godot 3.2.3, GLES3 and I noticed that when you have multiple models with materials marked as Transparent and you add to the scene a source of light that can cast shadows, this will happen: https://youtu.be/2uhuNK9D7mg . As you can see, those leaves that are further away from the camera start to appear in front of the closer ones. I colored them red so it's really obvious. This only happens when there are multiple transparent objects with opaque pre-pass enabled and they are lit using a light source that casts shadows. The "Do not receive shadows" flag doesn't make a difference though. This issue is present on version 3.2.2 as well. When using the GLES2 renderer it behaves in the opposite way, rendering farther objects in front of closer ones when the depth draw mode is set to Opaque only.

Godot version:
3.2.3-stable

OS/device including version:
GPU: nVidia GeForce 1650 Mobile with nvidia-driver-450.80.02-0ubuntu0.20.04.2 driver
CPU: Intel core i5 9300H
OS: Linux Mint 20

Steps to reproduce:
Set up a scene with multiple overlapping objects which are configured as follows:
-Transparent flag enabled
-Depth draw set to Opaque pre-pass
-No alpha scissoring
Then add a directional light and enable shadows.

Example project:
Depth_sorting_issue.zip

[clayjohn]

Do you mind uploading the MRP as a zip file? You can just drag and drop it into the comment.

The link you provided has enough sketchy popups that I am uncomfortable downloading from it.

[kubajz22]

Done. Didn't see those due to my overpowered adblock.

[volzhs]

Depth_sorting_issue.zip
I made MRP smaller to 4.3MB

[EzraT]

Can confirm, same thing happens on my system using Godot 3.2.3
Linux Mint 20 x86_64
Intel i7-2600K
NVIDIA GeForce GTX 1060 6GB, driver 450.80.02

Also tested it on a Windows 10 system for good measure, same thing happens.
AMD Ryzen 7 4800HS
NVIDIA GeForce GTX 1660ti 6GB

[Calinou]

Related to #36669.

[kubajz22]

Happens on Mesa Intel UHD Graphics 630 as well so it probably isn't a nVidia related issue.

[TokisanGames]

The problem is related to the direction of the DirectionalLight and is limited to (caused by?) DirectionalLights. The renderer apparently orders by light direction instead of camera direction.

• Setup: Trees w/ alpha textures for leaves, directional light with shadows. Light comes from the side of the red trees. All are the same tree object, with two different materials. The only difference is the red tint. depth_draw_alpha_prepass enabled, along with alpha in the texture.

• View with the light at our backs. Depth is correct.

• View towards the light. Depth is reversed. All of the trees are showing through each other if at the wrong angle, which is painfully obvious when moving the camera. I'm just distinguishing between red and green to demonstrate through still images.

• If I rotate the whole block 90 degrees, and look at the trees perpendicular to the light, then the part of the camera that faces the light is wrong, and the half that faces away is correct.

• If I disable the directional light and use ambient light, it occludes properly. (not shown)

• Enabling the directional light but disabling shadows occludes properly. (not shown)

• If I put a giant spotlight in the sky, it also occludes properly, however I get horrible shadow bias banding on the flat plane ground. (shown below)

• If I use both the directional light and the spotlight, it incorrectly shows the red trees through the green. (not shown)

• Enabling alpha scissors is a workaround, especially if the default shader is fixed as in Material with opaque prepass enabled, even little alpha, appears invisible if no other object is behind. #36669.

• However alpha scissors is ugly. I'm working with AAA assets, with the transparent pipeline on the right. The left is the best I can make them look with alpha scissors. Yikes. It only looks good close up. Far away and it looks like PS1 style graphics. It makes the 3D mesh look like a low res billboard.

Godot 3.3.2
Win 10/64
Nvidia GTX 1060

[TokisanGames]

More specifically, it is the shadows. It always impacts DirectionalLight, and it temporarily* affects OmniLight and SpotLight.

DL Shadow on/off and light angle determines if it occludes properly.

With DL, none of these things have any effect:

• Unshaded mode
• Unshaded render mode in the material
• Vertex lighting
• Orthographic view
• Directional Light shadow settings
• DirectionalShadow modes or other settings except max distance

---

Reducing DirectionalShadow Max Distance does occlude properly if the distance is set lower than the distance between the camera and the far tree. e.g. 2. Then as you move the camera closer and farther, the back one will overlap or not depending on the distance. When overlapping, you can move the closer tree and it makes no difference. It only depends upon the camera and the far tree distance.

*Temporary effect

• I switched to an OmniLight, then retested SpotLight and they do the same thing. Here I have two trees. The omnilight has a range of around 10-20. If I push the light outside of the lighting range, green always occludes.
• At any angle where the light is on the green tree side of the perpendicular line between the trees, green occludes properly all of the time.
• If I push the omni/spot light back just over that line, the red tree will overlap. However, it only does it while I'm moving the light. If I let go of the mouse button, the green snaps in front of the red. If I hold the mouse down but stop moving, it takes about 1 second, then the green snaps in front of the red. Once it's been corrected, I can rotate the camera around the pair and it always occludes properly.

[Calinou]

However alpha scissors is ugly. I'm working with AAA assets, with the transparent pipeline on the right. The left is the best I can make them look with alpha scissors. Yikes. It only looks good close up. Far away and it looks like PS1 style graphics. It makes the 3D mesh look like a low res billboard.

Alpha hashing, alpha antialiasing alpha to coverage can be implemented to improve the appearance of alpha scissor. These can likely be backported to 3.x (for alpha hashing and alpha antialiasing at least, alpha-to-coverage may or may not be available in GLES).

In general, using alpha blending for vegetation will always be a difficult thing to get right due to sorting issues.

[Zireael07]

In general, using alpha blending for vegetation will always be a difficult thing to get right due to sorting issues.

Unfortunately most free/cheap tree assets or tutorials use alpha scissor or alpha blending for leaves.

[TokisanGames]

In general, using alpha blending for vegetation will always be a difficult thing to get right due to sorting issues.

@Calinou If we couldn't handle "difficult", we wouldn't have chosen to develop in 3D.

I appreciate the workarounds, however I'm interested in fixing this bug so I don't have a shoddy looking game. Other engines have beautiful alpha. Godot also has it with 2/3rds of the light types, and 50% of the directional light. I'm convinced it's a simple bug, though finding it is quite challenging as I'm out of my element in the rendering pipeline.

Here's what I've found so far. After this line:

godot/drivers/gles3/rasterizer_scene_gles3.cpp

Line 4565 in f4e3701

render_list.sort_by_reverse_depth_and_priority(true);

I wrote a function to print out the alpha render_list (last 3 indices of elements). I also print out element->instance->depth.

If DL shadow is disabled, the depth of the trees change, and so the renderlist gets sorted with the farthest away rendered first. I see values like 4-12 as I move the camera.

If DL shadow is enabled, the render list is always green then red, with the light behind the red. The depth of the objects always reports the same negative values. green tree: (-7.72), red tree: (-11.24)!

So that explains why it's sorting wrong half of the time. The depth value doesn't update.

Now why that is, I'm still searching for the cause. I'd appreciate help on this.

[TokisanGames]

Looks like a problem in VisualServer, not the rasterizer. The scene objects already have their distance set (wrong or right) when the visual server calls the rasterizer and passes along the objects in instance_cull_result to render_scene here:

godot/servers/visual/visual_server_scene.cpp

Line 2189 in f4e3701

VSG::scene_render->render_scene(p_cam_transform, p_cam_projection, p_cam_orthogonal, (RasterizerScene::InstanceBase **)instance_cull_result, instance_cull_count, light_instance_cull_result, light_cull_count + directional_light_count, reflection_probe_instance_cull_result, reflection_probe_cull_count, environment, p_shadow_atlas, scenario->reflection_atlas, p_reflection_probe, p_reflection_probe_pass);

Edit: Since the problem is not in the renderer or OpenGL, I should be able to track it down in the next day or two.

[TokisanGames]

In VisualServerScene::_prepare_scene() all object instances calculate and store their distance to the camera in instance->depth.

godot/servers/visual/visual_server_scene.cpp

Line 2015 in f4e3701

ins->depth = near_plane.distance_to(ins->transform.origin);

Directional Lights
For DLs, every frame, shadows are calculated by checking the distance to the DL and storing it in the same instance->depth variable, overwriting it.

godot/servers/visual/visual_server_scene.cpp

Line 2068 in f4e3701

_light_instance_update_shadow(lights_with_shadow[i], p_cam_transform, p_cam_projection, p_cam_orthogonal, p_shadow_atlas, scenario);

The distance to the camera is never recalculated. Instead, the wrong information is passed to the renderer, which sorts incorrectly depending on angle.

OpenGL or other functions in the renderer may be correcting depth sort order for opaque objects, I haven't tested, but it does not do so for alpha objects, which are sorted based upon the information from the visual server.

Omni Lights
For omnis, the exact same thing happens. Instance->depth is calculated relative to the final plane of the shadow cube, overwriting the camera depth. This incorrect information is then passed to the renderer, and for that frame at least, transparent objects are sorted incorrectly depending on angle.

What makes omnis work is that they only update the shadows if the shadows are marked dirty. So the instance depth is wrong for one frame, then corrected every frame after until the shadows are recalculated.

If I tell Omnis to recalculate every frame like DLs, depth is sorted incorrectly as well.

godot/servers/visual/visual_server_scene.cpp

Line 2165 in f4e3701

if (redraw) {

So I see four possible solutions:

1. Initial camera depth is not used before it's overwritten. Move this calculation after shadows are calculated.
2. Recalculate instance depth a second time after shadows are calculated.
3. Calculate shadow depth on a copy or an additional variable so as to not to overwrite the camera depth.
4. Use dirty shadows for DLs also and only update periodically.

1 seems the easiest and least intrusive. I have implemented it and so far everything works. Is this the best solution? I'll write up a PR for review.

Here the camera is facing towards the directional light, with shadows enabled. Front trees properly occlude back trees. Opaque house, terrain, and trunks are properly occluded everywhere, and quality alpha without using alpha scissors anywhere (which also works as any normal opaque object). Omnilights and spotlights also work better as they never occlude improperly

(Image from my game, see my profile.)

[TokisanGames]

I did find one instance where incorrect sort ordering still occurs:

You can see it occurs when one object is within another. Vertex, face, or pixel-based depth rendering would be needed to fix this and the VisualServer only calculates based on entire objects. It doesn't appear that the rendering pipeline even considers this possibility either.

No longer a problem. See below.

[clayjohn]

@tinmanjuggernaut

You can see it occurs when one object is within another. Vertex, face, or pixel-based depth rendering would be needed to fix this and the VisualServer only calculates based on entire objects. It doesn't appear that the rendering pipeline even considers this possibility either.

Indeed, the rendering pipeline only does per-object depth sorting. Per-vertex or per-pixel are leagues more complex and usually require some form of Order Independent Transparency (which comes with its own issues and limitations).

1 seems the easiest and least intrusive. I have implemented it and so far everything works. Is this the best solution? I'll write up a PR for review.

I agree, please make a PR so myself and others can review. :)

Great work by the way!

[TokisanGames]

Indeed, the rendering pipeline only does per-object depth sorting. Per-vertex or per-pixel are leagues more complex and usually require some form of Order Independent Transparency (which comes with its own issues and limitations).

Unlike alpha-scissoring, this is an acceptable tradeoff. A few colliding trees here and there, no one's going to notice. (no more, see below)

Great work by the way!

Thanks! I'm very happy with the results. This bug has been one of the worst problems with the renderer forever.
PR is in and ready.

[TokisanGames]

I was working off of 3.3. Apparently, there was a lot of work done in the 3.x branch on the VisualServer and/or renderer. This branch with my patch now occludes even the overlapping trees shown above!

[akien-mga]

Fixed by #50721.