ArrayMeshes converted from Godot 3.x display incorrectly in 4.0

[TokisanGames]

Godot version

4.0alpha 12-15

System information

Win 10/64 NVIDIA GeForce GTX 1060/PCIe/SSE2

Issue description

I've converted over a thousand meshes to GD4 for my game. Countless objects respond to light as if their faces have inverted normals. However, that isn't the case as demonstrated by exporting from Godot and viewing in blender. There's nothing wrong with the normalmap. Inverting it does not help.

Maybe 50% 100% of them are messed up! Hundreds of assets. They previously looked the same as all the other pieces in GD3. I have assets from the same asset pack, the same artist, that were imported using the same version of Godot, through the same process, around the same time, and even using the same material, yet some pieces from the pack are messed up and others are fine.

I also have other objects that flicker and invert depending on camera angle, though it's the same wood material that everything in the village uses. I'm still testing this. It may be another issue.

We imported our meshes with the default settings in GD3 and saved them in Godot's native arraymesh format in tscn files so git isn't overwhelmed by minor updates on binary files. They were converted to GD4 using aaronfranke's script https://gist.github.com/aaronfranke/79b424226475d277d0035b7835b09c5f, which basically just renames poolarray to packedarray and some other renames. The array data should have remained the same until modified by Godot.

The only way I've found to fix this so far is to reimport from the original fbx/glb. But we're talking over 1000 assets need to be audited, hundreds of assets that need to be reexported from our source again, processed through blender, reimported into Godot, then setup in our complex master scenes (eg static bodies, collision shapes, custom materials, LOD, and other scripts). That's several weeks of work if not a couple months.

Examples:

Foot bridge

Here's a bridge with an empty standard material, albedo at 50V. You can see the sun is at X=-30 and the bridge is lit up normally.

Now the sun is at X=-150 (30 degrees off of 0). The top of the bridge should be lit up exactly the same as above. Look at the sphere and the ground.

I have to rotate it this far (-120 or 50 degrees off 0) to get the top of the bridge to light up when the rotation is on that side.

In addition to this strange light angle needed for normal light, the light changes depending on camera angle. It reminds me of this issue #58592 as the brightness of materials shifts a bit when I move the camera around, though it is much more subtle than that issue.

Beams

These two objects are using the exact same material. You can see the light wood on the top faces of the left beam. But the side faces are all dark. They should be as light a the right piece. They are from the same asset pack and were imported the same way.

Standard white material

Exporting the beams from Godot to blender, the colors look messed up there too. However, the normals point outside. If I recalculate normals, it messes up the mesh even worse. The faces are likely separate.

If I bring in the original asset into Godot 4, it looks completely fine. So it appears Godot 4 messed up the arraymesh, requiring reimporting the asset to fix it.

Minimal reproduction project

test_gd4_mesh.zip
(Updated 8/1)

[TokisanGames]

Yikes. I'm going to have to look at every single face on every single object to audit this.

Maybe 5% so far are messed up

Maybe higher. I keep running into them.

[Calinou]

Note that unlike Godot 3.x, Godot 4.0 doesn't have mesh octahedral compression on import yet.

[TokisanGames]

I looked at the beam and footbridge assets after converting through the latest --convert-3to4 in #63672, but the meshes are no different than shown here.

[TokisanGames]

I'm continuing to find more and more messed up assets. I have a bunch of stumps that look like their normals are inverted. Here's GD4 on the left, GD3 on the right, same light and camera angle.

If I invert the normalmap in GD4 it doesn't fix the problem. It's not a normal map issue.

[TokisanGames]

I've finished converting all of my static assets. Trees, plants, rocks, dungeon pieces, village pieces, weapons, shields. Around 800 individual assets, and many have 2-3 LODs, so maybe 1600-2000 meshes, and I haven't started on my creatures or humans yet.

The number of broken meshes is huge, around 50% of them. Hundreds of assets! As noted above, some meshes may only be a few faces, so I have to audit every angle on every mesh, and every LOD. We're talking several weeks for me to manually audit and reimport everything. There must be a fix in the engine that can be done, as these are the same arraymeshes GD3 was happy with.

Reconverting the original gd3 to gd4 meshes is no problem for me, as I have git and the scripts to do it in a few days. Getting all of the original assets out of UE/Unity/Sketchfab exported, processed through Blender, then into Godot again. Weeks and weeks of work.

I updated the MRP, and added three scenes:

• Beam-gd3_unconverted.tscn for the mesh right out of gd3 - 36kb
• Beam-gd3_converted.tscn for the renamed attributes, then resaved by Godot4 - 43kb
• Beam-gd4_imported.tscn for a freshly imported version w/o shadowmesh or autolod - 15kb

The scenes are mostly identical except for transform and the mesh data. Note the differences in sizes! Running a diff on them shows:

• gd3 orig vs gd3 converted - same vertex count. Vertex_data starts off similar, then diverges
• gd3 converted vs gd4 import - vertex count 274 (gd3) vs 75. vertex_data and attribute_data are completely different.

[Zireael07]

Does resaving fix the problem or do those resaved meshes have issues?

(TBH I've migrated a fairly big GD3 project to GD4 and have yet to see an issue, though I have like 5 or so meshes actually imported and the rest is procgen in editor ;) )

[TokisanGames]

Does resaving fix the problem or do those resaved meshes have issues?

Loading the renamed attributes GD3 asset into Godot 4 they are messed up. Saving them and reopening the scene, they are still messed up. Exporting to blender, the mesh still looks and acts weird, even though the normals are on the outside.

(TBH I've migrated a fairly big GD3 project to GD4 and have yet to see an issue, though I have like 5 or so meshes actually imported

5 meshes I could manually reimport without complaint. 400 meshes, or maybe 800 w/ 2-3 lods is a big task. There's clearly a bug in the engine here.

[Zireael07]

Another thought, I don't use LODs at all, my meshes being very low-poly. Are all of your affected meshes using LODs?

[TokisanGames]

The custom LODs are not built into the meshes as they are in GD4's autolods. They are separate MeshInstances and separate ArrayMeshes with their own array data in the scene files. We have the same structure in blender, which is also how it comes from UE/Unity/sketchfab: A parent node, and the LOD children as separate meshes. The whole package gets imported into Godot. Then it gets disassociated from the GLB so we can work with it, combine and adjust materials, set up additional nodes, and save materials as shaders, and meshes as text. The script on RootNode handles turning on or off LODs and managing shadow impostors.

Here I have made both meshes visible and rotated one so you can see both LOD meshes are messed up.

Here's an object that came in with no additional LOD meshes. So custom LODs is not the cause. I have assets from so many sources, so that's not it. Even assets from the same pack have some that are fine, and others that are messed up like the Beams above.

[TokisanGames]

Animated meshes have been completely destroyed. Here's what they look like. All animations on all my objects look crumpled like this. No movement of bones in the animation.

Can you guess what this is?

Here is the reimported mesh:

One more. What is this?

Did you get it right?

[TokisanGames]

Now with #64854 and octehedral compression enabled by default, 100% of my meshes are messed up in Alpha 15. All of my old ones, and all of the new ones we already reimported into earlier versions of GD4, such as all of the animals above.

[djmaesen]

alpha 15 messed up all my models

[TokisanGames]

@djmaesen Your issue is likely caused by #64854 specifically, which is octehedral compression. This issue is more for GD3 formatted meshes that were converted over to GD4. It looks like your meshes were fine in GD4 already. However with Alpha 15, the issues have gotten a bit blended because all meshes are messed up now.

[lyuma]

Confirmed MRP on 4.0-rc2 with the beams/Beam-gd3_unconverted.tscn example scene. This scene has SpatialMaterial and MeshInstance so came from Godot 3, but seems to have lost normals.

Beam example (Beam-gd3_unconverted.tscn vs Beam-original.glb):

WARNING: Mesh uses old surface format, which is deprecated (and loads slower). Consider re-importing or re-saving the scene. Path: "res://beams/Beam-gd3_unconverted.tscn::1"
     at: (scene/resources/mesh.cpp:1148)

For footbridge, it was already improperly converted during an earlier alpha so it probably cannot be fixed.

beam mesh data:

[sub_resource type="ArrayMesh" id=1]
blend_shape_mode = 0
surfaces/0 = {
"aabb": AABB( -0.143955, -1.49708, -0.132141, 0.25617, 4.51391, 0.250194 ),
"array_data": PoolByteArray(  181, 170, 196, 189, 160, 248, 24, 64, 76, 19, 233, 61, 167, 0, 89, 0, 167, 2, 167, 127, 4, 57, 113, 59, 140, 53, 231, 57, 233, 208, 229, 189, 32, 208, 63, 64, 154, 90, 189, 61, 167, 0, 89, ... )
"array_index_data": PoolByteArray( 0, 0, 1, 0, 2, 0, ... )
"blend_shape_data": [  ],
"format": 97591,
"index_count": 282,
"material": SubResource( 2 ),
"name": "Wood_01",
"primitive": 4,
"skeleton_aabb": [  ],
"vertex_count": 274
}

format 97591 is 0x17d37
I think this is FORMAT_VERTEX | FORMAT_NORMAL | FORMAT_TANGENT | FORMAT_UV | FORMAT_UV2 | FORMAT_INDEX | COMPRESS_NORMAL | COMPRESS_TANGENT | COMPRESS_COLOR | COMPRESS_TEX_UV | COMPRESS_TEX_UV2 | COMPRESS_WEIGHTS

I don't know if this means octahedral compression is enabled or not, but my guess is it's the same issue as #64854

The UVs and indices and vertex positions are all correct, so it's only an issue with the normals and/or tangents.

[wilfredjonathanjames]

I had this same issue when importing from 3.2 to 4.0.3: #78502

[QbieShay]

Possibly related #84079