Discussion about Virtualized Geometry (as introduced by UE5's Nanite)

[tajo48]

• So I guess most people have already taken a look at UE5's impressive new features by now. Now, I know that UE has a very different focus (and a amount of resources) than Bevy.
  Nevertheless, it is never a bad idea to draw some inspiration from other technology for Bevy's long-term future. So maybe this is a good place for discussion, even if implementing similar features in Bevy is not realistic at this point.
  One of the big new advertised features of UE5 is Virtualized Geometry (called Nanite)
  https://docs.unrealengine.com/5.0/en-US/RenderingFeatures/Nanite/
• These are the advertised benefits of this technology:
  • Multiple orders of magnitude increase in geometry complexity, higher triangle and objects counts than has been possible before in real-time
  • Frame budgets are no longer constrained by polycounts, draw calls, and mesh memory usage
  • Now possible to directly import film-quality source arts, such as ZBrush sculpts and lower quality/slightly tuned photogrammetry scans
  • Use high-poly detailing rather than baking detail into normal map textures
  • Level of Detail (LOD) is automatically handled and no longer requires manual setup for individual mesh's LODs
  • Loss of quality is rare or non-existent, especially with LOD transitions
• While the increase in visual quality of games using this technology in the future is immediately obvious, the more interesting fact is, that developement time for 3d assets might be significantly decreased, since the usual tedious process of optimizing and tweaking models will mostly disappear (at least this is what this technology promises)
  This sounds like a big deal for small teams or solo developers, which makes it in turn potentially very interesting for Bevy's target audience.
• This discussion could be added in 3D Graphics Tracker alongside Mesh LOD Support as they are representing different approach and aim to solve different issues while virtualized geometry will probably introduce even more issues such as how to handle animation of meshes crated with virtualized geometry and compatibility with hardware.
• This issue from godot's repo could be used as inspiration or source of conclusions for discussion about:
  • State of Vulcan and device readiness for VK_EXT_mesh_shader on gpuinfo.org
  • What model should we use software vs hardware rasterizers
• In case of going with purely meshlets design there is this pr that probably needs a little more attention
• If you and other devs aren't hooked yet by this idea i think what 2plus2makes5 in this godot's issue said would change your mind

  • Unless there's a huge catch or drawback Nanite is clearly the future of 3d graphics, Unity and other 3d engines won't lag behind and will implement their own version, if that will be the case like it or not Godot will have to do the same, not immediately because we still know almost nothing about Nanite, especially the drawbacks, but in a not too distant future.

• Also it would probably mean that those technologies should be merged:
  • Virtual Textures mentioned 3D Graphics Tracker
  • Asset Streaming from Add Support for Asset Streaming mentioned in Bevy Asset V2 Tracking Issue
• I have stolen 99% of this topic from one of godot proposals by novemberist.
• This discussion is also made for wanderers who are interested in exploring Bevy but are hesitant due to the lack of support or even an announcement in the roadmap for features like Nanite. Seeing that the engine is working on it could encourage them to stay.

[JMS55]

I've been actively working on the meshlet PR. A couple quick notes:

• Meshlets + visibility buffer + two pass occlusion culling + gpu-driven rendering gives you 60-70% of Nanite's benefits. That PR should already give a large order of magnitude performance improvement over using bevy's current renderer, and support some fairly dense meshes.
• Mesh shaders are not required, only indirect indexed draws. Wgpu/naga doesn't have support for mesh shaders yet, but once that's added, it should be pretty easy to implement for supported platforms for improved performance, with the existing method being a fallback. Not a blocker.
• Traditional LODs, compression, and streaming can all be added as additional optimizations later. We're not going to have them right off the bat, but they can be added in a future release.

And then there's what Nanite enables, which is near unlimited micro-poly geometry. This involves a lot more work. Not only is streaming and compression necessary right off the bat, but the LOD scheme is extremely complex, and you have to write a custom rasterizer in compute shaders. This is not at all an easy task. Furthermore, not many developers, much less indie developers (which is Bevy's main audience atm), are even equipped with the asset pipeline to be able to use this kind of feature.

So TLDR; #10164 will already give a large improvement in things along the direction of Nanite, but fully implementing Nanite is way out of scope any time soon.

[tajo48]

@JMS55 Really good points and great job, I mentioned #10164 in this line as i think it should get more attention in a long run:

In case of going with purely meshlets design there is this pr that probably needs a little more attention

And I'm glad for it as it basically solves poly count issue for a next couple of years!
However its still worth pointing out that it should be announced to the community in some way or another.
And perhaps in far future it could be cool place to start planning on how to implement more complex solution with support for "near unlimited micro-poly geometry".
Any way...
Big luv for your work <3