Implement Feature Build Profiles

[reduz]

This PR is a continuation of #50381 (which was implemented exactly a year ago!). I was too busy to finish this, but I really wanted it before Beta so it does not have to wait for 4.1.

• Add a visual interface to select which classes should not be built into Godot (well, they are built if something else uses them, but if not used the optimizer will remove them out).
• Add a detection system to scan the project and figure out the actual classes used.
• Added the ability for SCons to load build profiles.

Obligatory Screen:

Usage:

$ scons p=windows target=release tools=no build_feature_profile="soso.build"

A simple test with a couple of nodes in the scene resulted in a 25% reduction for the final binary size. With the ability to disable more core parts of the engine, gains should be considerably even more dramatic.

TODO:

• Script languages need to implement used class detection (left for another PR).
• Options to disable servers or server functionalities (like 2D or 3D physics, navigation, etc). Are missing, that should also greatly aid in reducing binary size.
• Options to disable some modules would be desired.
• More options to disable drivers (OpenGL, Vulkan, etc) would be desired.
• There is a problem in how NavigationServer was made, because it uses resources in scene/ and this causes link to fail. This needs to be fixed in a separate PR but in the meantime add NavigationMesh and NavigationPolygon to the forced detect field at the bottom.

In general this PR is a starting point for more contributors to improve and enhance this functionality.

[bruvzg]

I would also include PCK encryption key into the same dialog / config file. Currently, it's set using environment variable, which is not a clear way to do it for many users.

• Options to disable servers or server functionalities (like 2D or 3D physics, navigation, etc). Are missing, that should also greatly aid in reducing binary size.
• Options to disable some modules would be desired.

It probably should have separate section for selecting TextServer implementation (dummy, HB/ICU, fallback), control embedding of ICU data and disabling related modules (freetype, graphite, msdfgen). Everything is already customizable with scons command line argument, but will be good to have all in one interface. I'll do a follow-up PR when this one is merged.

[Calinou]

Awesome work 🙂

In a future PR, it could be useful to expose the "detect from project" functionality as a command line argument, so you can run it automatically when using custom export templates for your project on CI on every commit. (That said, if you only intend to recompile your export templates occasionally, it may be better to generate the build profile once on your own machine instead and commit it to version control.)

[reduz]

@Calinou I think it should be possible to do this without much effort, so you pass the .build file and it updates it.

[jcostello]

Is there any performance gain in the editor or in the game build when disabling features?

[Calinou]

Is there any performance gain in the editor or in the game build when disabling features?

No, run-time performance is expected to remain identical.

[reduz]

@jcostello nope

[neikeq]

If we have automatic usage detection, wouldn't it be better to disable everything by default? If for some reason this is not the default, should we expect any issues from disabling everything manually to delegate it completely to the auto-detector?

• Script languages need to implement used class detection (left for another PR).

How would this work with APIs like ClassDB where the input may only be known at runtime? Are they just considered as not supported during detection?

[reduz]

@neikeq auto-detect disables what is not used already by default. If you use ClassDB manually and somehow don't use the class, then it will not auto detect it. Put the classes you use in the bottom text edit so they are force detected.

[akien-mga]

We don't have these options yet, is that placeholders for future work?
We do have disable_advancedgui` though.

[reduz]

Yeah they are placeholders, so eventually they need to be implemented.

[reduz]

Disable advanced gui kind of loses making much sense with this and should likely be removed as this feature does everything its supposed to do.

[akien-mga]

Looks great!

[akien-mga]

Thanks!

[bfelbo]

Is there any performance gain in the editor or in the game build when disabling features?

No, run-time performance is expected to remain identical.

What about load times? We're seeing 10s+ load times for HTML5 games on old devices and it'd be really amazing if this reduced it 🤞

Awesome work by the way 🙌

[Calinou]

What about load times? We're seeing 10s+ load times for HTML5 games on old devices and it'd be really amazing if this reduced it crossed_fingers

Load times will probably not improve by more than 10% or so, even with nearly all features disabled. Unfortunately, I don't know any other low-hanging fruits to reduce the WebAssembly bundle size.

Remember that Godot is asking a lot from the browser, especially on old/low-end devices. Loading a WebAssembly payload that is 10+ MB is not an easy task for the browser. For old/low-end devices, native ports will always be a better experience.

[bfelbo]

True, it is really impressive that Godot even runs in the browser on those old mobile devices.

Load times will probably not improve by more than 10% or so, even with nearly all features disabled. Unfortunately, I don't know any other low-hanging fruits to reduce the WebAssembly bundle size.

Interesting. What makes up the bulk of the WebAssembly bundle size if it's not features?

[aaronfranke]

@bfelbo It is features. But disabling a feature in a build profile does not necessarily mean it will be not compiled. With the feature build profile system introduced in this PR, what it does is cause the classes to not be registered in ClassDB. Any code that isn't registered or referred to anywhere will be removed by the linker.

However, a lot of the code will still be referred to anyway. For example, a Godot game must have Viewport, and Viewport depends on the Camera2D and Camera3D nodes. Even if those nodes are not registered with ClassDB, their code will still be included because Viewport needs those classes.

@reduz summarizes this well in his post:

Add a visual interface to select which classes should not be built into Godot (well, they are built if something else uses them, but if not used the optimizer will remove them out).

If you want to disable more of the engine, a more thorough approach than just not registering classes is needed. I have a PR that allows disabling 2D nodes, and I have another PR for moving 2D and 3D resources to subfolders so that they can be more easily disabled, but these have not been merged yet. To be clear, the ideal solution is a combination of this PR's approach of not registering classes, and my PRs for forcefully disabling parts of the engine.

[h0lley]

little feedback report, I tried this the other day and was able to reduce my ~57 mb master branch binary (release, linux 64 bit) to 40 mb. so that's really nice I think.

I did took me quite a while though to manually compile a list of classes my project uses that the auto detect feature was not able to detect. it's probably not something that can be expected of the average user.

also, is there any chance of this being backported to 3.x? for people wanting to optimize for size, this would be especially interesting to use with 3.x as it compounds with the already significantly smaller code base size.

for instance, if Godot 3.x could rival Flutter in wasm size (~7mb) that would be pretty cool.
without feature build profiles I get 3.x wasm down to ~10 mb currently.

[aaronfranke]

@h0lley I don't think Godot 4 will be able to go down to 7 MB. My guess is we can get it down to 20 MB if we combine this PR with my other PRs and disabling modules and likely some more additional work.

https://docs.google.com/spreadsheets/d/1aWmWiA6MWoHr4Mgg22tybDFPa6BJX5oHTSjfN21plyY/edit#gid=0

Here is a spreadsheet I made over a year ago, it should still be roughly accurate. A lot of the size is from the core folder, and that code can't / shouldn't be removed. I'm curious to see what the results would be if we recreated this spreadsheet with as many changes as we can to reduce the size, to see what's left as the biggest pieces.

Anyway, this is getting off-topic for this PR. Feel free to contact me elsewhere and we can work on this together.

[Calinou]

also, is there any chance of this being backported to 3.x? for people wanting to optimize for size, this would be especially interesting to use with 3.x as it compounds with the already significantly smaller code base size.

This relies on a lot of backwards-incompatible changes, so I'm afraid not.

[neikeq]

I did took me quite a while though to manually compile a list of classes my project uses that the auto detect feature was not able to detect. it's probably not something that can be expected of the average user.

I think you should open an issue if the auto-detect feature fails to detect a class you use. Unless the classes are accessed dynamically (via ClassDB::instantiate(class_name) for example); I don't think we can do anything about those.

[h0lley]

I think you should open an issue if the auto-detect feature fails to detect a class you use. Unless the classes are accessed dynamically (via ClassDB::instantiate(class_name) for example); I don't think we can do anything about those.

I believe only classes referenced in tscn and tres are detected atm, and nothing that's only in script.
but this is presumably due to this still missing:

Script languages need to implement used class detection (left for another PR).

so at least reduz is already aware.

via ClassDB::instantiate(class_name) for example

right, having to add one or two classes in those very niche use-cases to the "forced classes on detect" field is by no means asked too much. for reference though, here's what I had to specify manually:

"OS,Engine,MainLoop,SceneTree,DisplayServer,AudioServer,ProjectSettings,ResourceLoader,InputMap,Input,Time,Directory,File,ConfigFile,Thread,ResourceFormatLoader,RandomNumberGenerator,SceneTreeTimer,Tween,CallbackTweener,IntervalTweener,MethodTweener,PropertyTweener,RegEx,RegExMatch,AudioEffectSpectrumAnalyzerInstance,InputEventAction,InputEventKey,InputEventJoypadButton,InputEventJoypadMotion,AudioBusLayout,World2D,RichTextEffect,StyleBoxLine,ButtonGroup,ViewportTexture,ConfirmationDialog,ParallaxLayer,CheckButton,PopupMenu,TabBar,Skeleton2D,Bone2D"

so yea, according to current auto detection apparently my project doesn't use MainLoop, for instance 😅

[dustdfg]

Is there any performance gain in the editor or in the game build when disabling features?

No, run-time performance is expected to remain identical.

What about nowadays? :)

[h0lley]

What about nowadays? :)

you can use this to make very slim builds in terms of size, not so much for performance.