Use separate resource files for each region

[SlashScreen]

Update by TokisanGames

Separates the monolithic storage resource file into one file per region.

Fixes #356
Fixes #165
Fixes #159
Fixes #419

TODO:

• Get separate region files working for editing, loading, and saving - finished by SlashScreen
• Review updates to Storage
• Adjust the API for more region location focus
• Redesign and reimplement Storage & Region
  • Don't remove region files from disk until saved
• Rework height tracking f/ AABBs
• Rework instancer storage
• Add overlay to show region locations
• Rework undo
  • Undo add/delete region doesn't work
  • Simplify
  • Work with foliage

Test:

• get_texture_id
• edited area AABB
• setting heights

See #433 for API changes

More API changes

Functions

Old Name	New Name
Terrain3DStorage
get_region_location_from_id	get_region_locationi
get_region_id(position)	get_region_idp(position), get_region_id(location)
has_region(position)	has_regionp(position), has_region(location)
get_height_rid	get_height_maps_rid
get_control_rid	get_control_maps_rid
get_color_rid	get_color_maps_rid
set_height_maps	removed
set_control_maps	removed
set_color_maps	removed
set_map_region	removed
get_map_region	removed
set_maps	removed

Enums / Constants

Old Name	New Name
Terrain3DStorage::MapType	Terrain3DRegion::MapType

Future features not in these PRs:

• Expanding region limits from 256 to 2048
• Adjustable region sizes Support more region sizes #77
• Update Update TextureArray indices instead of regenerating the whole array #381

---

Original by SlashScreen

Resolves #356 , and expands terrain limit from 256 to 2048. (admin)

[TokisanGames]

Thanks for working on this.

Here's what I'm thinking we'll do for upgrades:

• Terrain3DRegionManager:Object is the new version of Storage that you're working on.
• Terrain3DStorage:Resource will be a deprecated class with all of the bound functions.
• Storage calls will be redirected to RegionManager so user code won't change. But user's scene files will as the storage resource will stop being saved.
• A Setup Wizard... button above the storage directory will appear that will guide users to select their old data file, and the new directory, so we can split the files.
• In a future version, once users no longer have Storage as a resource saved in their scenes, we will rename RegionManager:Object back to Storage:Object. Once again their code won't change.

So you could get started on a setup wizard.tscn and a button in the inspector. It should include:

• "This version of Terrain3D uses separate storage files; one for each region. Select your old storage file below, and the directory where you'd like save the new files so we can convert your data."
• "File names include the region coordinates for convenience. So an old storage file like: world.res can instead be changed to world/terrain3d_01_03.res, world/terrain3d_01_04.res. Positive coordinates are represented by underscores (_), negatives by hyphens (-)."
• Old Storage resource file: [xxxxx]
• New Storage Directory: [xxxxxx]
• Hide SetupWizard for this project (Project settings)
• Hide SetupWizard for all projects (Editor settings)
• Cancel, OK

Here's are some issues I found while testing:

• Upon loading the node demo, I clicked the file directory, created a new blank folder, and accepted it. The existing regions did not clear. They are still there.

  • Then I saved and Godot crashed. On the filesystem, one file is there terrain_-01-01.res. Which looks like the correct size.

• I removed one of the four regions and saved. All 4 files show in the filesystem panel. However, when I restarted, the region is gone. I see the file is removed in windows. I think you need to unload/unref the resource so godot closes it.

• I removed all regions and it did in the display, but it did not remove one file:
  Terrain3DStorage#1768:remove_region: Failure to remove region file at res://terrain/terrain3d__00-01.res

  • When I restarted, that one file loaded again.

• I added a region, then removed it before saving and got this error:
  Terrain3DStorage#1825:remove_region: Failure to remove region file at res://terrain/terrain3d__06_00.res

• I dragged in the Player and World scenes and can get it to play with 49 regions. 50 and up starts to produce vulkan errors. These might be bugs out of scope for this PR. However it does load them all in the editor, so maybe not.

• I tweaked the importer.gd script (adding: var storage: Terrain3DStorage) and upon importing it reported ERROR: Terrain3DStorage#7944:import_images: Storage not initialized

[SlashScreen]

I removed one of the four regions and saved. All 4 files show in the filesystem panel. However, when I restarted, the region is gone. I see the file is removed in windows. I think you need to unload/unref the resource so godot closes it.

I haven't figured out how to solve this.

I dragged in the Player and World scenes and can get it to play with 49 regions. 50 and up starts to produce vulkan errors. These might be bugs out of scope for this PR. However it does load them all in the editor, so maybe not.

Vulkan is out of my wheelhouse. If it's an issue with me, I may not be able to fix it.

I tweaked the importer.gd script (adding: var storage: Terrain3DStorage) and upon importing it reported ERROR: Terrain3DStorage#7944:import_images: Storage not initialized

That makes sense; I think it's trying to instantiate a terrain storage that's designed to be a singleton, so what you're doing is actually incorrect. Variables should probably be defined as var storage:Terrain3DStorage = null. The real issue is that it should give you that message that you need to get the singleton through Terrain3D. I'm not sure how to do that.

[TokisanGames]

I removed one of the four regions and saved. All 4 files show in the filesystem panel. However, when I restarted, the region is gone. I see the file is removed in windows. I think you need to unload/unref the resource so godot closes it.
I haven't figured out how to solve this.

Are you using force rebuild or clear?

Use debug logging and get the log when removing a region. Then look at the debug log using the main branch when removing a region. What's the difference? Notepad++ or diff can easily compare two files. You should have logging in all of your new functions.

I dragged in the Player and World scenes and can get it to play with 49 regions. 50 and up starts to produce vulkan errors. These might be bugs out of scope for this PR. However it does load them all in the editor, so maybe not.
Vulkan is out of my wheelhouse. If it's an issue with me, I may not be able to fix it.

How many regions can you run the game with on your computer? What about in the main branch? If your branch has no issue, they should be the same.

I tweaked the importer.gd script (adding: var storage: Terrain3DStorage) and upon importing it reported ERROR: Terrain3DStorage#7944:import_images: Storage not initialized
That makes sense; I think it's trying to instantiate a terrain storage that's designed to be a singleton, so what you're doing is actually incorrect. Variables should probably be defined as var storage:Terrain3DStorage = null. The real issue is that it should give you that message that you need to get the singleton through Terrain3D. I'm not sure how to do that.

You're right, adding that one line didn't fix the import script. It also needs something like:

		if not storage:
			#storage = Terrain3DStorage.new()
			storage = get_storage()

You should fix this script in this PR.

you need to get the singleton through Terrain3D

Instance, not singleton. Only Terrain3D can be used as a singleton.

[SlashScreen]

• Fixed the filesystem issue. The reference issue made no sense to me, so I first pursued the simpler avenue of the filesystem simply needing to refresh. Turns out that's what it was.
• Indeed, Godot seems to slow to a crawl and eventually crash when adding a million terrains. Not only that, it crashed my graphics card too, causing it to reboot. I've experienced similar crashes before; It was caused by lots of data being transferred between the GPU and CPU (I think). I think that terrain3d may have swamped the bandwidth. This will have to wait until texture streaming becomes a thing, I think.

[SlashScreen]

My computer can handle about 130 regions before crashing.

[SlashScreen]

The only outstanding things I can think of are:

• There's a TODO in the importer.gd I left because I wasn't sure how to clear the region manager (since before, it simply made a new terrain 3D storage)
• There's still that issue of how many regions can be handled before things start to crash (but my number is different from yours so I'm not sure what's going on)
• Mark Terrain3DStorage class as deprecated

[SlashScreen]

Should clarify that I have now added the wizard.

[TokisanGames]

You've done git merge or used gh desktop sync twice in this PR. You should never do either. It creates a messy commit history, and you have changes in this PR that you didn't write as they came from other people's commits. Those merges need to be dropped and this properly rebased at some point.

[SlashScreen]

You've done git merge or used gh desktop sync twice in this PR. You should never do either. It creates a messy commit history. Those merges need to be dropped and this properly rebased at some point.

When did I do that? I haven't purposefully done either. This branch hasn't interacted with main, or any other branch, at all since rebasing last week, or whenever that was.

[TokisanGames]

Looks like two weeks ago

...

Because of this, if you look at all of the changes for this PR, you can see a lot of changes to terrain_3d_storage.cpp are in there that you didn't make, eg. get_texture_id.

[SlashScreen]

I have no recollection of doing this... nor do I have any idea how to undo it.

[TokisanGames]

@Xtarsia Fixed those issues. Use this PR for now. The other PR is for the conversion wizard, which I haven't started reviewing yet.
My bounds checker was very wrong. a9e846b.
It now rebuilds the region map before iterating through it. c3c13f5
And no longer reloads all storage data every time mesh settings are changed e135e9b

Update
Edited area AABB seems to work with MMI transforms updating when sculpting. However, I discovered it's broken (in main) at vertex spacing >=2. But Terrain3DObjects doesn't work.

Still need to test height ranges.

[TokisanGames]

Update_height functions seem to properly work on sculpting and undo. Terrain3DObjects also works. Not sure why it didn't work yesterday. So, I think this PR is done enough for wider testing in the main branch. Now I'll work on the conversion PR #476 before merging.