random getornull error spammed in console

[AndHalfAFish]

Godot version

3.4.4

System information

windows 10

Issue description

I've been getting a lot of 'getornull' errors lately.

I'm not sure what's causing them (seems random).
One thing that seems to trigger them sometimes, is adding a breakpoint.
I also got them with a faulty decision tree, and when trying to assign a non-existant animation to an animationplayer. I think these should give me different error messages, no ?

The only thing I can think of that might be causing this, is perhaps a high(er) vertex count ??? But the error doesn't happen all the time, so ... ?

Steps to reproduce

random error spam

Minimal reproduction project

No response

[Calinou]

@AndHalfAFish Please upload a minimal reproduction project to make this easier to troubleshoot.

In the future, please do not create issues if you have neither reproduction steps nor a minimal reproduction project.

[AndHalfAFish]

What are you telling me now ?
You don't want to hear about a recurring problem that spams the console (1000's of messages), that has a completely unhelpful error message ( 'core/rid.h') , and that prevents me from trouble shooting (breakpoints also sometimes set off the error) ... because I can't give you the steps to reproduce the problem ? You do know that 'intermittant' bugs (the ones that are hard to solve because they can't be easily reproduced) exist, right ?

.

[Calinou]

Thanks for including the complete error message this time 🙂
It's important to include the entire message, ideally by copying it, or at least taking a screenshot.
Paraphrasing the error message doesn't give us enough useful information, and it makes it difficult to search for the error message (or the error condition) in the codebase.

You said that this occurs regardless of whether your project is using any breakpoints. Could you upload a project where you've gotten this error message to appear even when there are no breakpoints set?

What are you telling me now ? You don't want to hear about a recurring problem that spams the console (1000's of messages), that has a completely unhelpful error message ( 'core/rid.h') , and that prevents me from trouble shooting (breakpoints also sometimes set off the error) ... because I can't give you the steps to reproduce the problem ? You do know that 'intermittant' bugs (the ones that are hard to solve because they can't be easily reproduced) exist, right ?

We have a lot of issue reports. Godot's issue tracker is one of the most active issue trackers on all of GitHub – about 25 new issues are created every day on average. However, we only have a limited amount of contributors willing to triage, comment and resolve issues. We have to be selective about the kinds of issue reports we can accept, so we can ensure that valid issue reports have a chance to be resolved. Over the years, we are becoming more selective on issue reporting to keep issue quality high. Most software projects out there do something similar as the community grows.

For reference, getornull errors were also referenced in #38229. See the complete list of issues referencing getornull: https://github.com/godotengine/godot/issues?q=is%3Aissue+sort%3Aupdated-desc+getornull

cc @tinmanjuggernaut, as they made #54650 but it apparently didn't resolve the issue on your end. Perhaps @lawnjelly knows something about this too.

[AndHalfAFish]

I was just about to delete my (snippy) reply (sorry - I was frustrated), but you had already answerred. Thanks for taking the time ! =)

I do understand your point, but, having worked as a software analyst and project manager (small team, but working together with other teams on a larger project), I have always preferred to be made aware of a problem even if I knew my team would not have the time to look into it / fix it.
It seems weird (in my opinion) to be selective about the bugs you allow users to report ...

I did (of course) do a web search before reporting the problem, and I did find out that this also happened randomly in other people's projects and with different versions of Godot.
As I stated : I can't give a minimal project - the error happens randomly & in different places in my game code base.
The only thing I can add, is that I'm at no point deleting objects from scenes - I'm only changing visibility.

[lawnjelly]

This might need a project that shows these errors, or running with a tracked_handles build.

Although the main build doesn't give much info, it's a bit of a generic error which if I remember right usually means it is trying to get an object from a RID that is referring to an object that has already been deleted (dangling RID), or a RID from another owner or garbage RID is calling get_or_null(). The nature of RIDs makes it rather easy to end up with a dangling RID, as the default implementation is not a lot more than a glorified pointer.

If you use a build with the rids=tracked_handles scons option, it should give you a source file and line number of the offending code (I tend to use this option for all my local builds, so it makes RID errors easier to diagnose). Another option is for one of us who uses windows to make such a build for your platform (I only have linux here or I would try to make one), if you aren't used to compiling the engine. The rids=tracked_handles option builds with an alternative implementation of RIDs which uses handles rather than pointers, having an extra layer of indirection for protection against dangling RIDs, and has much more tracking and debugging information.

My memory is a bit hazy of this, maybe get_or_null() flags an error in DEBUG builds instead of returning NULL, because maybe this would cause a crash in release build.

Probably also the gdscript for those functions in the callstack would be useful info too, or some indication what they are doing.

[TokisanGames]

cc @tinmanjuggernaut, as they made #54650 but it apparently didn't resolve the issue on your end.

@Calinou They?

@lawnjelly did not accept my RID rewrite and rolled his own in #54907

Also my other rid PR #55764 has long been ignored, even after prompting. Maybe that PR fixes the issue for Op. However, it's just as plausible that a new instance of rid abuse was created, as the root cause wasn't fixed in what was merged. #54907 (comment)

At the least, rid tracking should be the default setting in release_debug builds so that rid abuse can be identified quickly by all users. If it takes effort just to get an error message from users, how much more effort will it take to get them to make a custom build? Lol. The next 3.5 beta is the ideal time to turn this on. @akien-mga

@AndHalfAFish RIDs are used throughout the entire engine. Right now, your environment is the only known one with this error message. Most other instance of these messages regarding Id maps and rids were resolved with other changes that came about due to #53374. This is an example of a thorough report that led to a major system getting rebuilt. Please check it out so you get a sense of what is useful when tracking intermittent errors.

It is not happening randomly. It is happening after you are doing something. Maybe several somethings. It could be switching scenes. It could be clicking a UI element. It could be just moving the mouse. Regardless, it's helpful to know what you and your code are doing at the moment the errors are triggered. This means having the console on screen all the time so you can observe, then documenting behavior.

For the instances you gave:

• adding a break point. - How? Pressing F9? Clicking? While the game is running, paused, or stopped? As soon as you press F9, the error popped up? Was it this same exact error? Or was it slightly different?

• faulty decision tree. - What is this? Your code or a Godot class? How can we interpret this to identify a portion of Godot?

• trying to assign a non-existant animation to an animationplayer. - How are you doing that? Has the error message appeared after doing this more than once?

• The only thing I can think of that might be causing this, is perhaps a high(er) vertex count. - do you have evidence of this or just a random guess? I have projects with up to 250M vertices. What about them? What are you doing with the vertices?

You could include some of your code identified in the error message stack trace. We need to know if you're operating with the editor in code, the physics system (2d or 3d), the visual server, the gridmap, or some other system that uses Id maps. We don't know anything about your project, whether it's 2D or 3D, or what portions of the engine you're using, or even a game/plugin/app, etc.

The fastest way for this to be fixed is for you to make a custom build of the engine with rid tracking as Lawnjelly stated, and described in #54907. This will print out the sources of rids so the right section of code can be identified.

Without any of this information, it could be months, years, or never until this is fixed as right now you're the only one with the environment and any info about it.

[Calinou]

I compiled a Godot 3.5beta4 64-bit Windows editor debug binary with rids=tracked_handles: https://0x0.st/oTTS.zip
_{(link expires in late 2022)}

Make a backup of your project before using this build, just in case.

[AndHalfAFish]

Thank you all !
This is very helpful and exactly the kind of information that should be given to users not familiar with c++.

Maybe this is a good time to remind you, the engine builders, that we, the Godot users, also care about what happens to the engine. Just like you, we have invested a lot of time and energy into it. Many of us want to help .. but the only way we know how to, is by report bugs to the best of our ability (which might not always be up to your standards). =)

[lawnjelly]

Also my other rid PR #55764 has long been ignored, even after prompting.

I think things like this need a check by reduz, you might need to pester him on rocketchat. I added another comment on the PR though as it was a while since I looked at it.

At the least, rid tracking should be the default setting in release_debug builds so that rid abuse can be identified quickly by all users. If it takes effort just to get an error message from users, how much more effort will it take to get them to make a custom build? Lol. The next 3.5 beta is the ideal time to turn this on.

This is perhaps something we could consider now some time has passed.

As far as I remember it from the time:

• The handles builds needed testing -
  I've now been using the rids=tracked_handles builds for 5 months or so now as a daily with no problems at all, for debug and release_debug editor builds. This is a sample size of one, on one platform, but on the flip side the type of problems likely to occur will probably occur on all platforms.
• The tracked_handles may bloat the executable size a bit and be a little slower. This could be tested. (On the plus side, memory use was actually lower when I tested this.)

The other question is whether we might use tracked_handles for the editor, and pointers for release templates (as ultimately these should be fastest, in theory at least. We need to profile of course though. In practice, the difference might be negligible, in which case robustness would win.). Given that these two are quite different there is the opportunity for bugs if the implementation of either diverges. An alternative is to make release templates with rids=handles, which has all the same robustness but can't provide such detailed error messages. This is the approach that Godot 4 takes, if I remember correctly.

On the other hand there is always the potential to introduce new nasty hard to find bugs if we change this in the release templates, particularly threading bugs, which the handles is more susceptible to. So on balance I am probably still slightly in favour of keeping the current release template to use pointers (if it ain't broke etc).

I'm personally fairly agnostic on this really, and we should also weigh in how often RID errors happen in practice (not very often it seems now that we have fixed the major existing bugs), and we do have the tools to investigate them now when they do crop up.

Another option is to build with rids=tracked_handles for DEV_ENABLED builds, so that more eyes are likely to see RID bugs earlier. Or we could publish a debug build of the editor with all debugging enabled and rids=tracked_handles with each release, for windows only might be okay as linux users could run it under wine. I think there was some interest in having debug builds available so maybe this could be a "kill two birds with one stone" thing.

[AndHalfAFish]

Update :

@Calinou : last night, I tried your Godot 3.5beta4 version and I did no longer see the bug brought on by placing a breakpoint in my code, and then running the game (F5)

I am using Godot 3.4.4 in my project, so I was wondering if maybe I caused the bug by changing something in the Godot settings (project or editor). So, I re-downloaded from the webside.
The breakpoint bug is still there.

This morning, I tried compiling from source :

setting up : 1) Python 3.8.1 , 2) SCons script: v3.1.2 (2019-12-17), engine: v3.1.2 (2019-12-17)

download src : 3.4.4-stable (source code as zip) from github releases

build with default settings to see if everything works :
>scons
tried my project without breakpoint : runs fine
tried project with breakpoint placed (BP is placed before pressing F5) : bug reappears

build with tracked_handles arg (not sure if I did this correctly) :
>scons rids=tracked_handles
tried project with breakpoint placed (BP is placed before pressing F5) : bug reappears

Notes :
. just to make sure, I always used a fresh copy of my game project.
. breakpoint is always placed before pressing F5 ; the bug appears when I'm dropping an inventory item on another item in the game

Let me know if there's anything else I can do.

godot.344.w.BP.1.mp4

[TokisanGames]

Good, you have a reproducible error. As soon as you do that action in the video 112 errors appear. Let's look at your console, and the log file. Please post the whole thing of a single run. Attach text files. Also what is the code that is run immediately after the click, and for the next 3 seconds while the errors appear?

Edit: Also project settings are saved in your project folder, so apply to any version of the engine you run.

[AndHalfAFish]

I found a log file here : C:\Users\sc\AppData\Roaming\Godot\app_userdata\Shade\logs
I hope that's the right one ?

The 2 different warnings are from before the time when this error started showing up.

godot_trackedhandles.log

The error happens with the breakpoint placed at different places within that function ( do_hsd_click() ), eg/ BP at lines 39, 43, 55, ... The one shown in the clip is for line 79. Only 1 breakpoint is placed per run.

do_hsd_ftion.txt

After running through that function, some _process() code runs in several files (setting a bool for an animationplayer, or checking player move ftionality). These bits of _process() code are used by other files as well.

After testing some other files that have a do_hsd_click() function (placed BP on the first line of code within) : it appears they have the same problem even though the scenes don't share assets and the code within the function is different.

The function do_hsd_click() is called from an obj_under_mouse() function. I'm currently using 2 different versions but they give the same result (error if BP).

game.txt

Next when I have some more time, I will check inventory code.

[TokisanGames]

I found a log file here : C:\Users\sc\AppData\Roaming\Godot\app_userdata\Shade\logs I hope that's the right one ?

Yes.

These log files did give us new information:
RID_Owner<struct RasterizerStorageGLES3::Texture>::getornull

The function do_hsd_click() is called from an obj_under_mouse() function. I'm currently using 2 different versions but they give the same result (error if BP).

You mean two versions of the engine? 3.4.4-stable and ?? But the 3.5b4 from Calinou does not produce errors?

These errors only appear if there is a breakpoint set, otherwise not?

I don't see the RID database, that should have been built with rids=tracked_handles. It should be reporting info like this to the console when you quit the game. Are you looking at the console? The black window? Editor/Toggle System Console. I'm not sure if this info goes to both the log and the console, but I don't see it in what you posted.

ERROR: RID_Database leaked 21 objects at exit.
   at: RID_Database::shutdown (core\rid_handle.cpp:142)
ERROR: 18 RID objects leaked
   at: struct VisualServerCanvas::Item (scene\2d\canvas_item.cpp:1268)
ERROR: 3 RID objects leaked
   at: struct RasterizerStorageGLES3::Texture (scene\resources\texture.cpp:828)

Ah, #54907 has the RID database, which was merged into 3.x on Dec 2021. It is not in 3.4. Please build from the 3.x source and see if you get an RID database. Perhaps the reason you don't see the errors in Calinou's 3.5 build is because the problems were already fixed.

Most likely the cause is in the engine, not your code. But you may be using a function of the engine that has the bug that not many others are using, or if they are, they aren't triggering the error. RID errors are like a 2-key door. One key causes the problem, another key triggers the error but is unrelated to the cause, both are necessary to get the error.

Let's wait until you can get the error message in any of the 3.5 builds, or preferably your own 3.x build with rids=tracked_handles. If you can't reproduce the error message, we could consider this addressed.

[AndHalfAFish]

@tinmanjuggernaut : Thanks for your help !

'You mean two versions of the engine?'
No. I meant I was using 2 different versions of how I called the do_hsd_click() function (the old way, and a new way I'm moving to).

'Please build from the 3.x source and see if you get an RID database.'
I'm not sure what you're saying here ? I downloaded the 3.4.4 source code & built the engine by using
>scons rids=tracked_handles
Is this incorrect ?

'Most likely the cause is in the engine, not your code.'
I already assumed as much, since the error is talking about rid's suddenly pointing to nothing ... while I'm only changing visibility of objects ; not deleting any.

[TokisanGames]

'Please build from the 3.x source and see if you get an RID database.'
I'm not sure what you're saying here ? I downloaded the 3.4.4 source code & built the engine by using scons rids=tracked_handles
Is this incorrect ?

The command is correct, the code base is not. Rid handles is not in 3.4. It's only in 3.5, or the 3.x branch.

[AndHalfAFish]

Sorry, what is the 3.x branch ? Is 3.4 not a 3.x branch ???

[TokisanGames]

The branch is called 3.x
https://github.com/godotengine/godot/tree/3.x

That is where 3.5 and beyond are being developed.

[AndHalfAFish]

thanks

[lawnjelly]

Thanks to @tinmanjuggernaut for helping with this.

• rids=tracked_handles is indeed only available from after the time [3.x] Add option to use handles to RID #54907 was merged, so is only in 3.5 onwards. We did backport some of the rid fixes to 3.4, but not the ability to use the handles for debugging.

When it is compiling correctly you should see a "WARNING: Building with RIDs as tracked handles." message in the scons output.

If it isn't happening in Calinou's 3.5 build, then it suggests:

• Possibly already fixed in 3.5
• Possibly not occurring because it is a debug build
• Possibly not occurring because of the different implemention of RIDs

It would be interesting to try it with e.g. 3.5 beta 4 and confirm whether or not the problem occurs, if you haven't already, that should let us know whether it is something that is fixed in 3.5 at least.

One thing that can definitely cause all kinds of bugs is multithreading, if you are using any of this in your scripts, that would be helpful to know. From your later output it seems like the problem is occurring with a texture RID.

Other than that, it may be necessary to create a minimum reproduction project, as tracking down RID bugs is incredibly difficult without the handles build (well that was our experience with the previous RID bugs). In fact even with info from the handles build it is likely we'd end up needing an MRP to solve this.

Some tips for creating a MRP are here:
https://github.com/lawnjelly/Misc/blob/master/MRP2.md

Another option if and only if it is fixed in 3.5, is for you to manually bisect the commits and find the commit that fixed it. This can be quite time consuming, and is only worth providing instructions if we can confirm it is fixed in 3.5 and have exhausted other options I guess.

[AndHalfAFish]

Alright, some more info :

. Calinou's 3.5b4 build (debug build) : placing breakpoint is fine
. 3.5b4 download from downloads page : (? release) : placing breakpoint results in getornull error
. my 3.4.4 build (debug) : placing breakpoint results in getornull error
. 3.4.4 download (?release) : placing breakpoint results in getornull error

. I'm not using multithreading

. I'll try to create am MRP ... but, this isn't a project I started last month =)

Question : what actually happens when I place a breakpoint in the editor (I have no experience with engine code) ? How can it introduce a bug/bring a bug to the foreground in something I wrote ?

[lawnjelly]

Question : what actually happens when I place a breakpoint in the editor (I have no experience with engine code) ? How can it introduce a bug/bring a bug to the foreground in something I wrote ?

I'm unfamiliar with how the editor breakpointing is implemented, but you could have, for example, the editor trying to print out the value of some local variable that has just been deleted (QTCreator actually has / had a similar bug).

There's still a huge amount of possibilities here, that probably only an MRP can be used to pin down. In terms of differences between the handles build and non-handles, one possibility is that there is a mutex in the handles build which may make it more thread safe in some situations. If you managed to compile and run both a 3.x (i.e. the current Godot 3 master branch) regular debug build and a 3.x tracked_handles debug build that would be comparing apples to apples and be slightly better to deduce this. But really the MRP is the biggest priority.

For the MRP, you should be able to use https://en.wikipedia.org/wiki/Bisection_method to eliminate large parts of your project quickly, you don't have to remove one node at a time, remove entire branches. And the project doesn't have to "run", if the error appears in the editor. If it is taking you a significant amount of time, you are probably doing it wrong (it should probably take 5-30 mins max).

Another possibility if you feel this is beyond you, is to email a private confidential link to one of the core devs (we often end up doing this for bespoke help with games).

[AndHalfAFish]

@lawnjelly : I have managed to create a much smaller project (zip 2.75MB) ... How do I get it to you ?
(I never want to do this again ! =) )

[Zireael07]

@AndHalfAFish: You can attach files to GitHub comments or to the opening post of this issue. Zip files are accepted.

[lawnjelly]

Yes you can either (preferably) just drag and drop a zip file into a post on here, or you can email me if it contains assets you don't want public. My username here @gmail.com . 🙂

[AndHalfAFish]

The 2 files of interest are res://game.gd and res://p5_spiro.gd
I explain inside the 2 files how to get the error.

When you run the project, you'll see a 'gear' object and a 'table' object, as well as an inventory.
Click the gear to 'pick it up'.
Click the gear icon in the inventory, next, click the table (= drop item ; don't use drag, just click)

Shade_test_min.zip

[lawnjelly]

Quoting from the doc I posted:

Strictly speaking, by definition, a minimum reproduction project is a project that demonstrates the bug, but does nothing else. It should contain no superfluous scenes, nodes, gdscript, addons, textures, models, animations, sounds etc.

Ideally there should be nothing present in your project that we could remove, such that the bug would still exhibit.

Given that this is a reference counting bug, it is possible that it is a misuse of references in the gdscript (rather than an engine bug). This should become more obvious with an MRP.
(That is not to say that the engine should not better detect such a bug if it is in the gdscript, but we first need to identify where the bug is.)

[lawnjelly]

I suspect the error is happening here:

func reset_mouse_icon():
	mouse_icon.visible = false
	mouse_icon.texture = mouse_icon_empty

That might tally in with it calling set_texture() in the c++. Do the two texture types mentioned above tally in with this? i.e. changing from "res://icons_inv/objs/p5_p" to "icons_inv/empty_cell.png"?

Can confirm if that 3rd line is commented out, the RID error is gone. So it looks like the refcount of "res://icons_inv/objs/p5_p" is out of sync because it is still being used somewhere. Where else are you using this "res://icons_inv/objs/p5_p" texture?

Ok the plot thickens, as I suspected something is also being set as a result of mouse click (which may come in out of sync or on another thread?):

func on_invent_item_clicked(cell:Grid_Cell):
...
		invent.update_mouse_icon(cell.texture)

This line is also being called perhaps causing a race condition here.

My best guess is that something in the ordering of events is getting mucked up by having the breakpoint in the debugger in the Godot IDE. I have absolutely no idea how the debugger works, so this issue may end up needing to be solved by someone who works on the debugger.

But yes paring it down to an MRP is necessary I think before spending more time investigating this.

[AndHalfAFish]

You know ... that's a weird name "p5_p". It should have some numbers, and end in "_cell.png" or "_dv.png". Can you let me know if the name really is "p5_p", or if it is "p5_p20c_cell.png" ?

The problem area you're describing, is where I set an image following the mouse pointer, depending on whether an item is clicked in the inventory. If something is selected, the mouse image is changed to that of the selected object. If nothing is selected, the image goes back to the 'empty' image.

But yes, I'll continue to try and get rid of more stuff in the project.

[lawnjelly]

You know ... that's a weird name "p5_p". It should have some numbers, and end in "_cell.png" or "_dv.png". Can you let me know if the name really is "p5_p", or if it is "p5_p20c_cell.png" ?

This is a limitation of the debugger in QTCreator I'm currently using, it only seems to display a certain number of characters. So it is one of those two textures likely.

[AndHalfAFish]

@lawnjelly : Based on all the information you gave me, I decided to print out the RIDs of the Textures used in the inventory.
I added the print code to the grid_cell.gd 'find_cell_img()' method, which is the place where I load the images to be displayed in the inventory cells. This method is called whenever something has changed in one of the inventory collections (eg/ add an object, etc) :
print(res.get_rid().get_id())

Result :
. conditions (game.gd) of no err with bp :

under.do_hsd_click(selected_item_id) # calls p5_spiro.gd (bp)
g_game.invent.use_item(selected_item_id) # calls grid_cell.gd (print rids)

=> bp is reached before last RIDs print (= expected)

. conditions (game.gd) of err with bp :

g_game.invent.use_item(selected_item_id) # calls grid_cell.gd (print rids)
under.do_hsd_click(selected_item_id) # calls p5_spiro.gd (bp)

=> last RIDs print before bp is reached ; seems as if bp stops inventory code right in the middle of it removing the gear texture from the inventory (=white image displayed)

I have made recordings of the RID prints in both situations (1 and 1.7 MB) and another recording focussing on the white image. I'm not sure if it's ok to post all of them, so I'll just post the last one. Let me know if you want to see the other ones too.

getornull_a_bp_err_2_cut.mp4

[AndHalfAFish]

For completeness : there is an even smaller project now.
Shade_test_min2.zip

I also added the other movie clips (show the timing of the RID refresh for the cell images in the inventory)
getornull_movies.zip

[Calinou]

@AndHalfAFish Can you (or anyone else) still reproduce this bug in Godot 3.5.1 or any later release?

[AndHalfAFish]

Hi, I tried the minimal_project in Godot 3.5.1. I still get the same error (only when I place a breakpoint), but there is no longer any console spamming. [attachment console.png] I have found a way to work arround the issue, so for me it's ok if you put it in the 'some-day' bin. It may be some weird timing issue specific to my project ... Siska. Sent with [Proton Mail](https://proton.me/) secure email.

…

------- Original Message -------

On Sunday, November 6th, 2022 at 6:32 PM, Hugo Locurcio ***@***.***> wrote: ***@***.***(https://github.com/AndHalfAFish) Can you (or anyone else) still reproduce this bug in [Godot 3.5.1](https://godotengine.org/article/maintenance-release-godot-3-5-1) or any later release? — Reply to this email directly, [view it on GitHub](#60406 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AOCYBZSH2NPCEEI25NVNIELWG7TUNANCNFSM5T6PU7ZA). You are receiving this because you were mentioned.Message ID: ***@***.***>

[Zireael07]

Attachment didn't attach

[siskax07]

I was referring to this : #60406 (comment)