[🐧] Lua in Mono megathread

[YoshiRulz]

Note: 2.9 has a new Lua engine, NLua+Lua, as the default.

---

If you're using 2.8 and getting a crash immediately upon opening the Lua Console like this:

System.NullReferenceException: Object reference not set to an instance of an object
at BizHawk.Client.EmuHawk.LuaConsole.AskSaveChanges () [0x00006] in <9a8bc2924d9d4ee0895a11a1f5b9e023>:0 
...

...it's probably because the Lua engine defaults to "Lua+LuaInterface", which usually crashes (the environment required for it to not crash is unclear). Change it to "NLua+KopiLua" in Config > Customize... > Advanced, or just grab a dev build.

[Dsm0]

On a fresh build of Bizhawk, upon opening the Lua editor, bizhawk crashes with terminal output

warning: File "/usr/bin/mono-sgen-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /usr/bin/mono-sgen-gdb.py
line to your configuration file "/home/{my username}/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/will/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
0x00007f126e972a2f in wait4 () from /usr/lib/libc.so.6

and then a bunch of thread dumps.

I don't know if this is a configuration error on my part, but if it isn't, and you'd like more details, I can open a new issue.

[YoshiRulz]

The actual error message:

* Assertion: should not be reached at metadata.c:3041

Looks like this was fixed by mono/mono#19434, and all Mono releases from 6.12.0.151 are unaffected. Unfortunately that version isn't available in the Manjaro repos so I'll have to wait. I built from source (see below) and confirmed that the fix works.

I can't offer you a workaround beyond deleting your config every time you launch EmuHawk, as that seems to avoid the crash somehow. edit: New, more helpful workaround: change Lua engine to "NLua+KopiLua". From 2.9 (41de03e) this will be the default on Linux.

[magpie514]

About the crash on mono < 6.12.151, Is there any other workaround besides deleting the config every time?
I tried building mono (6.12.0.158, the libgdiplus dependency on gtest is a problem, set -Wno-error=maybe-uninitialized in CXXFLAGS) and setting up a parallel environment for this, but the error persists, although instead of a plain crash it gives this (displayed in a dialog).

Using OpenTK 3 for host input (keyboard + gamepads)
Method '<Module>:_getFiberPtrId ()' in assembly '/home/magpie/Downloads/BizHawk-2.7-linux-x64/dll/lua51.dll' contains native code that cannot be executed by Mono on this platform. The assembly was probably created using C++/CLI.

System.TypeInitializationException: The type initializer for '<Module>' threw an exception. ---> <CrtImplementationDetails>.ModuleLoadException: The C++ module failed to load.
 ---> System.MissingMethodException: Method contains unsupported native code assembly:<unknown assembly> type:<unknown type> member:(null)
  at (wrapper managed-to-native) <Module>._getFiberPtrId()
  at <Module>.<CrtImplementationDetails>.LanguageSupport._Initialize (<CrtImplementationDetails>.LanguageSupport* ) [0x00024] in <01e64c95deda41b0ac7590707eabc864>:0 
  at <Module>.<CrtImplementationDetails>.LanguageSupport.Initialize (<CrtImplementationDetails>.LanguageSupport* ) [0x00028] in <01e64c95deda41b0ac7590707eabc864>:0 
   --- End of inner exception stack trace ---
  at <Module>.<CrtImplementationDetails>.ThrowModuleLoadException (System.String errorMessage, System.Exception innerException) [0x00007] in <01e64c95deda41b0ac7590707eabc864>:0 
  at <Module>.<CrtImplementationDetails>.LanguageSupport.Initialize (<CrtImplementationDetails>.LanguageSupport* ) [0x00041] in <01e64c95deda41b0ac7590707eabc864>:0 
  at <Module>..cctor () [0x00008] in <01e64c95deda41b0ac7590707eabc864>:0 
   --- End of inner exception stack trace ---
  at NLua.Lua.init () [0x00000] in <33fe4f6b9b274d9dab88987da2521403>:0 
  at NLua.Lua..ctor () [0x00027] in <33fe4f6b9b274d9dab88987da2521403>:0 
  at BizHawk.Client.EmuHawk.Win32LuaLibraries..ctor (BizHawk.Client.Common.LuaFileList scriptList, BizHawk.Client.Common.LuaFunctionList registeredFuncList, BizHawk.Emulation.Common.IEmulatorServiceProvider serviceProvider, BizHawk.Client.EmuHawk.MainForm mainForm, BizHawk.Client.Common.DisplayManagerBase displayManager, BizHawk.Client.Common.InputManager inputManager, BizHawk.Client.Common.Config config, BizHawk.Emulation.Common.IEmulator emulator, BizHawk.Emulation.Common.IGameInfo game) [0x00000] in <2e9a6324a4084dc4bc01031a9fd36595>:0 
  at BizHawk.Client.EmuHawk.LuaConsole.Restart () [0x0012f] in <2e9a6324a4084dc4bc01031a9fd36595>:0 
  at BizHawk.Client.EmuHawk.ToolManager.Load[T] (System.Boolean focus, System.String toolPath) [0x00173] in <2e9a6324a4084dc4bc01031a9fd36595>:0 
  at BizHawk.Client.EmuHawk.MainForm.OpenLuaConsole () [0x00000] in <2e9a6324a4084dc4bc01031a9fd36595>:0 
  at BizHawk.Client.EmuHawk.MainForm.LuaConsoleMenuItem_Click (System.Object sender, System.EventArgs e) [0x00000] in <2e9a6324a4084dc4bc01031a9fd36595>:0 
  at System.Windows.Forms.ToolStripItem.OnClick (System.EventArgs e) [0x00019] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.ToolStripMenuItem.OnClick (System.EventArgs e) [0x00090] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.ToolStripMenuItem.HandleClick (System.Int32 mouse_clicks, System.EventArgs e) [0x00000] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.ToolStripItem.FireEvent (System.EventArgs e, System.Windows.Forms.ToolStripItemEventType met) [0x00054] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.ToolStripItem.FireEvent(System.EventArgs,System.Windows.Forms.ToolStripItemEventType)
  at System.Windows.Forms.ToolStrip.OnMouseUp (System.Windows.Forms.MouseEventArgs mea) [0x00048] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.ToolStripDropDown.OnMouseUp (System.Windows.Forms.MouseEventArgs mea) [0x00000] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.Control.WmLButtonUp (System.Windows.Forms.Message& m) [0x00078] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.Control.WndProc (System.Windows.Forms.Message& m) [0x001b4] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.ScrollableControl.WndProc (System.Windows.Forms.Message& m) [0x00000] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.ToolStrip.WndProc (System.Windows.Forms.Message& m) [0x00000] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.ToolStripDropDown.WndProc (System.Windows.Forms.Message& m) [0x00017] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.Control+ControlWindowTarget.OnMessage (System.Windows.Forms.Message& m) [0x00000] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.Control+ControlNativeWindow.WndProc (System.Windows.Forms.Message& m) [0x0000b] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
  at System.Windows.Forms.NativeWindow.WndProc (System.IntPtr hWnd, System.Windows.Forms.Msg msg, System.IntPtr wParam, System.IntPtr lParam) [0x00085] in <1110f0149ba64c4fb0c8e7333cb663a5>:0 
BizHawk has completed its shutdown routines, killing process...

Seems more informative than the backtraces I was getting with the system-wide version of Mono, but hm.
What can be done? It's so close to working (and it does indeed work when deleting config.ini) but only runs once before having to reconfigure all the emulator and keys, so it's not very useful, sadly. Do I need to build bizhawk with this Mono install for it to work?
I don't mind that on principle, but it's going to be a giant amount of development packages just for one thing and that bothers me at a fundamental level (unless it's sure to work, then it's okay)

[YoshiRulz]

If you'd care to join me down the rabbit-hole, I've added Nix build scripts to the repo for both EmuHawk and the required version of Mono. That build is definitely immune from the crash. Unfortunately, I had to uninstall the older Mono and have broken my host OS such that I can't reinstall it, so I can't help look for another workaround.

[magpie514]

Nix, huh? I've been meaning to look into it, but won't have time until the weekend. That said, I'll give it a spin then, maybe I can set up a builder VM.

[bigbass1997]

If you'd care to join me down the rabbit-hole, I've added Nix build scripts to the repo for both EmuHawk and the required version of Mono. That build is definitely immune from the crash.

The necessary mono versions (6.12.0.151+) aren't available yet in the ubuntu package repos either. Never used nix before now, but I took a stab at it.

After installing nix, I cloned the bizhawk repo, and ran nix-build in the repo's root directory. However after running for awhile it failed the build. Here's the full output from nix, wasn't sure which parts were relevant to bizhawk/mono, so figured I'd include everything. Every following build attempt shows this.

[YoshiRulz]

Hmm I thought it might be unstable because it includes /.git, but it's the same after I pushed a commit... guess that was just a typo on my part. You could also build the cloned copy of the source with --arg useCWDAsSource true which obviously will bypass the hash check.

[bigbass1997]

That seemed to get a little farther, but now there's another issue. CLI output here. And the full log mentioned in the error.

[YoshiRulz]

That's the same problem GitLab CI had on fresh clones, which I fixed by building BizHawk.Version first. I could add that to the Nix expression, or you could do it manually (in nix-shell to use the same SDK). But this is straying further off-topic, so if we could move to Discord that'd be appreciated.

edit: fixed in 9448e56

[magpie514]

That's the same problem GitLab CI had on fresh clones, which I fixed by building BizHawk.Version first. I could add that to the Nix expression, or you could do it manually (in nix-shell to use the same SDK). But this is straying further off-topic, so if we could move to Discord that'd be appreciated.

Please post the results of that conversation if something comes up, I'm interested in setting up in a couple days.

[magpie514]

I succeeded in building it. Wow, Nix is really interesting, now I really want to learn more about it.

You can see it opens fine when a configuration is already existing in there. Thank you very much, YoshiRulz, this was a very functional and educative solution.
How does this work when updating, though? Do I just run the same commands after a git pull?

[YoshiRulz]

By the config.json I assume you copied it out of the Nix store or something? I actually got it working in the store, just run emuhawk-monort-2.7. It should be available in $PATH if you run nix-env -f default.nix -iA emuhawk from the repo (or do it declaratively—I haven't looked into that yet). As for updating, pass --arg useCWDAsSource true to nix-build/nix-env to make a dev build from the checked-out commit. By default it will grab the 2.7 source tarball instead. The dev build can be "installed" to your profile alongside 2.7, but only one dev build at a time, as they are all version 2.7.1-local.

[magpie514]

Oh, I meant to say it ran (and exited) once, generated a config and then ran again fine, meaning the crash doesn't happen anymore with this method. My bad. This builds works perfectly as far as my basic tests of the Lua interface go.
Thanks for the updating tips, I'll stick to this method as things go forward, until it can run with standard distro packages.

[TauAkiou]

I've been trying to get this working with no success for a few days now.

I've tried on two seperate systems, one running Fedora and one running Arch.

System mono provides similar results to @magpie514, with the same mono regression (crash on subsequent runs of the Lua Console)

Building mono from source (from latest master) exhibits similar behavior, though it creates an unhandled exception instead: Mono exception output

Running it through Nix has been unsuccessful on both Arch and Fedora, which seem to be attached to Nix: LibGL/driver failures on Fedora, and build failures on Arch. I don't feel like fighting with Nix, at least not right now.

Please let me know if any additional information is necessary.

[YoshiRulz]

I have a patch for debugging that NRE as another user had reported it (probably want to manually copy diff): https://gitlab.com/TASVideos/BizHawk/-/commit/8798cbcc7dc99b2270730fd4ff530380a268451d
IIRC it will only ever print "... is null" in the case of a use-after-free, which the runtime shouldn't be doing in safe managed code.

I'm very interested in debugging Nix failures though. You can DM me the logs at YoshiRulz#4472 on Discord or wherever.

[TauAkiou]

it turns out that this is the same error that @magpie514 has, pretty much exactly.

This still happens on the latest git release & the latest nightly build from the official repos which is mono 6.13.0 and by most accounts should be a later version then the version of mono with this bug.

Even using the Preview build, which is 6.12.0.158 exhibits the same issue.

[magpie514]

@TauAkiou I solved that with the Nix method, but I honestly can't tell why it's failing for you.
For whatever it's worth I did the process on Ubuntu and using the nVidia GL setup. I only ran into a minor issue, I had to use --arg useCWDAsSource true on the cloned repo for it to build right. Can you try that on Arch since that's the one that failed to build?

[TauAkiou]

The issue with Nix on Arch is related to it not finding autoconf or libtool in the Nix environment, which more refers to a problem with setup on Arch or, more likely, me not knowing how Nix works particularly well and not setting it up properly.

What's more confusing, though is that it works perfectly fine on Fedora, but Fedora doesn't have OpenGL configured properly, and that's a separate issue.

I'm more confused as to why BizHawk is exhibiting the same problem despite being run on versions that shouldn't be vulnerable to the bug. My guess is that somehow, the metadata fixes either aren't working or aren't applied in these builds for some bizarre reason.

[YoshiRulz]

On Arch, are you using nix-build --pure? Is Nix up-to-date?

On Fedora, pull (to 54e7bf2) and try again. I borrowed a Fedora machine and added the missing paths to the wrapper script.

edit: btw we resolved this (mostly)

[N00byKing]

I've had issues with a script using luasockets (Specifically, it fails to find the library). Is that a known issue?
Script in Question is here.

Still, thanks for getting this far already :D

[Zixiken]

I'm new to bizhawk, and pretty unfamiliar with lua in general. I've been messing with the co-op lua script, and I've managed to get to the point of it trying to load luasocket which fails with this error:

NLua.Exceptions.LuaScriptException: error loading module 'socket.core' from file '/usr/lib64/lua/5.4/socket/core.so':
/usr/lib64/lua/5.4/socket/core.so: undefined symbol: lua_gettop

Google seems to be suggesting this has something to do with compile flags bizhawk was compiled with, but I'm not sure. I do know I can use luasocket through the lua interpreter. I'm on gentoo, with mono-6.12.0.122.

[CasualPokePlayer]

On Linux, the Lua used will be the system lua (since 339915c / 2.9 rc3), so it would be the fault of the package manager here if the symbol was truly missing.

However, we do use that function internally, so if it was missing it would crash sooner. Perhaps there's something weird with that .so where it can't properly find the lua used?

EDIT: Searching through the internet seems to also point to possibly lua sockets being compiled with flags which give it a static linking requirement for liblua due to not actually explicitly linking against liblua, which would not work for us (we have to dynamically link), so that might also be an issue.

[Zixiken]

Hmm, looking at the files belonging to the lua package I'm only seeing a .so library, no .a. So I'm not sure it's possible that sockets statically linked? Regardless, I'm willing to try adding whatever link flags are necessary to ensure it compiles correctly, just not sure what those are.

I've been messing with strace, and I can see bizhawk opens and loads dlls/NLua.dll to memory, twice. Then, when I open the lua console, it loads /usr/lib64/liblua5.4.so to memory. Can you explain what part NLua.dll is playing here?

Continuing with strace, if I tell bizhawk to load the co-op script, it reads in related .lua files and eventually comes to /usr/lib64/lua/5.4/socket/core.so, which it maps to memory and then immediately unmaps and writes out the error. If I load the co-op script just with the interpreter on the command line, I see /usr/lib64/liblua5.4.so loaded to memory, the co-op .lua files are read, and /usr/lib64/lua/5.4/socket/core.so is mapped in the same fashion as bizhawk did. But then it continues on loading from that point until hitting a predictable error relating to missing bizhawk lua API. So it certainly seems that the system .so files know how to work together, and bizhawk does load them, but I guess it must be doing something different.

[YoshiRulz]

Is /usr/lib64/lua in LD_LIBRARY_PATH? Run mono EmuHawk.exe instead of using the launch script.

[Zixiken]

Is /usr/lib64/lua in LD_LIBRARY_PATH? Run mono EmuHawk.exe instead of using the launch script.

No change when run that way, and LD_LIBRARY_PATH is unset in my shell. But I did have to modify line 11 of EmuHawkMono.sh to make it gentoo-aware, bizhawk wouldn't start otherwise. Before:
"fedora") libpath="/usr/lib64";;

After:
"fedora"|"gentoo") libpath="/usr/lib64";;

[Zixiken]

Scratch that, it would start but crash when running a game.

[CasualPokePlayer]

Regardless, I'm willing to try adding whatever link flags are necessary to ensure it compiles correctly, just not sure what those are.

Add in -llua5.4 to the linker flags.

[Zixiken]

Add in -llua5.4 to the linker flags.

Alright, now I'm getting somewhere. I confirmed that link flag was missing by default when the libraries are linked, and confirmed that after adding the flag the output of ldd shows they now link to my system's /usr/lib64/liblua5.4.so.0. Running the co-op script in bizhawk now gets further, the error regarding luasocket is gone, and I had to adjust some of the script that was written with windows in mind. But I have a different error now, with the game script I'm trying to load in the co-op script. More specifically this is the offending line and error generated:

local game_code = string_from_byterange(memory.readbyterange(0x134, 9))

Error: NullHawk does not implement memory domains
NLua.Exceptions.LuaScriptException: A .NET exception occured in user-code
System.NotImplementedException: Error: NullHawk does not implement memory domains
  at BizHawk.Client.Common.MemoryApi.<get_Domain>g__LazyInit|12_0 () [0x00034] in <333d6f50a3904bd9a49c43cc2ba29b6b>:0 
  at BizHawk.Client.Common.MemoryApi.get_Domain () [0x00008] in <333d6f50a3904bd9a49c43cc2ba29b6b>:0 
  at BizHawk.Client.Common.MemoryApi.NamedDomainOrCurrent (System.String name) [0x0003c] in <333d6f50a3904bd9a49c43cc2ba29b6b>:0 
  at BizHawk.Client.Common.MemoryApi.ReadByteRange (System.Int64 addr, System.Int32 length, System.String domain) [0x00000] in <333d6f50a3904bd9a49c43cc2ba29b6b>:0 
  at BizHawk.Client.Common.MemoryLuaLibrary.ReadByteRange (System.Int64 addr, System.Int32 length, System.String domain) [0x00011] in <333d6f50a3904bd9a49c43cc2ba29b6b>:0 
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <a6a5ba8fc13a4797a32a4dc4ae25c772>:0

I'm not sure how to interpret this, is there any chance this will be implemented any time soon? Is there a better/other way the game script could be doing what it's trying to do? I do see that memory.readbyterange is deprecated, but I'm not sure if that's related.

[CasualPokePlayer]

"NullHawk" is the dummy "core" when no game is loaded. Load a game :P

[YoshiRulz]

NullHawk is the core used when no rom is loaded. sniped

I'm sure most of your problems would go away if you (or the script's author) used our provided socket library.

[Zixiken]

"NullHawk" is the dummy "core" when no game is loaded. Load a game :P

OH. I suppose that makes sense XD Running the game does indeed seem to work.

I'm sure most of your problems would go away if you (or the script's author) used our provided socket library.

That sounds great, though it's not exactly my project. The co-op script is here, and at the moment I'm mainly exploring if it's possible to even run on linux. I'm more interested specifically in Zelda Oracles randomizers, and I'd love to contribute at some point. So I'll certainly keep the provided socket library in mind!

[YoshiRulz]

Now we've ruled out our Lua host as the cause of your issue, we're done here. Enjoy 2.9 everyone.

[black-sliver]

The A problem [a lot of people are seeing with NLua] is that the EXE does not export the symbols required to use any native lua module. So it is the lua host's fault.

A work-around is to load the system lua instead of depending on the EXE exporting the correct symbols:
$ LD_PRELOAD="/usr/lib/liblua5.4.so" ./EmuHawkMono.sh (or wherever your lua 5.4 lib is)

This is a very fragile thing to do, so I'm not certain it's not gonna break stuff.

[CasualPokePlayer]

The lua host can't statically link lua due to the fact this is c# not some c/c++/whatever language. So we have to dynamically link. And that seemingly means the symbols aren't available for other modules.

However, looking at the dlopen page seems to indicate a solution here: https://man7.org/linux/man-pages/man3/dlopen.3.html

 RTLD_GLOBAL
              The symbols defined by this shared object will be made
              available for symbol resolution of subsequently loaded
              shared objects.

We probably just need to set that flag when it's opened so modules will see the other symbols.

[CasualPokePlayer]

Something to note of ^, I originally intended to use RTLD_NOLOAD, but that isn't part of POSIX and thus not actually available for all Linux OSes (at the very least Debian 11 did not support it).

[black-sliver]

are the gitlab builds nightly builds? so in ~14 hours we can fetch a dev build from gitlab CI/CD?

[YoshiRulz]

That's right.

[black-sliver]

Tested the nightly and works with our socket.so on my machine
First time [native] Lua [modules] actually are usable on my machine 🥳