Code runs in Wasmer (Cranelift JIT) but crashes in wasmtime with out of bounds memory access #2638
Comments
abbec
added
the
bug
abbec
changed the title
|
I was not able to reproduce this with 0.22.0 or main (e4827ad) on x64 linux. I assume from your homedir path that your crash is on x64 macos? |
|
Running the install script from wasmtime.dev again should install the latest build based on the main branch, so that should work for testing this, I think. |
|
And indeed, with the latest build I don't get a crash. I do get an error about a missing import, though, which is what #2637 is about. Output with the latest build |
|
@tschneidereit you need --dir pointing to a clone of cpython (or at least something containing a Lib folder) |
|
Ah, I see. Does it work for you with that in the latest Wasmtime build? |
Do you get the same as @tschneidereit because python needs a copy of its lib dir or else you will not "get to" the crash. I did try latest but --dir seems broken there |
Ah that readdir fix does look interesting, will try that. Unfortunately the large wasi rewrite seems to have broken --dir (as stated in the comment above) so it isn't possible to test there :( |
|
@abbec hm can you provide the exact invocations you're using of the wasmtime CLI? I'm only able to get the errors that @tschneidereit mentioned above regardless what version of wasmtime I use, I've never been able to get the original trap you mentioned. |
|
Er, and to clarify, I don't know what repository of cpython you're using, what commit you're using, or the exact working directory of the invocation you gisted above. |
|
Python needs access to its' standard library when running and I am using a checkout of cpython (the 3.8 branch) and passing that in as --dir . The exact command line inside a cpython (branch 3.8) clone is |
|
I can create a self-contained example |
|
Here is a self-contained bundle with the crash. Just unpack and run |
|
We made some subtle but, it turns out, breaking changes to how WASI rights work with the new wasi-common that landed yesterday. Without source code for how your python's wasi integration works, I can't debug much further, but here's what is going on to get you the In short, it looks like when you opened the The entire WASI rights system is difficult to understand, poorly documented, implemented completely different (if at all) across various WASI implementations, and not actually very useful. We are going to get rid of it soon. In the meantime, I'm not sure why your code requested file rights as the base (this could very well be a bug in the wasi-common rewrite) and we should figure out how to debug that. I'm happy to look at your python WASI build source to help debug it, or hop on a zoom call together this afternoon... |
|
It is a largely unmodified Python except for a few "hacks". A zoom call sounds super helpful and I am in the CEST time zone but suggest a time :) Although that breakage is interesting though, this is more about the crash I get before that change :) |
|
I backed up to |
|
Also, the code producing the binary is more or less here |
|
ok. mailed you at the address in your github profile as well if you want to zoom in 30 mins |
Wow! Nice find! Then we know that :) Thanks a lot! |
|
Just wanted to say that this was some serious first-class support! Thanks everyone! ❤️ |
the fdstat of a dirfd needs to include both the file and dir rights in the inheriting field. The wasi-libc path_open bases the base rights of child directories off the inheriting rights of the parent, so if we only put file rights in there, opening a child directory will not have any directory operations permitted. Fixes #2638
the fdstat of a dirfd needs to include both the file and dir rights in the inheriting field. The wasi-libc path_open bases the base rights of child directories off the inheriting rights of the parent, so if we only put file rights in there, opening a child directory will not have any directory operations permitted. Fixes #2638
What are the steps to reproduce the issue?
Run
wasmtime run python.wasm --dir . -- -m testin a clone of cpython version 3.8 (or at least the Lib folder).python.wasm.zip
What do you expect to happen? What does actually happen? Does it panic, and
if so, with which assertion?
I expect it to run without crashing like Wasmer using the Cranelift JIT backend does.
Callstack
Detailed Callstack
With Wasmer
Which Wasmtime version / commit hash / branch are you using?
0.22.1
If relevant, can you include some extra information about your environment?
This is a custom port of Python to WASI and some things are not expected to work. However, running
-m testwith Wasmer gives the expected Python error whereas running it with Wasmtime gives a memory corruption.Debugging further, the Python debug malloc (this is a debug build) gets an invalid allocation from
dlmalloccausing it to overwrite some internaldlmallocdata structures.Can also add that simpler things work fine in Wasmtime:
If there is a better way we can debug this to provide more/better info, please let us know!
The text was updated successfully, but these errors were encountered: