Merge upstream changes that have occurred since the fork #59

Instead of relying purely on the assumption that type handles can be compared cheaply by pointer equality, fallback to a more expensive walk of the type tree that recursively compares types structurally. This allows different components to call into each other as long as their types are structurally equivalent. Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>

* Start refactoring wasi-http * Checkpoint * Initial implementation of response future handling * Lazily initialize response headers and body * make wasmtime-wasi-http compile * wasi-http wit: make a way to reject outgoing-request in outgoing-handler before waiting for the future to resolve * wasi: sync wit from wasi-http * outgoing handler impl: report errors to userland * test-programs: get wasi-http-components kicking over, delete modules and components-sync tests wasi-http-components-sync will come back once we get done with other stuff, but its superfulous for now. wasi-http-modules will not be returning. * Process headers * Add HostIncomingBody::new * Add trailers functions * Add TODO for body task outline * Rework incoming-response-consume to return a future-trailers value as well * Fix the wit * First cut at the worker loop * wasi-http: change how we represent bodies/trailers, and annotate own/borrow/child throughout * Update types_impl.rs for wit changes * Split body management into its own module * Checkpoint * more work on incoming body and future trailers * Fill out some more functions * Implement future-trailers-{subscribe,get} * Implement drop-future-trailers * Rework fields, but make the borrow checker mad * Fix borrow error * wasi-http-tests: fix build * test-runner: report errors with stdout/stderr properly * fix two trivial wasi-http tests the error type here changed from a types::Error to an outbound_handler::Error * Remove unnecessary drops * Convert a `bail!` to a `todo!` * Remove a TODO that documented the body worker structure * fill in a bunch more of OutputBody * Remove the custom FrameFut future in favor of using http_body_util * Move the outgoing body types to body.rs * Rework the handling of outgoing bodies * Fix the `outgoing request get` test * Avoid deadlocking the post tests * future_incoming_request_get shouldn't delete the resource * Fix the invalid_dnsname test * implement drop-future-incoming-response * Fix invalid_port and invalid_dnsname tests * Fix the post test * Passing a too large string to println! caused the large post test to fail * Format * Plumb through `between_bytes_timeout` * Downgrade hyper * Revert "Downgrade hyper" This reverts commit fa0750e. * Restore old https connection setup * Sync the wasi and wasi-http http deps * Fix tests * Remove the module and component-sync tests, as they are currently not supported * Fix the reference to the large_post test in the components test * Fix wasi-http integration * sync implementation of wasi-http * Slightly more robust error checking * Ignore the wasi-http cli test prtest:full * Consistent ignore attributes between sync and async tests * Fix doc errors * code motion: introduce intermediate `HostIncomingBodyBuilder` rather than a tuple * explain design * Turn FieldMap into a type synonym * Tidy up some future state (bytecodealliance#7073) Co-authored-by: Pat Hickey <phickey@fastly.com> * body HostInputStream: report runtime errors with StreamRuntimeError HostInputStream is designed wrong to need that in the first place. We will fix it in a follow-up as soon as resources land. --------- Co-authored-by: Pat Hickey <phickey@fastly.com> Co-authored-by: Alex Crichton <alex@alexcrichton.com>

* Support resource maps in `component::bindgen!` This commit adds support to `component::bindgen!` to specify resource types using the `with` key of the macro. This can be used to configure the `T` of `Resource<T>` to use a preexisting type rather than unconditionally generating a new empty enum to have a fresh type. * Reenable tests

…ance#6935) * Include Wasm-defined globals and memories in core dumps Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * Narrow scope of unsafe block * Add missing skip-miri attribute for test that calls into Wasm --------- Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* Remove manual impl of Error/Display These conflict with auto-generated ones. * Fix tests

* riscv64: Delete `int_load_op` helper * riscv64: Delete `default_memflags` * riscv64: Rename `gen_amode` * riscv64: Better matching for amode * riscv64: Delete `emit_load` helper * riscv64: Rename some load variables * riscv64: Merge `iadd` into amode computations

…nce#7080) * riscv64: Add `c.li` and `c.lui` * riscv64: Add CB type instructions `c.srli` / `c.srai` / `c.andi` * riscv64: Add `sp` relative load instructions * riscv64: Return Option from try_emit_compressed * riscv64: Implement stack based stores * riscv64: Add compressed stores * riscv64: Add compressed loads

This commit makes it so that the library type for core dumps is serializable into the standard binary format for core dumps. Additionally, this commit makes it so that we use the library type for generating core dumps in the CLI. We previously were using a one-off implementation of core dump generation that only had backtrace information and no instances, modules, globals, or memories included. The library type has all that information, so the core dumps produced by our CLI will both be more featureful and be generated by shared code paths going forward. Along the way, implementing all this required some new helper methods sprinkled throughout `wasmtime` and `wasmtime-runtime`: * `wasmtime::Instance::module`: get the module that a `wasmtime::Instance` is an instance of. This is public, since it seems generally useful. This involved adding a new return value from `ModuleRegistry::register_module` that is an identifier that can be used to recover a reference to the registered module. * `wasmtime::Instance::all_{globals,memories}`: get the full global/memory index space. I made these `pub(crate)` out of caution. I don't think we want to commit to exposing non-exported things in the public API, even if we internally need them for debugging-related features like core dumps. These also needed corresponding methods inside `wasmtime-runtime`. * `wasmtime::{Global,Memory}::hash_key`: this was needed to work around the fact that each time you call `{Global,Memory}::from_wasmtime`, it creates a new entry in the `StoreData` and so you can get duplicates. But we need to key some hash maps on globals and memories when constructing core dumps, so we can't treat the underlying `Stored<T>` as a hash key because it isn't stable across duplicate `StoreData` entries. So we have these new methods. They are only `pub(crate)`, are definitely implementation details, and aren't exposed in the public API. * `wasmtime::FrameInfo::module`: Each frame in a backtrace now keeps a handle to its associated module instead of just the name. This is publicly exposed because it seems generally useful. This means I also deprecated `wasmtime::FrameInfo::module_name` since you can now instead do `frame.module().name()` to get that exact same info. I updated callers inside the repo.

* Optimize wasm calls ever-so-slightly * Fix riscv64 * Remove stray comment * Shuffle around where `unsafe` lies

…7086) * Wasmtime: Move `Global` and `Table` to `externals` submodules Just mechanical code motion, not functional changes. * Wasmtime: Add `hash_key` methods for `Func` and `Table` * Rename `wasmtime::Func::caller_checked_func_ref` to `wasmtime::Func::vm_func_ref` We removed the "caller checked" part of the `VMFuncRef` type's name a while ago, so update this method to be in line with that. No functional changes, just mechanical renaming. * cargo fmt * Fix doc link

…ance#7089) These registers require an additional byte to reference when encoded in certain AVX instruction formats (and maybe other situations as well?) so prefer xmm{0..7} when they are available and only fall back to xmm{8..15} when register pressure is higher.

…7087) * Add docs for debugging with core dumps * Fix reference to old style CLI flag * Add `no_run` to example that is only there to trap

…ytecodealliance#7093) For `XmmRmiRVex`-format instructions, when the opcode is commutative, the first operand is one of xmm{0..7}, and the second operand is one of xmm{8..15}, then we can swap the operands to save a byte on instruction encoding.

This update gets rid of a panic that occurs when using `--profile=vtune` from the command line.

* Bump wasm-tools crates Two major changes/reasons for this update: * Primarily pulling in support for semicolons-in-WIT files. Semicolons are not currently required, though, so I'll follow-up later with actual semicolons. * The syntax for parsing `(if ...)` was fixed in `wast`. Previously it did not require `(then ...)` but this is required by the spec. New spec tests require this as well. This breaks existing text format tests which don't use `(then ...)` inside of an `(if ...)`. Most tests were updated by hand but `embenchen_*` tests were updated by running through the old parser to produce non-s-expression using code. * Fix an example `*.wat`

* delete preview2::pipe::pipe this is not used anywhere, not documented, and trivial for the user to build on their own if they need it. i suspect they probably only want one end of their pipe wrapped with AsyncReadStream or AsyncWriteStream. * fix cfg_attr to ignore forking tests on qemu * delete "async fd stdin" test, which is just the worker thread one anyway

@fitzgen

…7072) * Add memory protection keys (MPK) In order to use MPK on an x86_64 Linux system, we need access to the underlying `pkey_*` system calls (`sys`), control of the x86 PKRU register (`pkru`), and a way of determining if MPK is even supported (`is_supported`). These various parts are wrapped in a `ProtectionKey` abstraction along with a `ProtectionMask` that can be used `allow` the CPU to access protected regions. * Integrate MPK into the pooling allocator This change adds "stripes" to the pooling allocator's `MemoryPool`. Now, when requesting a slot in which to instantiate, the user (i.e., `InstanceAllocationRequest`) will be transparently assigned to one of the stripes, each of which is associated with a protection key. The user can also request a specific protection key to use, which will override the original "find me a slot logic". This has implications for how instances get allocated: once a store is assigned a protection key, it will only allocate requests with that key, limiting how many slots it has access to. E.g., if 15 keys are active, the store can only ever access 1/15th of the slots. This change also includes a tri-bool configuration field, `memory_protection_keys`, which is disabled by default for the time being. * Address review comments This is a rollup of 43 commits addressing review comments of various kinds: bug fixes, refactorings, documentation improvements, etc. It also ensures that CI runs all checks. A big thanks to @fitzgen and @alexcrichton for the review! prtest:full Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> Co-authored-by: Alex Crichton <alex@alexcrichton.com> --------- Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> Co-authored-by: Alex Crichton <alex@alexcrichton.com>

…ance#7098) * x64: Fix false dependencies in int-to-float conversions This commit is a result of the investigation on bytecodealliance#7085. The int-to-float conversion instructions used right now on the x64 backend will implicitly source the upper bits of the result from a different register. This implicitly creates a dependency on further consumers using the conversion result on whatever previously defined the upper bits, even though they aren't used. This false dependency is the primary reason for the slowdown witnessed in bytecodealliance#7085. The fix chosen in this commit is to model the int-to-float instructions with a new shape of instruction instead of the previous `GprToXmm{,Vex}`. This previous shape was modeled as single-input and single-output, but this does not reflect the actual nature of the `cvtsi2s{s,d}` instructions. Instead these now use `CvtIntToFloat{,Vex}` which have two source operands and one destination operand, modeling how the upper bits of a different register are used. In lowerings using this instruction the upper bits to preserver are always sourced from a zero'd out register to force breaking dependencies between instructions. Closes bytecodealliance#7085 * Remove now dead code * Remove outdated test Golden test output covers this test case anyway nowadays * Review comments * Fix emit tests

Pull in bytecodealliance/wasm-tools#1221

* Mark some frequently used methods as `inline` * Seed the preview2 context with the thread_rng, instead of using a syscall

This has no longer been used since bytecodealliance#7056 and I believe it's just an accident that it wasn't removed as part of that PR.

* ci: Upgrade QEMU to `8.1.1` This adds support for RISC-V's Zcb extension that includes some extra compressed instructions. It also removes the current cpuinfo patch, that has been released in 8.1 * wasmtime: Don't assert the exact faulting address for wasm traps

) * Move the incoming_handler impl into http_impl * Remove the incoming handler -- we need to use it as a guest export * Start adding a test-programs test for the server side of wasi-http * Progress towards running a server test * Implement incoming-request-method * Validate outparam value * Initial incoming handler test * Implement more of the incoming api * Finish the incoming api implementations * Initial cut at `wasmtime serve` * fix warning * wasmtime-cli: invoke ServeCommand, and add enough stuff to the linker to run trivial test * fix warnings * fix warnings * argument parsing: allow --addr to specify sockaddr * rustfmt * sync wit definitions between wasmtime-wasi and wasmtime-wasi-http * cargo vet: add an import config and wildcard audit for wasmtime-wmemcheck * cargo vet: audit signal-hook-registry * Remove duplicate add_to_linker calls for preview2 interfaces prtest:full * Add a method to finish outgoing responses Co-authored-by: Adam Foltzer <acfoltzer@fastly.com> Co-authored-by: Pat Hickey <phickey@fastly.com> * Mark the result of the incoming_{request,response}_consume methods as own * Explicit versions for http-body and http-body-util * Explicit `serve` feature for the `wasmtime serve` command * Move the spawn outside of the future returned by `ProxyHandler::call` * Review feedback --------- Co-authored-by: Trevor Elliott <telliott@fastly.com> Co-authored-by: Adam Foltzer <acfoltzer@fastly.com>

* Remove usage of `is-terminal` and `atty` crates This functionality is now folded into the standard library itself. * Fix syntax * Fix a unix/windows cfg

* winch(x64): Call indirect This change adds support for the `call_indirect` instruction to Winch. Libcalls are a pre-requisite for supporting `call_indirect` in order to lazily initialy funcrefs. This change adds support for libcalls to Winch by introducing a `BuiltinFunctions` struct similar to Cranelift's `BuiltinFunctionSignatures` struct. In general, libcalls are handled like any other function call, with the only difference that given that not all the information to fulfill the function call might be known up-front, control is given to the caller for finalizing the call. The introduction of function references also involves dealing with pointer-sized loads and stores, so this change also adds the required functionality to `FuncEnv` and `MacroAssembler` to be pointer aware, making it straight forward to derive an `OperandSize` or `WasmType` from the target's pointer size. Finally, given the complexity of the call_indirect instrunction, this change bundles an improvement to the register allocator, allowing it to track the allocatable vs non-allocatable registers, this is done to avoid any mistakes when allocating/de-allocating registers that are not alloctable. -- prtest:full * Address review comments * Fix typos * Better documentation for `new_unchecked` * Introduce `max` for `BitSet` * Make allocatable property `u64` * winch(calls): Overhaul `FnCall` This commit simplifies `FnCall`'s interface making its usage more uniform throughout the compiler. In summary, this change: * Avoids side effects in the `FnCall::new` constructor, and also makes it the only constructor. * Exposes `FnCall::save_live_registers` and `FnCall::calculate_call_stack_space` to calculate the stack space consumed by the call and so that the caller can decide which one to use at callsites depending on their use-case. * tests: Fix regset tests

…dealliance#7029) * Rename `Host*` things to avoid name conflicts with bindings. * Update to the latest resource-enabled wit files. * Adapting the code to the new bindings. * Update wasi-http to the resource-enabled wit deps. * Start adapting the wasi-http code to the new bindings. * Make `get_directories` always return new owned handles. * Simplify the `poll_one` implementation. * Update the wasi-preview1-component-adapter. FIXME: temporarily disable wasi-http tests. Add logging to the cli world, since stderr is now a reseource that can only be claimed once. * Work around a bug hit by poll-list, fix a bug in poll-one. * Comment out `test_fd_readwrite_invalid_fd`, which panics now. * Fix a few FIXMEs. * Use `.as_ref().trapping_unwrap()` instead of `TrappingUnwrapRef`. * Use `drop_in_place`. * Remove `State::with_mut`. * Remove the `RefCell` around the `State`. * Update to wit-bindgen 0.12. * Update wasi-http to use resources for poll and I/O. This required making incoming-body and outgoing-body resourrces too, to work with `push_input_stream_child` and `push_output_stream_child`. * Re-enable disabled tests, remove logging from the worlds. * Remove the `poll_list` workarounds that are no longer needed. * Remove logging from the adapter. That said, there is no replacement yet, so add a FIXME comment. * Reenable a test that now passes. * Remove `.descriptors_mut` and use `with_descriptors_mut` instead. Replace `.descriptors()` and `.descriptors_mut()` with functions that take closures, which limits their scope, to prevent them from invalid aliasing. * Implement dynamic borrow checking for descriptors. * Add a cargo-vet audit for wasmtime-wmemcheck. * Update cargo vet for wit-bindgen 0.12. * Cut down on duplicate sync/async resource types (#1) * Allow calling `get-directories` more than once (#2) For now `Clone` the directories into new descriptor slots as needed. * Start to lift restriction of stdio only once (#3) * Start to lift restriction of stdio only once This commit adds new `{Stdin,Stdout}Stream` traits which take over the job of the stdio streams in `WasiCtxBuilder` and `WasiCtx`. These traits bake in the ability to create a stream at any time to satisfy the API of `wasi:cli`. The TTY functionality is folded into them as while I was at it. The implementation for stdin is relatively trivial since the stdin implementation already handles multiple streams reading it. Built-in impls of the `StdinStream` trait are also provided for helper types in `preview2::pipe` which resulted in the implementation of `MemoryInputPipe` being updated to support `Clone` where all clones read the same original data. * Get tests building * Un-ignore now-passing test * Remove unneeded argument from `WasiCtxBuilder::build` * Fix tests * Remove some workarounds Stdio functions can now be called multiple times. * If `poll_oneoff` fails part-way through, clean up properly. Fix the `Drop` implementation for pollables to only drop the pollables that have been successfully added to the list. This fixes the poll_oneoff_files failure and removes a FIXME. --------- Co-authored-by: Alex Crichton <alex@alexcrichton.com>

…es run out. (bytecodealliance#7114) * Cranelift: return programmatic error rather than panic when temporaries run out. Cranelift currently has a limit of `2^21` vregs per function body in VCode. This is a consequence of (performance-motivated) bitpacking of `Operand`s into `u32`s in regalloc2. As a result of this, it is possible to produce a function body that will fail to compile by running out of vreg temporaries during lowering. Currently, this results in a panic. Ideally, we would propagate the error upward and return it programmatically. This PR does that, with a "deferred error" mechanism. A cleaner solution would be to properly thread the `Result` types through all layers of lowering. However, that would require supporting `Result`s in ISLE, and that is a deeper language-design and `islec`-hacking question that I think we can tackle later if we continue to see motivating cases. The deferral works by returning a valid but bogus `ValueReg`s to the lowering rule, but storing the error and checking for it in the toplevel lowering loop. (Note that we have to return a bogus `v0` rather than `VReg::invalid()`, because the latter causes the `ValueRegs` to think there is no register provided.) This PR also includes a test at the Wasmtime level. Note that it takes ~22s to run on my (relatively fast) laptop, because it has to run until it runs out of VRegs in a debug build of the compiler. We could remove the test if we feel we're otherwise confident. Thanks to Venkkatesh Sekar for reporting this issue! The integration test uses one of the example inputs from the report. * Review feedback. * Handle alloc after a deferred error occurs. * Add uncommitted fix for previous.

* Fix compile warnings in preview1 adapter This isn't part of the "main build" so it's not inheriting the "deny warnings" flag passed in CI, so fixup some warnings which have leaked through over time. * Remove some usage of `UnsafeCell` in the adapter Move to abstractions such as `OnceCell` from the standard library as well as `RefCell` to avoid some unsafety. This shouldn't have any practical ramifications on the adapter, mostly just trying to help over-time maintenance. * Deny test-programs warnings in CI

* wasi: Add typed helpers to `Table` This will help cut down on the `Table*Ext` traits while retaining type safety. * Remove `TableNetworkExt` trait * Remove `TableTcpSocketExt` trait * Remove the `TableReaddirExt` trait * Remove `TableFsExt` trait This involed a fair bit of refactoring within the preview2-to-preview1 adapter to handle the new ownership of resources, but nothing major. * Remove `TableStreamExt` trait Additionally simplify some stream methods while I'm here. * Remove `TablePollableExt` trait * Fix tests * Fix some more tests * Use typed accessors for terminal-{input,output} * Remove dead code in `Table`

This appears to be failing, so temporarily disable it while the issues are worked out.

* c-api: Expose host memory creation Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Review feedback Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Fix invalid reference in docs Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Fix doxygen documentation Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> --------- Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

…ance#7121) * riscv64: Remove some stray whitespace * riscv64: Refactor int-to-float emission Use ISLE matching to generate different instructions rather than having a method deeper in the backend select which instruction to use. Additionally remove the usage of the `normalize_fcvt_to_int` helper. * riscv64: Refactor and simplify `lower_bmask` Take a `Value` as input to simplify 128-bit handling and additionally simplify the rules a bit with other helpers. * riscv64: Refactor `normalize_cmp_value` and `truthy_to_reg` Remove the normalization completely by folding it into callers and refactor `truthy_to_reg` to perform sign extension internally in addition to handling 128-vs-64 * riscv64: Remove 128-bit handling from extension helper This is only needed in a small handful of spots which can pretty easily handle the sign extension on their own, and this further simplifies sign extension to purely dealing with a single register. * riscv64: Remove destination type in extension This is always `$I64`, so no need to pass it around. * riscv64: Remove `ext_if_need` helper Fold 128-bit handling into callers as necessary * riscv64: Refactor `sext` to take a `Value` This commit refactors the `sext` helper to take a single `Value` as an argument rather than an `XReg Type` combo. This will enable, in future commits, to pattern-match the structure of the input and skip the sign extension if possible. For example an `icmp` result is already sign-extended. This involved moving some `lower_*` declarations directly into `lower.isle`. Additionally a special-case of lowering `brif` was removed which caused regressions in generated code due to new `and` instructions being generated to sign-extend the result of an `icmp`. Once `zext` is also migrated to this pattern this regression should be fixed, so this should just be a temporary state of affairs. * riscv64: Fold `lower_umulhi` helper into lower.isle * riscv64: Fold `lower_uadd_overflow` into lower.isle * riscv64: Fold `lower_popcnt` into `lower.isle` This slightly pessmizes codegen when the `zbb` extension isn't available due to a sign extension for smaller-than-64-bit types, but this is possible to fix in the future if necessary and helps simplify this in the meantime. Additionally this lifts the special cases for `has_zbb` directly into the lowering rule. * riscv64: Refactor clz and cls This commit refactors the lowering rules for these two CLIF instructions to move the sext/zext operations to the top rather than having them buried in lowering rules. This additionally moves them to `lower.isle` instead of `inst.isle`. This did result in some minor duplication but overall feels a bit better due to them being less coupled now. * riscv64: Move `gen_bswap` to `lower.isle` Additionally shuffle rules around to remove the dependence on `zext` because when this zero-extension happens it's known that many of the extension-related optimizations are not applicable. This does pessimize code slightly but it's hopefully not too bad given the current size of the bswap lowering. * riscv64: Change `zext` to taking a `Value` This is similar to a prior commit for `sext` and will enable future optimizations to skip the zext entirely if the structure of the value allows. * riscv64: Fold `extend` helper into `zext` and `sext` This commit folds the existing rules of the `extend` helper into the corresponding functions of `zext` and `sext`. They are the only callers of `extend` and this keeps the sign extension-related logic truly in one final resting place now at this point. A new matcher `val_already_extended` is additionally added to add more cases in a follow-up commit to enable skipping sign-extension operations based on the structure of the `Value` input. * riscv64: Optimize some sign extensions As the culmination of all prior commits, this commit adds a few cases where a sign extension is not necessary in the backend and can be skipped. For example extending a sign-extended value is not required because the backend always extends to the full register width. Additionally extending the result of an `icmp` is not necessary since that always produces a 0 or 1 value at the full register width.

This commit fixes a compatibility issue with Rust 1.70.0 on Windows targets. Rust 1.71.0 stabilized `AsSocket for Arc<T>` which is used here implicitly, so to compile successfully on 1.70.0, our current MSRV, a slight code change is required. Closes bytecodealliance#7127

* riscv64: Add `c.mul` * riscv64: Add `c.not` * riscv64: Add zbb and zba dependent compressed instructions * riscv64: Add `Zcb` loads and stores * riscv64: Restrict immediate encoding for halfword compressed stores and loads * riscv64: Reverse imm bits for bytewise compressed loads and stores

This commit removes the usage of `gen_icmp` in `uadd_overflow_trap`. The comparison previously done with an explicit comparison is now bundled directly into the conditional branch to go to the trap itself.

…7131) Currently any 32-bit constant can be materialized without a load from a constant pool on RISCV-64 but once constants start getting larger than this they're always loaded from the constant pool. This commit adds another special case for loading constants which appears to match what LLVM does which is to consider materializing a smaller constant and than shifting it left. This is done by chopping off all trailing zeros from an immediate and then testing if the immediate can be materialized as a 32-bit constant. This means that the current constant loading sequence can optionally be followed by a trailing `slli` instruction to shift the zeros back into the constant. This namely means that loading `i64::MIN` (1 << 63) no longer falls back to the constant pool.

This commit refactors how the `Pollable` resource is created in WASI. Previously this was created manually via two variants of a `Pollable` enum but this was somewhat error prone for a few reasons: * The more common representation, `TableEntry`, had boilerplate associated with it that had dynamic downcasts and such. This is now all hidden behind a more typed API. * Callers had to remember to use `push_child_resource` which is easy to forget. * The previous signature of the readiness check returned a `Result<()>` which made it accidentally easy for errors to be propagated into traps rather than being returned as a "last operation failed" error. This commit replaces the previous API with a single `subscribe` function which takes a `Resource<T>` and returns a `Resource<Pollable>`. The `T` type must implement a new trait, `Subscribe`, which indicates the asynchronous readiness check. This readiness check additionally returns `()` so it's no longer possible to accidentally return errors (or trap). This namely required refactoring the `HostOutputStream` trait and implementations. The trait itself now has a `check_write` method corresponding to to the same WASI function. The old `write_ready` function is now a convenience helper combo between `ready` and `check_write`. Concrete implementations are refactored to store errors discovered during `ready` to get communicated through `check-write` later on.

…in tests (bytecodealliance#7137)

…e#7136) * riscv64: Refactor implementation of `{u,s}{div,rem}` This commit's goal is to remove the usage of `gen_icmp` with division-related instructions in the riscv64 backend. I've opted for a slightly more verbose approach than what was present prior to make sign/zero extensions more precise and additionally enable some immediate-related optimizations. Divison/remainder by an immediate will no longer emit checks to see if a trap needs to be emitted. Instead only the raw `div`/`rem` instructions are emitted. Additionally a few minor sign extensions are now avoided such as the dividend in 32-bit remainders/division because only the low 32-bits are inspected. Finally, while I was here, I went ahead an added `has_m` guards to all these lowering rules. The M extension is always required with riscv64 right now and won't work if it's turned off, but I figure it's not too bad to add in terms of completeness for now. As to the meat of the commit, the check for trapping value as part of division now happens without `gen_icmp` but instead `gen_trapif` which enables avoiding materializing the comparison's value into a register. * Fix tests

* c-api: Add a feature for async Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Add support for async config Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Add support for calling async functions Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Add ability to yield execution of Wasm in a store Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Introduce wasmtime_linker_instantiate_async Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Support defining async host functions Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * gitignore: ignore cmake cache for examples Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * examples: Add example of async API in C Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Consolidate async functionality into a single place Put all the async stuff in it's own header and own rust source file Also remove the wasmtime_async_continuation_new function, users can just allocate it directly. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Make async function safe Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Remove wasmtime_call_future_get_results Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Simplify CHostCallFuture Move the result translation and hostcall_val_storage usage into an async function Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Simplify C continuation implementation Remove the caller, which means that we don't need another struct for the future implementation. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Improve async.h documentation Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Cleanup from previous changes Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * examples: Fix example Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Simplify continuation callback This gives more duality with calling an async function and also means that the implementation can pretty much mirror the sync version. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Fix async.h documentation Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Fix documentation for async.h Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Review feedback Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * examples: Downgrade async.cpp example to C++11 Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: initialize continuation with a panic callback Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * prtest:full Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> --------- Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>

…#7133) * Handle `lower_branch` consistently amongst backends This commit is a refactoring to consistently implement `lower_branch` among Cranelift's backends. Previously each backend had its own means of extracting labels and shuffling along information, and now there's prelude methods for all backends to access and use. This changes a few display impls but the actual meat of what's emitted shouldn't change amongst the backends. * Fix compile

* Allow backlog size to be set before initial listen * Move set_listen_backlog_size into example_body * Format code * Let cap_net_ext handle the default value. * retrigger checks * Got lost while pulling in changes from main.

Found in CI for bytecodealliance#7099 this commit updates the durations used in the `resolves_immediately` and `never_resolves` tests. For immediately resolving it should be ok to wait for a generous amount of time since the timeout is only a protection against tests hanging. For `never_resolves` it should be ok to wait a short amount of time and any failure there will show up as a spurious failure.

* c-api: Support InstancePre Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Consolidate all the instance_pre functionality Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Add async instantiate support to pre instances Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Add star to comment for doxygen prtest:full Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> --------- Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

) * riscv64: Add codegen tests for fcmp and fcmp branches * riscv64: Refactor and simplify some branches/fcmp This commit is aimed at simplifying the layers necessary to generate a branch for a `brif` statement. Previously this involved a `lower_cond_br` helper which bottomed out in emitting a `CondBr` instruction. The intention here was to cut all that out and emit a `CondBr` directly. Along the way I've additionally taken the liberty of simplifying `fcmp` as well. This moves the "prefer ordered compares" logic into `emit_fcmp` so it can benefit the `fcmp` instruction as well. This additionally trimmed some abstractions around branches which shouldn't be necessary any longer.

* Fix race condition in `AsyncWriteStream` This commit fixes a synchronization issue with `AsyncWriteStream` where the writer task would get wedged and not make any further progress. The underlying reason is that `notify_waiters` was used which did not buffer its notification, meaning that if a waiter wasn't actually waiting then it would miss the notification and block forever on the next call to `notified`. By using `notify_one` instead this will buffer up a single notification for the other end for when it gets to waiting for work. Additionally this removes creation of the `Notified` future ahead-of-time as that's no longer necessary. * Fix http bodies too

…ance#7141) This switches the `extractor` to a `pure partial` constructor to be used in `if-let` instead of the extraction position, which while not quite as ergonomic does enable matching more constant values in more locations.

…#7090) * streams.wit: delete stream-status, rename write-error to stream-error, transform all input-stream methods * preview2: use StreamError throughout input-stream * preview2: passes cargo test * preview1: fixes for input-stream stream-error. * wasmtime-wasi-http: fixes for HostInputStream trait changes * component adapter: fixes for input-stream changes * test programs: fixes for input-stream * component adapter: handle StreamError::Closed in fd_read * sync wit definitions to wasi-http * fix!! * preview1: handle eof and intr properly prtest:full * Fix preview1 stdin reading * Touch up stream documentation --------- Co-authored-by: Alex Crichton <alex@alexcrichton.com>

…nce#7109) * Implement the `wasi:sockets/ip-name-lookup` interface This commit is an initial implementation of the new `ip-name-lookup` interface from the `wasi-sockets` proposal. The intention is to get a sketch of what an implementation would look like for the preview2 release later this year. Notable features of this implementation are: * Name lookups are disabled by default and must be explicitly enabled in `WasiCtx`. This seems like a reasonable default for now while the full set of configuration around this is settled. * I've added new "typed" methods to `preview2::Table` to avoid the need for extra helpers when using resources. * A new `-Sallow-ip-name-lookup` option is added to control this on the CLI. * Implementation-wise this uses the blocking resolution in the Rust standard library, built on `getaddrinfo`. This doesn't invoke `getaddrinfo` "raw", however, so information such as error details can be lost in translation. This will probably need to change in the future. Closes bytecodealliance#7070 * Validate the input domain name. * Use a crate-level helper for `spawn_blocking` --------- Co-authored-by: Dave Bakker <github@davebakker.io>

* Further optimize RISCV constant loading * Simplify existing rule * Fix * Bless missing tests * Use right signedness for Imm20

…e#7143) * Refactor HTTP tests to be less flaky and more robust This commit refactors the wasi-http tests we have in this repository with a few changes: * Each test now uses a dedicated port for the test. The port 0 is bound and whatever the OS gives is used for the duration of the test. This prevents tests from possibly interfering with each other. * Server spawning is abstracted behind a single `Server` type now which internally has http1/2 constructors. * Timeouts for server shutdown are removed since they were failing in CI and are likely best handled for each test individually if necessary. As a minor cleanup the `tokio` usage in guest programs was removed as it was boilerplate in this case. This shouldn't affect the runtimes of tests, however. * Remove unused import * Don't panic from worker thread * Improve failure error message prtest:full * Fix some windows-specific issues * Fix async tests * Relax test assertion string

…iance#7150) This allows embedders to implement those traits themselves rather than be restricted to using the built-in implementations. Fixes bytecodealliance#7149 Signed-off-by: Joel Dice <joel.dice@fermyon.com>

…e#7134) * Refactor option handling in the `run` and `serve` commands * Use a real thread to manage epoch increments * Move module/component loading into src/common.rs * Require wasi-http and component-model to not be disabled in serve

…alliance#7145) * riscv64: Add codegen tests for min/max * riscv64: Remove unused `Type` payload from `Select` * riscv64: Add codegen test for brif * riscv64: Consolidate conditional moves into one instruction This commit removes the `IntSelect` and `SelectReg` pseudo-instructions from the riscv64 backend and consolidates them into the `Select` instruction. Additionally the `Select` instruction is updated to subsume the functionality of these two previous instructions. Namely `Select` now operates with `ValueRegs` to handle i128 and additionally takes an `IntegerCompare` as the condition for the conditional branch to use. This commit touches a fair bit of the backend since conditional selection of registers was used in quite a few places. The previous `gen_select_*` functions are replaced with new typed equivalents of `gen_select_{xreg,vreg,freg,regs}`. Furthermore new `cmp_*` helpers were added to create `IntegerCompare` instructions which sort-of match conditional branch instructions, or at least the pnemonics they use. Finally since this affected the `select` CLIF instruction itself I went ahead and did some refactoring there too. The `select` instruction creates an `IntegerCompare` from its argument to use to generate the appropriate register selection instruction. This is basically the same thing that `brif` does and now both go through a new helper, `lower_int_compare`, which takes a `Value` and produces an `IntegerCompare` representing if that value is either true or false. This enables folding an `icmp` or an `fcmp`, for example, directly into a branching instruction. * Fix a test * Favor sign-extension in equality comparisons

) The usage of `?` accidentally caused the extra information to not get printed, so avoid the use of `?` until the extra error info has been attached.

Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>

) This commit moves the optimizations from `sextend` for various shapes of instructions into the `sext` helper. This enables these optimizations to kick in when generating other instructions which require sign-extensions such as `icmp` and `brif`. This additionally fixes an issue introduced in bytecodealliance#7121 where uextend-of-sextend wasn't translated correctly and the outer `uextend` was mistakenly omitted.

* Update release notes for 14.0.0 Wow quite a bit happened! * Review comments

* Bump regalloc2 to 0.9.3 * Update filetests for the regalloc bump

…#7161) * Remove `own` and `borrow` annotations from wasi-http * Fix the tests, as borrows are now needed * Don't delete headers that we're borrowing

* Change `bindgen!`'s trappable error to take an input type This commit removes the generated type for `trappable_error_type` from the `bindgen!` macro to allow users to provide a type instead. This works similarly as before with new conversion functions generated in traits which are used to convert the custom error into the ABI representation of what a WIT world expects. There are a few motivations for this commit: * This enables reducing the number of errors in play between async/sync bindings by using the same error type there. * This avoids an auto-generated type which is more difficult to inspect than one that's written down already and the source can more easily be inspected. * This enables richer conversions using `self` (e.g. `self.table_mut()`) between error types rather than purely relying on `Into`. This is important for bytecodealliance#7017 where an error is going to be inserted into the table as it gets converted. * Fix tests * Update WASI to use new trappable errors This commit deals with the fallout of the previous commit for the WASI preview2 implementation. The main changes here are: * Bindgen-generated `Error` types no longer exist. These are replaced with `TrappableError<T>` where type aliases are used such as ```rust type FsError = TrappableError<wasi::filesystem::types::ErrorCode>; ``` * Type synonyms such as `FsResult<T>` are now added for more conveniently writing down fallible signatures. * Some various error conversions are updated from converting to the old `Error` type to now instead directly into corresponding `ErrorCode` types. * A number of cases where unknown errors were turned into traps now return bland error codes and log the error instead since these aren't fatal events. * The `StreamError` type does not use `TrappableError` since it has other variants that it's concerned with such as a `LastOperationFailed` variant which has an `anyhow::Error` payload. * Some minor preview1 issues were fixed such as trapping errors being turned into normal I/O errors by accident. * Add an `error` resource to WASI streams This commit adds a new `error` resource to the `wasi:io/streams` interface. This `error` resource is returned as part of `last-operation-failed` and serves as a means to discover through other interfaces more granular type information than a generic string. This error type has a new function in the `filesystem` interface, for example, which enables getting filesystem-related error codes from I/O performed on filesystem-originating streams. This is plumbed through to the adapter as well to return more than `ERRNO_IO` from failed read/write operations now too. This is not super fancy just yet but is intended to be a vector through which future additions can be done. The main thing this enables is to avoid dropping errors on the floor in the host and enabling the guest to discover further information about I/O errors on streams. Closes bytecodealliance#7017 * Update crates/wasi-http/wit/deps/io/streams.wit Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com> * Update wasi-http wit too * Remove unnecessary clone --------- Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>

To prepare this commit I've run `WIT_REQUIRE_SEMICOLONS=1 ./ci/run-tests.sh` locally which configures `wit-parser` to generate an error for missing semicolons in WIT files. This led me to add quite a few semicolons in quite a few places in what is going to be the first of a few batches of semicolons. CI checks for this cannot be added just yet because the wasi-nn spec is a submodule which needs to be updated with semicolons before this repository can require semicolons. Nevertheless that doesn't stop us from using semicolons in the meantime (yay gradual rollout of changes!) so I figure this would be good to get in sooner rather than later.

The task was disabled due to a failing checksum. Since then, the `install-openvino-action` has improved and now _almost_ has cross-platform support. This change just restores the status quo in expectation that bytecodealliance#6895 will significantly improve the testing story. prtest:full

* chore: convert wasi http cli test to component * fix(wasmtime-cli): wrong linker in wasi http * fix(wasmtime-cli): fix multiple functions added to linker We will allow shadowing in linker when using WASI HTTP in order to prevent the following error: > import of `wasi:clocks/wall-clock` defined twice * chore: update to latest io streams wit definition * chore: rename main function of core module prtest:full * fix(wasmtime-cli): add only http functions to linker * chore: add test building http request and response * chore: disable temporarily wasi http test

This commit updates CI to require semicolons in all WIT files and parsed WIT documents. Some minor updates were required in existing WIT files and the wasi-nn proposal was additionally updated to its latest version with semicolons. The wasi-nn update brought some minor changes to the WIT which required some minor changes here as well.

…liance#7177)

This should be wasmtime_instance_pre_t Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* winch(x64): Add support for table instructions This change adds support for the following table insructions: `elem.drop`, `table.copy`, `table.set`, `table.get`, `table.fill`, `table.grow`, `table.size`, `table.init`. This change also introduces partial support for the `Ref` WebAssembly type, more conretely the `Func` heap type, which means that all the table instructions above, only work this WebAssembly type as of this change. Finally, this change is also a small follow up to the primitives introduced in bytecodealliance#7100, more concretely: * `FnCall::with_lib`: tracks the presence of a libcall and ensures that any result registers are freed right when the call is emitted. * `MacroAssembler::table_elem_addr` returns an address rather than the value of the address, making it convenient for other use cases like `table.set`. -- prtest:full * chore: Make stack functions take impl IntoIterator<..> * Update winch/codegen/src/codegen/call.rs Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com> * Remove a dangling `dbg!` * Add comment on branching --------- Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>

…ce#7165) * WIP veriwasm 2.0 Co-Authored-By: Chris Fallin <cfallin@fastly.com> * PCC: successfully parse some simple facts. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: plumb facts through VCode and add framework on LowerBackend to check them. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: code is carrying some proofs! Very simple test-case. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: add a `safe` flag for checked memory accesses. * PCC: add pretty-printing of facts to CLIF output. * PCC: misc. cleanups. * PCC: lots of cleanup. * Post-rebase fixups and some misc. fixes. * Add serde traits to facts. * PCC: add succeed and fail tests. * Review feedback: rename `safe` memflag to `checked`. * Review feedback. --------- Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

…nce#7170) * Update syntax for `trappable_error_type` in `bindgen!` Following through on a suggestion from bytecodealliance#7152 this makes the syntax a bit more consistent with the rest of `bindgen!` today. This additionally updates the documentation on the `bindgen!` macro itself. * Update parsing to afford for versions

This commit fixes a bug in the `bindgen!` macro where when faced with multiple packages that differ only in version number invalid bindings were generated. The fix here is to add version number information to package module names if necessary in situations such as this. This required some refactoring internally to have a single source of truth for what the name of a module should be and avoid having it implicitly calculated in two locations.

…ytecodealliance#7180) * PCC: add semantics for core add/shift/extend/amode ops on AArch64. This PR adds verification of facts on values produced by adds, shifts, and extends on AArch64, handling the various combination instructions (adds with builtin extends or shifts, for example), and also adds verification of all addressing modes, including those with builtin extends and shifts. It also splits the test suite into"succeed" and "fail" sets, and provides cases that PCC should catch. * Review feedback.

) This prevents confusing label locations when there are multiple functions with the same Name.

* Use Rustix::Errno to unify error code mapping. * Clarify Connect failure state * Allow accept() to return transient errors. The original provision was added to align with preview3 streams that may only fail once. However, after discussing with Dan Gohman, we came to the conclusion that a stream of result<> could do the trick fine too. Fixes: WebAssembly/wasi-sockets#22 * Fold `ephemeral-ports-exhausted` into `address-in-use` There is no cross-platform way to know the distinction between them * Clarify `local-address` behavior on unbound socket * Remove `concurrency-conflict` clutter, and just document it to be always possible. * Simplify state errors. They were unnecessarily detailed and mostly have no standardized equivalent in POSIX, so wasi-libc will probably just map them all back into a single EOPNOTSUPP or EINVAL or ... EISCONN/ENOTCONN can be derived in wasi-libc based on context and/or by checking `remote-address`. For example, `shutdown` can only be called on connected sockets, so if it returns `invalid-state` it can be unambiguously mapped to ENOTCONN. * Document that connect may return ECONNABORTED * Remove create-tcp/udp-socket not supported errors. These stem from back when the entire wasi-sockets proposal was one big single thing. In this day and age, when an implementation doesn't want to support TCP and/or UDP, it can simply _not_ implement that interface, rather than returning an error at runtime. * Simplify "not supported" and "invalid argument" error cases. There is a myriad of reasons why an argument might be invalid or an operation might be not supported. But there is few cross platform consistency in which of those error cases result in which error codes. The error codes that have been removed were fairly specific, but: - Were still missing error cases. So additional error codes would have needed to be created. - Implementations would have to bend over backwards to make it work cross platform, especially beyond just Win/Mac/Linux. - Didn't all have an equivalent in POSIX, so they would map back into a generic EINVAL anyways. * Move example_body out of lib.rs into its own test-case make room for other tests. * Refactor TCP integration tests: - Ad-hoc skeleton implementation of resources. - Add blocking wrappers around async operations. * Fix get/set_unicast_hop_limit on Linux * Test TCP socket states * Keep track of address family ourselves. Because I need the family for input validation. And the resulting code is more straightforward. * Add more tests and make it work on Linux * Fix Windows * Simplify integration tests. All platforms supported by wasmtime also support dualstack sockets. * Test ipv6_only inheritence * Test that socket options keep being respected, even if listen() has already been called * cargo fmt * Duplicate .wit changes to wasi-http * prtest:full * Fix BSD behavior of SO_SNDBUF/SO_RCVBUF * fmt * Fix type error * Got lost during merge * Implement listen backlog tests * Manually inherit buffer size from listener on MacOS. I suspect that these changes apply to any BSD platform, but I can't test that. * Keep track of IPV6_V6ONLY ourselves. - This provides cross-platform behaviour for `ipv6-only` - This eliminates the syscall in `validate_address_family` * Reject IPv4-compatible IPv6 addresses. * Remove intermediate SystemError trait * Fix ambiguous .into()'s * Fix IPV6_UNICAST_HOPS inheritance on MacOS

* riscv64: Use `CALL_PLT` relocation for all calls * riscv64: Set the ELF RVC flag only when the C extension is enabled * riscv64: Use PC Relative Relocations for all Calls * riscv64: Add support for PIC symbols

* feat(wasi-sockets): implement UDP This is based on TCP implementation Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> * refactor(wasi-sockets): simplify UDP implementation This introduces quite a few changes compared to TCP, which should most probably be integrated there as well Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> * feat(wasi-sockets): store UDP connect address in state Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> * fix(wasi-sockets): avoid `shutdown` on `drop` Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> * Remove explicit bind * Simplify start_connect&finish_connect. On UDP sockets, `connect` never blocks. * Move UDP test to single file, similar to `tcp_sample_application.rs` * Update UDP tests --------- Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> Co-authored-by: Dave Bakker <github@davebakker.io>

Continuation of bytecodealliance#7172 where imports and export _definitions_ are properly namespaced by version by the usage of exports was not.

Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>

If the input operand is an `icmp` or an `fcmp` there's no need to use `snez` since the output value is already guaranteed to be zero or one.

* Refactor the test-programs test suite This commit is a large refactoring that reorganizes `test-programs` and how we tests wasms in Wasmtime. Often writing tests requires complicated interactions with the guest which can't be done via hand-written `*.wat` syntax and requires a compiler to get engaged. For this purpose Wasmtime currently has the `crates/test-programs/*` test suite which builds files from source and then runs the tests. This has been somewhat cumbersome in the past though and it's not been easy to extend this over time, so this commit attempts to address this. The scheme implemented in this PR looks like: * All wasm test programs live in `crates/test-programs/src/bin/*.rs`. All of them, no exceptions. * Wasm tests have shared support located at `crates/test-programs/src/lib.rs` and its submodules, such as bindings generation for WASI. * Wasm tests are built by a new `crates/test-programs/artifacts` crate. This crate compiles modules and additionally creates components for all test programs. The crate itself only records the path to these outputs and a small amount of testing support, but otherwise doesn't interact with `wasmtime`-the-crate itself. * All tests in `crates/test-programs/tests/*.rs` have moved. For example wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy tests of wasi-common now live at `crates/wasi-common/tests/*.rs`. Modern tests for preview2 live at `crates/wasi/tests/*.rs`. * Wasm tests are bucketed based on their filename prefix. For example `preview1_*` is tested in wasi-common and wasmtime-wasi. The `preview2_*` prefix is only tested with wasmtime-wasi, however. * A new `cli_*` prefix is used to execute tests as part of `tests/all/main.rs`. This is a new submodule in `tests/all/cli_tests.rs` which executes these components on the command line. Many old "command" tests were migrated here. * Helper macros are generated to assert that a test suite is run in its entirety. This way if a `preview1_*` test is added it's asserted to get added to both wasi-common and wasmtime-wasi in the various modes they run tests. Overall this moved a number of tests around and refactored some edges of the tests, but this should not lose any tests (except one that wasn't actually testing anything). Additionally the hope is that it's much easier to add tests in the future. The process is to add a new file in `crates/test-programs/src/bin/*.rs` named appropriately. For example a preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When building the test suite an error is generated in the appropriate module then of "please write a test here", and then a test is written in the same manner as the other tests in the module. * Remove no-longer-needed fetches prtest:full * I'm worried wasi is running low on semicolons * Add the WASI target in all CI actions * Add unknown-unknown target on all CI builders too * Fix building test artifacts under miri Need to avoid wrappers for these cross-compiled targets * Break circular dependency for packaging Don't use the workspace dep for `wasmtime-wasi` since it injects a version, instead use a `path = '..'` dependency to fool Cargo into dropping the dependency during the package phase. * Fix some merge conflicts with tests * Fix rebase for new tests * Remove stray comment * Fix some flaky tests * Fix network tests in synchronous mode This commit is an attempt to fix some networking tests in synchronous mode in our test suite. Currently networking tests don't actually run in synchronous mode on CI which is why no failures have been surfaced yet, but the refactoring in bytecodealliance#7182 is going to start doing this. Currently the `udp_sample_application.rs` test blocks infinitely in synchronous mode for me locally, most of the time. This appears to be an interaction between how Tokio handles readiness and how we're entering the event loop. We're effectively entering the Tokio event loop with a future that's always ready which ends up starving Tokio of otherwise performing its background work such as updating flags for readiness of reading/writing. The fix here is to add a yield at the start of an `in_tokio` block which is used in synchronous mode. This is a kludge fix but the intention is to enable Tokio to have a chance to update readiness flags and process events from epoll/kqueue/etc. An additional fix to this issue is WebAssembly/wasi-sockets#64 where the test is waiting on `READABLE` or `WRITABLE`, but in this specific case it should only wait on `READABLE`. If it waited on just this then that would also fix this issue. Nevertheless having a `yield_now` is expected to have little-to-no overhead and otherwise fix this edge case of an always-ready future. * Fix passing empty arguments on the CLI * Add another blocking accept * Update crates/test-programs/src/bin/api_proxy.rs Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com> --------- Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>

* aarch64: Implement TLSDESC for TLS GD accesses * aarch64: Box ExternalName to avoid growing the `Inst` enum

Since bytecodealliance#7152 the error types of sync and async are the same so much of the prior `Ok`-wrapping isn't necessary any more.

…liance#7207)

* Move around some internal methods and internalize what we can. * Rename all `*_resource` methods to drop the suffix. * Lean more into "wrong table usage is a trap" and remove the `delete` logic which re-inserted a parent link, if needed, on downcast errors.

* Update MSRV to 1.71.0 With the 1.73.0 release today update our MSRV to a one-higher version. This will cause 1.73.0 to be tested in CI. * Fix a warning on windows * Don't store tables as `&'static mut` Use `SendSyncPtr` instead (a small wrapper around `NonNull`) which appeases MIRI.

* Introduce StackMemory and StackMemoryCreator This allows custom implementations of stack memory to be plugged into the async functionality for wasmtime. Currently, stacks are always mmapped, and this custom allocator allows embedders to use any memory they would like. The new APIs are also exposed in the C api. This has no effect on windows, as our hands are tied with fibers there. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * Add test for custom host memory Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * fix allocator test Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * Address review comments Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * Fix lint warnings prtest:full Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * fix windows lint warning prtest:full Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> --------- Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* riscv64: Fix encoding for `c.addi4spn` * riscv64: Add a few more `c.addi4spn` tests

@cfallin

When troubleshooting deadlocks in WebAssembly modules, it is important to understand which `wait` instructions are still pending a `notify`. It would be nice to have some kind of `--warn-deadlock-after=1s` flag available that would poll the parking lot for `wait`s hanging past the time limit, but I realized the real value would be to tie the `wait` instruction (through CLIF) to the original source code, if debug information were available. This did not seem to be entirely feasible, since CLIF loses the original Wasm source context (is this true?) and I was not confident that we would be able to use `addr2line` to map from Wasm instructions to source (e.g., see @cfallin's [issue](gimli-rs/addr2line#265)). Instead, this change simply logs each valid `wait` and `notify` execution, leaving it to the user to figure out which one is hanging (should not be too difficult) and how to map this back to their source code (more difficult).

* PCC: define memory types and add some examples in filetests (no parsing yet). * PCC: add a notion of memory types. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * Transition `points_to` to a `memory` (static memory) memory-type. * Review feedback. --------- Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* Update examples-profiling-perf.md Correct spelling * Update examples-profiling-perf.md Correct spelling 2.

* Update cli-install.md Correct grammar. * Update contributing-architecture.md Correct grammar.

* c-api: expose memory_init_cow Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: fix type in async.h I really wish these were autogenerated :/ Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> --------- Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* delete redundant cli test cli_panic is almost exactly the same as cli_exit_panic, so delete the first one * split stream pollable lifetimes test into two preview2 tests

* wit: delete wasi logging package missed this in a code review * command extended: use include keyword, delete import of logging * sync http wit definitions

…d, and add a basic vmctx/memory example. (bytecodealliance#7231) * PCC: add checking of struct field types on loads and stores. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: check that blockparams uphold facts appropriately. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: infer facts based on types' value ranges; add a mock vmctx example. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * Fix formatting. * Add two fixes required by new default facts: - Allow any fact at all to subsume a trivially-true range fact that states a bitslice is within the maximum range for its width. - Clamp the maximum value of a bitslice at the maximum value for the slice's width. --------- Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

This can be useful if you want change to set the features wasmtime builds with in CMake Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* Update to arbitrary 1.3.1 And use workspace dependencies for arbitrary. * Prune cargo vet's supply-chain files This is the mechanical changes made by running `cargo vet prune` which was suggested to me when I ran `cargo vet`.

…codealliance#7228) * winch: Add known a subset of known libcalls and improve call emission This change is a follow up to: - bytecodealliance#7155 - bytecodealliance#7035 One of the objectives of this change is to make it easy to emit function calls at the MacroAssembler layer, for cases in which it's challenging to know ahead-of-time if a particular functionality can be achieved natively (e.g. rounding and SSE4.2). The original implementation of function call emission, made this objective difficult to achieve and it was also difficult to reason about. I decided to simplify the overall approach to function calls as part of this PR; in essence, the `call` module now exposes a single function `FnCall::emit` which is reponsible of gathtering the dependencies and orchestrating the emission of the call. This new approach deliberately avoids holding any state regarding the function call for simplicity. This change also standardizes the usage of `Callee` as the main entrypoint for function call emission, as of this change 4 `Callee` types exist (`Local`, `Builtin`, `Import`, `FuncRef`), each callee kind is mappable to a `CalleeKind` which is the materialized version of a callee which Cranelift understands. This change also moves the creation of the `BuiltinFunctions` to the `ISA` level given that they can be safely used accross multiple function compilations. Finally, this change also introduces support for some of the "well-known" libcalls and hooks those libcalls at the `MacroAssembler::float_round` callsite. -- prtest:full * Review comments * Remove unnecessary `into_iter` * Fix remaining lifetime parameter names

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

…7239) * wasmtime cli: install tracing-subscriber, listen to WASMTIME_LOG, log to stderr * enable tracing colors * audit supply chain for tracing subscriber

In bytecodealliance#7220, we began logging the arguments of WebAssembly `wait` and `notify` as these instructions are executed. This change prints the addresses using hexadecimal, which is simply more convenient when dealing with addressing.

…iance#7225) * fuzzgen: Add Alias Analysis Memflags to Loads and Stores * fuzzgen: Fix typo * fuzzgen: Add note about stack_slot sorting

* riscv64: Constants are always sign-extended Skip generating extra sign-extension instructions in this case because the materialization of a constant will implicitly sign-extend into the entire register. * riscv64: Rename `lower_int_compare` helper Try to reserve `lower_*` as taking a thing and producing a `*Reg`. Rename this helper to `is_nonzero_cmp` to represent how it's testing for nonzero and producing a comparison. * riscv64: Rename some float comparison helpers * `FCmp` -> `FloatCompare` * `emit_fcmp` -> `fcmp_to_float_compare` * `lower_fcmp` -> `lower_float_compare` Make some room for upcoming integer comparison functions. * riscv64: Remove `ordered` helper This is only used by one lowering so inline its definition directly. * riscv64: Remove the `Icmp` pseudo-instruction This commit is the first in a few steps to reimplement bytecodealliance#6112 and bytecodealliance#7113. The `Icmp` pseudo-instruction is removed here and necessary functionality is all pushed into ISLE lowerings. This enables deleting the `lower_br_icmp` helper in `emit.rs` as it's no longer necessary, meaning that all conditional branches should ideally be generated in lowering rather than pseudo-instructions to benefit from various optimizations. Currently the lowering is the bare-bones minimum to get things working. This involved creating a helper to lower an `IntegerCompare` into an `XReg` via negation/swapping args/etc. In generated code this removes branches and makes `icmp` a straight-line instruction for non-128-bit arguments. * riscv64: Remove an unused helper * riscv64: Optimize comparisons with 0 Use the `x0` register which is always zero where possible and avoid unnecessary `xor`s against this register. * riscv64: Specialize `a < $imm` * riscv64: Optimize Equal/NotEqual against constants * riscv64: Optimize LessThan with constant argument * Optimize `a <= $imm` * riscv64: Optimize `a >= $imm` * riscv64: Add comment for new helper * Use i64 in icmp optimizations Matches the sign-extension that happens at the hardware layer. * Correct some sign extensions * riscv64: Don't assume immediates are extended * riscv64: Fix encoding for `c.addi4spn` * riscv64: Remove `icmp` lowerings which modify constants These aren't correct and will need updating * Add regression test * riscv64: Fix handling unsigned comparisons with constants --------- Co-authored-by: Afonso Bordado <afonsobordado@az8.co>

Some things I noticed from bytecodealliance#7239 which are very much not critical but I figure might be nice-to-haves: * Move the logging configuration to the `wasmtime-cli-flags` crate with the other logging configuration to keep it in one place. * Remove `pretty_env_logger` since `tracing-subscriber` probably supplants it. * Don't explicitly inherit env vars in tests since that happens automatically with `Command`.

…nce#7253) Otherwise the CLI no longer prints `log`-based logging messages.

This commit removes a clause in `register_module` which was preventing `perf` profiling the first function in a module because it was located at address 0. This check is relatively dated at this point and I think I originally added it in error (and/or it was copied from somewhere else), so this commit removes it.

* Support set_fuel in store APIs Fixes: bytecodealliance#5109 Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * rename set_fuel to reset_fuel To make it more clear that consumed fuel is being reset. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * update out of date documentation for fuel in C API Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> --------- Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

This commit optimizes some `select_spectre_guard` patterns which are frequently generated with the wasm backend when dynamic bounds checks are enabled by pattern-matching when one of the values being selected is zero. This enables shaving off a few instructions which can be constant folded away.

* cranelift: Add egraph rules for `bswap` WebAssembly doesn't currently have a byte-swapping instruction so this commit pattern matches what LLVM currently generates to simplify to a single `bswap` instruction in CLIF which can be lowered within each backend to respective native instructions. * Use an `optimize` instead of `compile` test

…liance#7251) * riscv64: Refactor `rotl` rules Move from `inst.isle` to `lower.isle` since it's the only caller, reorganize the rules to be a bit cleaner, add immediate shifting specializations. * riscv64: Refactor `rotr` lowerings Same as the prior `rotl` lowerings, move the rules to `lower.isle` and additionally add constant rules. * Fix shift-by-128 * Remove empty comments

* wit: splice and blocking-splice borrow input-stream. delete forward. * minimum viable implementation of splice and blocking splice * add cli test for splice * sync wits * improve wit docs for splice * code golf

@fitzgen

This removes the need for the awkward "max-range fact is subsumed by anything" rule noted by @fitzgen in [this comment](bytecodealliance#7231 (comment)). It also makes checking a little more efficient and logically clear, as only the facts that the frontend/producer added are verified, rather than all default facts as well.

) * riscv64: Implement the `bclr` instruction * riscv64: Implement `bext{,i}` lowerings * riscv64: Add support for `binv{,i}` * riscv64: Add support for `bset` instruction * riscv64: Add bclr lowerings for swapped arguments * riscv64: Add bclr lowerings for i8/i16

Some notable changes here are: * The `wasm-tools` crates have enabled the `relaxed-simd`, `threads`, and `multi-memory` proposals by default. For now I've left these disabled-by-default in Wasmtime to get enabled in a future PR. * The `wast` crate has support for parsing `thread` and `wait` constructs from the `threads` proposal for WebAssembly. They're left unimplemented for now and return errors. This will get filled in in a future update.

…e#7268) This was mistakenly left out of my recent refactor for remainder/division. Closes bytecodealliance#7245

…codealliance#7263) This is needed for soundness when verifying accesses to memtype fields: it's not enough to know that we're accessing an offset in `0` up to `field_offset` inclusive, we need to know the access is actually to `field_offset`. The simplest change that validates this turned out to be the most general one: making ranges two-sided rather than one-sided. The transform is *mostly* mechanical, but a few new tests verify that ranges are updated on both sides, and some fail-tests verify that "fuzzily imprecise" pointers to struct fields fail to validate.

…ytecodealliance#7274) * PCC: add facts to global values, parse and print them. No verification yet. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: propagate facts on GV loads and check them. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: support propagating facts on iteratively-elaborated GVs as well. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: fix up Wasmtime uses of GVs after refactors to memflags handling. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: working end-to-end for static memories! Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: add toplevel Wasmtime option `-C enable-pcc=y`. * Fix filetests build. * Review feedback, and blessed test updates due to GV legalization changes. --------- Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* riscv64: Refactor `bnot` lowering rules Move them to `lower.isle` and clean them up a bit too. * riscv64: Add support for xnor

This turns out to be quite simple: the fundamental operation during egraph-based optimization is to *merge* eclasses, which is an assertion that their value is equal. Since the values of either side of the merge are equal, a fact about one side is a fact about the other, and vice-versa. Since we only support *one* fact at most per value, we can't take the union of all facts; instead, if one side is missing a fact, or if both sides have exactly the same fact, we keep that one; otherwise we go to a special "conflict" fact that can't support any check. This is edging closer to a fact-lattice, but I opted not to go there with a full meet-function merge yet, for simplicity. (It's a little complex because of the "minimum fact" we can know about a value based on its type -- if we're going to do something better, I think we should account for that too.) In any case, that complexity mostly isn't needed, because the two cases that happen in reality are (i) opt rules rewrite to a new node, which will have no facts at all, so facts just propagate; or (ii) GVN merges two values in the input program into one, but if both are the same value, in practice the Wasm PCC frontend (for example) should be producing the same facts on both values, so the merge is trivial.

…liance#7285) * Enable threads, multi-memory, and relaxed-simd by default This commit enables by default three proposals that were advanced to stage 4 in the proposal process at last week's in-person CG meeting. These proposals are: * `threads` - atomic instructions and shared memory * `multi-memory` - the ability to have more than one linear memory * `relaxed-simd` - new simd instructions that may differ across platforms These proposals are now all enabled by default in Wasmtime. Note that they can still be selectively disabled via `Config` options. * Fix an x64-specific test

* c-api: Remove reexport of wasmtime crate This is a follow-up to bytecodealliance#6765 to remove this reexport since it was originally added to use both the C API and the `wasmtime` crate in the same downstream crate, but this should be possible through Cargo with: [dependencies] wasmtime = "13" wasmtime-c-api = { version = "13", package = "wasmtime-c-api" } and that way `wasmtime::*` is available as well as `wasmtime_c_api::*` * Add `pub use wasmtime;`

* Remove usage of `BTreeMap` for compiler flags No need for a datastructure here really, a simple list with static strings works alright. * Fix winch compile and a warning * Fix test compile

This'll have some knock-on effects when applied to bytecodealliance#7272 but that's left for a future PR. This one's just about getting the deps in.

* Put versions in all WASI WIT files This commit starts exercising the versioning feature of WIT by ensuring that all WASI descriptions have a version associated with them. The version chosen is 0.2.0 which reflects the upcoming "preview 2" release where in theory 0.1.0 was claimed by preview1. This is intended to stay as 0.2.0 for now and we'll determine how best to update these numbers in the future once preview2 is released. Closes bytecodealliance#7171 * Allow omitting versions in `with` keys As a convenience for now this enables omitting the version of an interface from a `with` key. This has a risk of not working well if two packages are present and one has a version and one doesn't, but that's left as a PR to fix in the future as the benefit of avoiding repetition seems good for now. * Allow omitting versions in trappable_error_types * Use 0.2.0-rc-2023-10-18 as a version number * More test fixes * Fix another test

…iance#7288) This adds a new `send_request` method to `WasiHttpView`, allowing embedders to override the default implementation with their own if the desire. The default implementation behaves exactly as before. I've also added a few new `wasi-http` tests: one to test the above, and two others to test streaming and concurrency. These tests are ports of the `test_wasi_http_echo` and `test_wasi_http_hash_all` tests in the [Spin](https://github.com/fermyon/spin) integration test suite. The component they instantiate is likewise ported from the Spin `wasi-http-rust-streaming-outgoing-body` component. Fixes bytecodealliance#7259 Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* Implement support for `thread` in `*.wast` tests This commit implements support for `thread` and `wait` in `*.wast` files and imports the upstream spec test suite from the `threads` proposal. This additionally and hopefully makes it a bit easier to write threaded tests in the future if necessary too. * Fix compile of fuzzing

* riscv: Add rule for vmerge.vim Additionally add an imm5_from_value extractor. Closes bytecodealliance#7189 * riscv: Add tests for vmerge.vim Closes bytecodealliance#7189 * riscv: format tests * riscv: format scalar_to_vector rule * riscv: add runtests and rm duplicate definition

Fixes a fuzz test case discovered last night

With updates to `wit-parser` and `wit-bindgen` some of these versions are now optional and can be omitted.

…#7296) While valid in WIT I keep finding it jarring to have different indentation in `wasi:sockets`. Additionally all other WASI WITs are using four spaces or two, so make it a bit more consistent.

…ce#7282) * Move `wasmtime explore` behind a Cargo feature Enable this Cargo feature by default, but enable building the CLI without the `explore` subcommand. * Move the `wast` subcommand behind a Cargo feature * Move support for `wat` behind a CLI feature This was already conditional in crates such as `wasmtime` and this makes it an optional dependency of the CLI as well. * Move CLI cache support behind a Cargo feature Additionally refactor `wasmtime-cli-flags` to not unconditionally pull in caching support by removing its `default` feature and appropriately enabling it from the CLI. * Move `rayon` behind an optional feature * Move `http-body-util` dependency behind `serve` feature * Add a Cargo feature for compiling out log statements This sets the static features of `log` and `tracing` to statically remove all log statements from the binary to cut down on binary size. * Move logging support behind a Cargo feature Enables statically removing logging support in addition to the previous compiling out log statements themselves. * Move demangling support behind a Cargo feature * Enable building the CLI without cranelift Compile out the `compile` subcommand for example. * Gate all profiling support behind one feature flag This commit moves vtune/jitdump support behind a single `profiling` feature flag that additionally includes the guest profiler dependencies now too. * Move support for core dumps behind a feature flag * Move addr2line behind a feature * Fix rebase * Document cargo features and a minimal build * Tidy up the source a bit * Rename compile-out-logging * Document disabling logging * Note the host architecture as well * Fix tests * Fix tests * Mention debuginfo stripping * Fix CI configuration for checking features * Fix book tests * Update lock file after rebase * Enable coredump feature by default

…on aarch64 (bytecodealliance#7281) * PCC: draw the rest of the owl: fully-working PCC on hello-world Wasm on aarch64 This needed a bit more inference / magic than I was hoping for at first, specifically around constants and adds. Some instructions can now generate facts on their output registers, even if not stated. This breaks away from the "breadcrumbs" idea, but seems to be the most practical solution to a general problem that we have mini-lowering steps in various places without careful preservation of PCC facts. Two particular aspects: - Constants: amodes on aarch64 can decompose into new constant-generation instructions, and we need precise ranges on those to properly check them. To avoid making the ISLE rules nightmarish, it's best to reuse the existing semantics definitions of the Add* ALU insts, and add a few rules for MovK/MovZ/MovN. - Adds: similarly, amodes decompose into de-novo add instructions with no facts. To handle this, there's now a notion of "propagating" facts that cause an instruction with a propagating fact on the input to generate a fact on the output. Together, these heuristics mean that we'll eagerly generate a fact for `mem(mt0, 0, 0) + 8 -> mem(mt0, 8, 8)`, but we won't, for example, generate ranges on every single integer operation. With these changes and a few other misc fixes, this PR can now check a nontrivial "hello world" Wasm on aarch64 down to the machine-code level: ``` $ target/release/wasmtime compile -C enable-pcc=y ../wasm-tests/helloworld-rs.wasm ``` * Review feedback.

This commit changes the `--dir` argument on the `wasmtime` CLI to be `HOST::GUEST` rather than `GUEST::HOST`. This matches Docker for example and is a little more consistent with only `--dir path` where the first argument is always treated as a host directory. In terms of breaking-ness the movement from `--mapdir` to `--dir` hasn't been released with Wasmtime 14 yet so my hope is that this can land on both `main` and Wasmtime 14.0.0 before it's released to avoid any breakage other than existing scripts migrating from `--mapdir` to `--dir`.

* Update WASI versions to `0.2.0-rc-2023-11-05` This commit updates the version numbers on `main` to no longer clash with the 14.0.0 release after bytecodealliance#7299. The version number is chosen as the branch point for the 15.0.0 release of Wasmtime, at which point we'll update the versions again. * Update another version

Help to excise more dependencies when building the C API.

@jameysharp

…dealliance#7304) @jameysharp and I noticed a couple of refactoring opportunities while reading through the elaboration pass: * The elaboration loop doesn't need to match on the top of the stack as a reference, because each case pops it immediately. Instead we can pop and match on the popped value. * Computing the cost of a `Result` value that's not in the DFG was using the block that contains the instruction to determine the level, but since the instruction is already known to not be in the DFG, this would default to `LoopLevel::root()` unconditionally. This also meant that the `Cost::at_level` function turned into an identity function on the cost given, making it unnecessary. Co-authored-by: Jamey Sharp <jsharp@fastly.com>

Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>

This doesn't shave off much but can help tidy up some symbols from exported objects if they aren't needed.

…alliance#7312)

…#7306) This reworks the way that remat and LICM interact during aegraph elaboration. In principle, both happen during the same single-pass "code placement" algorithm: we decide where to place pure instructions (those that are eligible for movement), and remat pushes them one way while LICM pushes them the other. The interaction is a little more subtle than simple heuristic priority, though -- it's really a decision ordering issue. A remat'd value wants to sink as deep into the loop nest as it can (to the use's block), but we don't know *where* the uses go until we process them (and make LICM-related choices), and we process uses after defs during elaboration. Or more precisely, we have some work at the use before recursively processing the def, and some work after the recursion returns; and the LICM decision happens after recursion returns, because LICM wants to know where the defs are to know how high we can hoist. (The recursion is itself unrolled into a state machine on an explicit stack so that's a little hard to see but that's what is happening in principle.) The solution here is to make remat a separate just-in-time thing, once we have arg values. Just before we plug the final arg values into the elaborated instruction, we ask: is this a remat'd value, and if so, do we have a copy of the computation in this block yet. If not, we make one. This has to happen in two places (the main elab loop and the toplevel driver from the skeleton). The one downside of this solution is that it doesn't handle *recursive* rematerialization by default. This means that if we, for example, decide to remat single-constant-arg adds (as we actually do in our current rules), we won't then also recursively remat the constant arg to those adds. This can be seen in the `licm.clif` test case. This doesn't seem to be a dealbreaker to me because most such cases will be able to fold the constants anyway (they happen mostly because of pointer pre-computations: a loop over structs in Wasm computes heap_base + p + offset, and naive LICM pulls a `heap_base + offset` out of the loop for every struct field accessed in the loop, with horrible register pressure resulting; that's why we have that remat rule. Most such offsets are pretty small.). Fixes bytecodealliance#7283.

…64 (bytecodealliance#7325) This is required by the ABI and prevents a miscompilation when calling LLVM compiled functions.

* riscv64: Extend distance trampolines can jump Use a PIC-friendly set of instructions to enable destination of the trampoline to be more than 4k away from the tail call site of the trampoline itself. * Build "min" artifacts on CI This commit updates the binary artifacts produced by CI to include "min" builds where all default features are disabled. Additionally all the stops are pulled in terms of build flags, nightly versions, etc, to get a build that is as small as possible without actual source code changes. This effectively codifies the instructions in bytecodealliance#7282 into an easily downloadable artifact. No new tarballs are created for github releases but instead tarballs that previously had a `wasmtime` executable for example now have a `wasmtime-min` executable. Furthermore the C API which previously had `libwasmtime.so` for example now has `libwasmtime-min.so`. The intention is that the minimum-size artifacts are handy for determining a rough size of Wasmtime but they're not so important that it seems worthwhile to dedicate entire release entries for. CI is refactored to support these minimum builds with separate builders. This means that a single tarball produced as a final result is actually two separate tarballs merged together, one for the normal build we do today plus a new "min" tarball produced on the new "min" builders. Various scripts and CI organization has been adjusted accordingly. While here I went ahead and enabled `panic=abort` and debuginfo stripping in our current release artifacts. While this doesn't affect a whole lot it's less to upload to GitHub Actions all the time. * Fix Windows unzip

…nce#7317) While not a large amount of binary size if the purpose of the `--no-default-features` build is to showcase "minimal Wasmtime" then may as well try to make `clap` as small as possible.

This was mistakenly removed during bytecodealliance#7300 with some local testing I was doing. This will eventually be required to get a minimal build of the C API but for now I didn't intend on deleting this so I wanted to rectify my mistake.

Helps with bindings generation on the Python side of things.

In an effort to simplify the many fuel related APIs, simplify the interface here to a single counter with get and set methods. Additionally the async yield is reduced to an interval of the total fuel instead of injecting fuel, so it's easy to still reason about how much fuel is left even with yielding turned on. Internally this works by keeping two counters - one the VM uses to increment towards 0 for fuel, the other to track how much is in "reserve". Then when we're out of gas, we pull from the reserve to refuel and continue. We use the reserve in two cases: one for overflow of the fuel (which is an i64 and the API expresses fuel as u64) and the other for async yieling, which then the yield interval acts as a cap to how much we can refuel with. This also means that `get_fuel` can return the full range of `u64` before this change it could only return up to `i64::MAX`. This is important because this PR is removing the functionality to track fuel consumption, and this makes the API less error prone for embedders to track consumption themselves. Careful to note that the VM counter that is stored as `i64` can be positive if an instruction "costs" multiple units of fuel when the fuel ran out. prtest:full Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

With a custom standard library disabling all features means disabling some support for feature-detection macros of the native platform. This meant that `wasmtime compile` output locally wasn't runnable in `wasmtime-min run` because it couldn't correctly detect that features were in fact available.

* riscv64: Specify rounding modes in instructions This commit updates how floating-point instructions specify their float rounding mode (FRM). Previously instructions stored `Option<FRM>` and this would mostly be `None`. All floating-point instructions in RISC-V have a 3-bit `rm` field, and most encode the FRM into this field but some have a require encoding of this field. For example `fsgnj.s` uses the `rm` field to differentiate between `fsgnj`, `fsgnjx`, and `fsgnjn`. Instructions like `fadd` however use this field for a rounding mode. All FPU instructions now store `FRM` directly. Instruction helpers like `fadd` require this to be specified explicitly. Instructions helpers like for `fsgnj` do not take this as an argument and hardcode the field as necessary. This means that all lowerings of floating point instructions, where relevant, now specify a rounding mode. Previously the default rounding mode was to use the `fcsr` register, meaning that the rounding mode would be determined dynamically at runtime depending on the status of this register. Cranelift semantics, however, are derivative of WebAssembly semantics which specify round-to-nearest ties-to-even. This PR additionally fixes this discrepancy by using `FRM::RNE` in all existing instructions instead of `FRM::Fcsr`. * riscv64: Refactor float-to-int conversions This commit removes the `FcvtToInt` macro-instruction in the riscv64 backend in favor of decomposing it into individual operation for `fcvt_to_{s,u}int*` instructions. This additionally provides a slightly different lowering for the `*_sat` operations which doesn't use branches. The non-saturating operations continue to have a number of branches and their code has changed slightly due to how immediates are loaded. Overall everything is in ISLE now instead of split a bit. * riscv64: Clean up some dead code in the backend Don't put `#![allow(dead_code)]` at the root, instead place it on some smaller items. * Fix emission tests * Add regression tests and bless output Closes bytecodealliance#5992 Closes bytecodealliance#5993 * Enable i8/i16 saturating float-to-int in fuzzgen * Better `fcvt_*_bound` implementations * Fix typo in match orderings * Fix tests on x64 Where float-to-int isn't implemented for i8/i16

* wasmtime: Fix Bench API Build * wasmtime: Enable cranelift feature in cli-flags

)

This commit introduces a wrapper crate which is now the new "real" C API. The purpose of this change is to enable using LTO when building the C API. Currently LTO is disabled because one of the crate types of the C API is an "rlib" which means that it can't have LTO performed due to rustc limitations. The solution here is to remove the "cdylib" and "staticlib" crate types from the "wasmtime-c-api" crate, rename that crate to "wasmtime-c-api-impl", and reintroduce 'wasmtime-c-api' as a new crate which wraps the previous crate and reexports it. This way LTO can be enabled when just building the artifacts and the use case from bytecodealliance#6765 is still satisfied by having a crate that can be linked to from Rust. Locally this reduces the size of the C API artifact for me by nearly 1M.

* Add release notes for 14.0.1 Closes bytecodealliance#7342 * Clarify this was just a cranelift point release

Previously, we assumed that the Wasmtime engine would be able to allocate keys 1-15 from the OS, in that order. (Recall that Linux reserves key 0 for itself). While enabling various tests for MPK, tests in `wasmtime_fuzzing::oracles` would fail because Wasmtime could only start allocating at key 2, e.g.; it turns out that the `diff_v8::smoke` test instantiates V8 which happens to allocate a key for itself. The reason for the "allocate keys 1-15 in order" assumption was that the logic for calculating the stripe each key owned was very simple: `key - 1`. We needed some way to map each key ID to the stripe ID it is associated with. With this change, we maintain a little bit more state in order to make the mapping less brittle. `ProtectionKey` stores the "key ID to slice ID" mapping as an additional `u32` in the struct. This means that, regardless of what other code in the process allocates MPK keys, Wasmtime should be able to work fine with the remaining keys it can allocate.

* Introduce UDP streams Introduce new `inbound-datagram-stream` and `outbound-datagram-stream` types and moved `receive` and `send` methods to those respectively. These streams are returned by `bind` can be individually subscribed to. This resolves a design issue where a UDP server would end up in a spin loop because `receive` returned EWOULDBLOCK but poll_* always returned immediately because the socket was ready for sending. In this new setup, users can poll each direction separately. Fixes WebAssembly/wasi-sockets#64 Additionally: - Enable send-like behaviour by making `outbound-datagram::remote-address` optional. Fixes WebAssembly/wasi-sockets#57 - Dropped the `network` parameter from the `connect` call, because `bind` is now _required_ to perform IO. * Align names with wasi-http * Revert previous changes to `bind`. Replace `connect` with `stream` Remove the Mutex again. Instead allow `stream` to be called multiple times, but trap if the previous streams are still active. * The code block was treated as Rust code. * Align more closely to wasi-io's input&output-stream * Use `send` instead of `sendto` on connected sockets. prtest:full

…e#7350) * Expand further minimization section of documentation This commit fills out the page about producing minimal builds a bit more. The intention here is to provide examples of ideas about how to reduce size further as well as current limitations and how they can be evaluated. * Review comments

) * Describe workflow for testing preview2 components * format text

…ce#7360) This follows up bytecodealliance#6807 and removes the last remaining reference to the removed `posix-signals-on-macos` feature flag. Note that `lib.rs` now imports `mod unix` on MacOS. This change is similar to the change in `traphandlers.rs` in bytecodealliance#6807. It is needed for hosts that use signals instead of Mach ports on MacOs.

…nce#7339) When binding a specific port allow `AddressInUse` errors to be returned in case the test is being concurrently run.

This change fixes a bug with `ProtectionKey::protect`: previously it initialized each stripe with read and write permissions (i.e., `pkey_mprotect(..., PROT_READ | PROT_WRITE)` under the mistaken assumption that these permissions were MPK-specific, "what MPK permissions will we be allowed to set in the PKRU for these regions in the future?". This assumption is incorrect: the regions were immediately made accessible for reading and writing. The fix is to initially protect the regions with `PROT_NONE` and allow Wasmtime's `memory.grow` implementation to mark pages with `mprotect(..., PROT_READ | PROT_WRITE)` as usual. Whether a store can access a slice is still determined by the CPU state set in `mpk::allow`.

…lliance#7369) This is required to compile for a target which doesn't have a cranelift backend. Before this change using any of the cranelift crates that depend on cranelift-codegen would forcefully enable all default features and thus host-arch. With this change only the std and unwind features are still forcefully enabled as cranelift-codegen doesn't compile with either disabled.

* Add release notes for 14.0.2 and 13.0.1 * More notes

* mpk: add `max_memory_protection_keys` If Wasmtime is ever embedded in an application that also uses memory protection keys, it could be useful to limit how many Wasmtime allocates and uses. This came up while examining `*.wast` tests: if there was no way limiting the number of keys used, then those tests configured a pool that reserved too much memory. This change takes that further to attempt to limit the initial number of keys allocated. The unfortunate side effect of using a `OnceLock` is that the `max` setting is only applicable on the first invocation, the one that sets the `OnceLock`. * mpk: use two protection keys for WAST tests This change stems from how slicing memory slots into MPK-protected regions limits the number of memories each store can access: e.g., with fifteen keys in use, a store only has access to a fifteenth of the available slots. If we simply multiple the number of memory slots needed to run the `*.wast` spec tests by fifteen, we run out of available memory. This limits the number of protection keys used to two, which still allows us to test the functionality without reserving too much memory. * mpk: ensure `keys` only ever returns `max` items This addresses a review comment to slice the list of keys down to the `max` hint regardless of how many are allocated in the first invocation. * fix: remove warning about unused parameter

…`min_size == max_size` (bytecodealliance#7374) * cranelift-wasm: Add max size to `HeapData` * cranelift-wasm: Add a small optimization for dynamic memories with `min_size == max_size` In this case, we don't need to load the dynamic heap bound from the vmctx because it actually has a constant size. Instead, we can use the constant directly. * Review comments

…ance#7366) * Add logging and request id tracking to `wasmtime serve` * Dispatch directly to stdout/stderr in LogStream::write

* mpk: add Wasm test This adds a Wasmtime-level test checking that, when MPK is available on the system, an MPK-configured memory pool obeys the same invariants as other memory pool configurations (e.g., `guards_present` for normal allocation, `guards_present_pooling` for non-MPK pooling allocation). It also fixes a bug. A previous commit had updated the validation logic of `MemoryPool` to check that the memory plan's bound would be reached before the next slot of the same stripe. But `MemoryPool::allocate` has some "double-check" logic that was triggered by this test. It now matches the logic in `MemoryPool::validate`. * mpk: explicitly disable MPK on certain tests Some Wasmtime tests rely on specific limits to the memory pool. When MPK is enabled, these tests fail because MPK splits access to the pool slices among different stores. This change does not yet enable MPK, though locally I run with MPK enabled. With this commit and MPK enabled locally, all Wasmtime tests now pass.

* Update some crates to align on latest `rustix` This commit addresses some dependabot warnings showing up on the Wasmtime repository by updating all dependencies to using the latest `rustix` release. * Debug CI prtest:full * Downgrade rustix to 0.38.14 Works around bytecodealliance/rustix#901

Added for bytecodealliance#7378

bytecodealliance#7344) * refactor: using emit_cmp helper to refactor lowering of selection instruction on x64 * chore: remove space before paran

…tecodealliance#7351)

* Add float comparison operators to Winch * Simplify gt and gte ops and add comments

* PCC: support x86-64. This PR extends the proof-carrying-code infrastructure to support x86-64 as well as aarch64. In the process, many of the mechanisms had to be made a little more general. One important change is that the PCC leaves more "breadcrumbs" on the frontend now, avoiding the need for magic handling of facts on constant values, etc., in the backend. For the first time a lowering rule also gains the ability to add a fact to a vreg to preserve the chain as well. With these changes, we can validate compilation of SpiderMonkey.wasm with Wasm static memories on x86-64 and aarch64: ``` cfallin@fastly2:~/work/wasmtime% target/release/wasmtime compile -C pcc=yes --target x86_64 ../wasm-tests/spidermonkey.wasm cfallin@fastly2:~/work/wasmtime% target/release/wasmtime compile -C pcc=yes --target aarch64 ../wasm-tests/spidermonkey.wasm cfallin@fastly2:~/work/wasmtime% ``` * Don't run regalloc checker if not requested in addition to PCC; it's fairly expensive. * Refactor x64 PCC code to avoid deep pattern matches on Gpr/Xmm types; explicitly match every instruction kind.

* filesystem: delete lock functions from wit * delete lock methods from filesystem impl * sync wits

Completes the upgrade from bytecodealliance#7370 with bytecodealliance/rustix#901 being published to crates.io.

Change an `a - b` expression to `a.saturating_sub(b)` to handle the case when a clock is subscribed to an instance/duration in the past.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge upstream changes that have occurred since the fork #59

Merge upstream changes that have occurred since the fork #59

Commits on Sep 21, 2023

Commits on Sep 22, 2023

Commits on Sep 24, 2023

Commits on Sep 25, 2023

Commits on Sep 26, 2023

Commits on Sep 27, 2023

Commits on Sep 28, 2023

Commits on Sep 29, 2023

Commits on Oct 2, 2023

Commits on Oct 3, 2023

Commits on Oct 4, 2023

Commits on Oct 5, 2023

Commits on Oct 6, 2023

Commits on Oct 7, 2023

Commits on Oct 9, 2023

Commits on Oct 10, 2023

Commits on Oct 11, 2023

Commits on Oct 12, 2023

Commits on Oct 13, 2023

Commits on Oct 14, 2023

Commits on Oct 15, 2023

Commits on Oct 16, 2023

Commits on Oct 17, 2023

Commits on Oct 18, 2023

Commits on Oct 19, 2023

Commits on Oct 20, 2023

Commits on Oct 21, 2023

Commits on Oct 23, 2023

Commits on Oct 24, 2023

Commits on Oct 25, 2023

Commits on Oct 26, 2023

Commits on Oct 27, 2023