Merge component bindings generation into this repository #104
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* initial work * Fix formatting * Fix references to Rust library for guest code Improve documentation Other minor fixes * Fix wastime -> host_wasmtime_rust references Fixed feature flags * Fix demo deploy update readme improve help text * Update readme Co-authored-by: Kyle Brown <kbrown@singlestore.com> Co-authored-by: Kyle Brown <kyleb@liquidrocketry.com>
* Remove `async func` support. The `async` keyword has been removed from the [async proposal], in favor of using plain functions that with `future` or `stream` types. See also the [component-model PR to remove the syntax]. This removes the (now) old `async func` suppport from wit-bindgen, to make it easier to develop the current Canonical ABI proposal. [async proposal]: https://docs.google.com/presentation/d/1MNVOZ8hdofO3tI0szg_i-Yoy0N2QPU2C--LzVuoGSlE/edit#slide=id.g1270ef7d5b6_0_111 [component-model PR to remove the syntax]: WebAssembly/component-model#98 * Remove support for the `async` checkbox in wit-bindgen-demo.
* Rename `canonical_abi_realloc` to `cabi_realloc`. This follows the current Canonical ABI. This doesn't rename `canonical_abi_free`, as that's expected to be removed when post-return functions are implemented. * Rename the "expected" type to "result". This follows the current Canonical ABI. * Implement function and value type name mangling. This implements the name-mangling scheme in the current canonical ABI, with the modification proposed in WebAssembly/component-model#104, though that can be easily removed if the proposal is declined. * Use name mangling in the bindings generators. * Use the export base name rather than the mangled name for python identifiers.
Signed-off-by: GitHub <noreply@github.com> Signed-off-by: GitHub <noreply@github.com>
* WIP: Update for component model multireturn and removal of unit This PR updates wit-bindgen for the latest changes to the component model: - the return of multireturn - removal of the `unit` type - a new syntax for optional types in results, streams, futures and variants all of which go hand in hand. This also pulls in the latest versions of wasm-encoder, wasmprinter, and wasmparser crates to get their updates for these component model changes. * Get all wit-bindgen tests working * Implement tests for multi-return Exercise this throughout the runtime tests and additionally add a few variants in codegen tests to ensure that this is exercised. * Update markdown generator Co-authored-by: George Kulakowski <gkulakowski@fastly.com>
This adds a few new pseudo-instructions plus a new method on `Interface`
to generate a `post-return` function. For now these are simply named
`{name}_post_return` and the integration point there will probably
change as the component model shapes up.
The integration here is intended to still be relatively primitive in
that the actual component model integration will likely look different
in the future. Given how `wit-bindgen` works today, though, this should
at least get things part of the way there.
* remove Canonical ABI name mangling this reverts a big chunk of the functionality introduced in bytecodealliance/wit-bindgen#309 * multi-return.wit: namespace these export functions the tests already export functions named a,b,c in strings.wit. We can't define more functions with that same name. * remove mangled names from tests/runtime/invalid/wasm.rs
Try to reduce the number of places versions are mentioned and encourage shared dependencies between crates.
This migrates everything from structopt and clap 3.* to 4.0.9, while preserving the (sub)command syntax of all the binaries in all the crates in this repo.
* Remove support for handles and resources This commit removes all support for the `resource` and `Handle` types from the AST of `wit-parser` and all related support in all code generators. The motivation for this commit is that `wit-bindgen` is on the cusp of actually being able to work with components: producing a component from guest output and consuming components in host generators. More detail about this is in #314. With components as an intermediate format, however, there is no way to encode resources since they are not part of the component model proposal yet. All is not lost for handles and resources, though. The official design for handles and resources is being worked on upstream in the component model repository itself at this time and once added all of this support will be re-added to `wit-bindgen`. In the meantime though I personally think that the best way forward is to remove the interim support for a few reasons: * Primarily it unblocks progress at this time towards fully integrating components and the `wit-bindgen` generators. The requirement to run existing tests that use handles would mean that no host generator could actually switch to components and/or modes for today's core-wasm-lookalike would need to be preserved. * Otherwise though the semantics of the current handles are basically invented out of thin air by myself and were never really formally specified, debated, or designed deliberately. I grafted `witx`-style handles into `wit-component` and added features as necessary over time, but it seems highly unlikely that the handles designed as part of the component model will be the ones that `wit-bindgen` currently supports. This inevitably means that a new system would need new code anyway and would likely result in removal regardless. As usual git always has the history of handles and this all may come back in one shape or another if only slightly tweaked. I'm confident in our history spelunking abilities, though, so I don't feel that keeping support in the repository is necessary for this purpose. * Remove resources from the demo * Fix rebase conflict
async-functions.wit, host.wit, wasi-next.wit, and resource.wit are no longer in the tree, so we no longer need to explicitly disable testing them. And add the comment about disabling tests to all the codegen tests, so that it's easy to find by people debugging them.
* Update all crates to heck 0.4 resolves #342 * rustfmt
This commit is a large refactor of how testing works in this repository for wit-bindgen and hosts/guests to simplify the flow of tests and make it easier to modify and less monolithic. Previously tons of logic was located in a `test_helpers` macro crate but now that large crate has been split up to delegate more work to each individual test. Some common functionality is still present in a `test_helpers` crate but it's more clear that the `test_helpers::codegen_test` macro is purely responsible for simply expanding a glob at compile time and delegating to a macro-per-test. The tests of Rust & other languages are now also more uniform where everything "just invokes a macro" where Rust's macros further invoke proc-macros for the respective generators and other languages delegate to shared functionality for "generate code" then "run verification program". Overall I'm hoping this makes the testing easier to follow and understand, and in upcoming changes for components it will make it easier to change each generator, individually, over to components ideally.
* Reimplement the host Python generator with worlds
This commit is the equivalent of #381 but for Wasmtime Python host
bindings. This is a large-scale refactor of the internals of the Python
host generator in which I took quite a few liberties in internal
restructuring as well as output restructuring. The tests are probably
the best to review and better reflect what changed, but there are some
aspects of worlds that the tests are not currently exercising which
we'll want to add with the introduction of worlds in the future.
This means that all host generators are now working with worlds as input
rather than individual `*.wit` files, and the only two remaining
generators are the C and Java generators for guests (which I hope are
easier).
The high level summary of the new output is:
out_dir/
__init__.py # top-level component exports/definitions
types.py # shared type information, currently just `Result`
imports/
__init__.py # only here if something is imported
foo.py # type and protocol definition per interface
exports/
__init__.py # only here if an instance is exported
bar.py # one per exported instance
"Default exports" will show up on the generated structure in
`out_dir/__init__.py` so all runtime tests, for example, do not generate
`exports` at this time. Lots of fiddly stuff went into structuring this
all right to get past `mypy` and additionally try to avoid name
conflicts. It's still somewhat easy to have name conflicts but ideally
they're of the more esoteric category rather than "really easy to run
into".
* Try to fix windows
* Update all codegen tests to world syntax This enables simplifying codegen tests as well by having each test exercise imports/exports instead of having all code generators have one case for imports and one for exports. * Update codegen tests for the C guest generator * Update teavm codegen tests for worlds * Remove the `--name` argument from the CLI No longer needed as the name is inferred from the `*.wit` world. * Get all js host generator tests working Update all `runtime/*` tests with new naming conventions, world files, etc. Minor updates were made to names as I ended up using different conventions for the `*.wit` world files than were previously exercised. This also fixes a few minor issues with the output for the JS generator in the default ESM mode. * Get all host-wasmtime tests working with worlds Various small updates here and there to namings and such. * Get all Python tests working with worlds Fix support for exported instances by using correct import paths for various types/intrinsics/etc. Additionally update the codegen tests to specify a mypy cache dir to avoid racing between tests. * Update demo build for worlds Also fix an issue with the Rust host generator where it used an interface's name instead of the name of the import for import module names. * Update demo with world files
This will be powered by `wit-bindgen`-stuff itself compiled to wasm
running within Wasmtime. The rough intention at this time is to enable
developers to execute:
python3 -m wasmtime.bindgen component.wasm --out-dir ./my-out-dir
and that's how Python bindings will get generated for the specified
component.
For now settle on the scheme of a `tests/codegen/test_*` suite of files which, when imported, will generated bindings on-the-fly which means that mypy type-checking, testing, etc, can all get folded into a single `pytest` with good error reporting. All that's left now is to fill out all the tests for all the various features.
|
I'm opening this as a draft as mostly a declaration of "this is happening". Otherwise though remaining items for me are:
The main plumbing of everything should be there and is almost ready and good to go. |
This'll help remove `wat` from the wasm blob eventually as it's not actually needed there.
Should help shrink the final wheel size a bit.
This was referenced Nov 18, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a very large PR which is tasked with, at a high level, merging the
wit-bindgen-gen-host-wasmtime-pygenerator from thewit-bindgenproject into this repository. My purpose for implementing this is to chart a path for moving the Wasmtime and JS host generators out ofwit-bindgenas well, so while this is Python-specific I'm hoping that all the lessons learned here can be used as inspiration for those migrations as well.This is, however, not "just a simple merge" of moving code around. The
wit-bindgenproject has its own way of integrating into a CLI and additionally has its own setup for tests. Neither of those fit very well into this repository so I opted to take a different path. What I've ended up settling on feels, I think at least, more natural for Python and more easily integrates with this repository. At a high-level the major points of this movement are:wit-bindgen-gen-host-wasmtime-pycrate is moved to this repository under arustdirectory. This is then exported as awasmtime.bindgenmodule from thewasmtimepackage this repository represents. This is internally implemented by compiling the Rust code itself to WebAssembly and then using thewasmtimepackage itself to run the wasm to power the Python APIs. More-or-less this is a prototype of "what if wasm modules were used as Python extensions?". I've opted to do this since I would like to keep the generation of Python in Rust as all*.wittooling is in Rust. Additionally I don't want to go through dllctypesto integrate everything into Python in the same manner the Wasmtime C API is integrated. Thus, this demos a self-hosted Python bindings generator.cisuch asci/download-wasmtime.pyandci/cbindgen.py(renamed fromci/bindgen.py)ci/build-rust.pywill build the Rust code into a component and generate thewasmtime.bindgenbindings to run this component.tests/codegen/test_*which will, for specific components, generate bindings and then run them locally.python -m wasmtime.bindgen the-component.wasm --out-dir foo. I intentionally haven't wrapped this up in a nice script since I don't have a good name for it. This, however, is intended to be the entry point for documenting how end users can actually use this. As a bonus it doesn't require building a bunch of Rust code as a Python developer.