PoC for inline target #4065

torfmaster · 2024-08-12T14:31:13Z

This PR adds an "isomorphic" or "inline" target next to the nodejs target as a proof of concept. It targets simple use cases as using small snippets of Rust Code in larger typescript codebases. In detail it allows (with a proposal that I will create for wasm-pack) creating npm packages that ship wasm modules inline.

Background

Using Rust code via wasm in existing nodejs typescript applications is challenging. It usually involves changing the bundler configuration and currently involves handling wasm artifacts.
The motivating example for a typescript lambda built via nx and bundled via esbuild (which is the de-facto standard for AWS Lambda in CDK). After a lot of experimentation (involving using different bundlers and writing plugins) I found no obvious way for my rather easy use case (a few lines of Rust Code embedded via wasm). Hence I

Relation to other issues

This touches at least #2265, rustwasm/wasm-pack#1334, rustwasm/wasm-pack#831.

Testing

I currently only did some rough manual testing. Once I know that this PR is somehow useful I can invest more.

Systemcluster

See my PR #4027 for the Node target and #2176/#3990 for the Deno target in regards to tests and other target-addition changes.

I was going to refactor the target generation a bit, but this change is small enough where it doesn't matter whether this PR or mine gets merged first. I'll ping you once I open it in either case.

Systemcluster · 2024-08-12T14:53:10Z

crates/cli/src/bin/wasm-bindgen.rs

@@ -110,6 +110,7 @@ fn rmain(args: &Args) -> Result<(), Error> {
            "nodejs" => b.nodejs(true)?,
            "deno" => b.deno(true)?,
            "experimental-nodejs-module" => b.nodejs_module(true)?,
+            "isomorphic" => b.isomorphic(true)?,


Since this is a proof-of-concept I think experimental-isomorphic for the name makes more sense.

change cli option

Systemcluster · 2024-08-12T14:59:21Z

crates/cli-support/src/js/mod.rs

+
+                for (id, js) in crate::sorted_iter(&self.wasm_import_definitions) {
+                    let import = self.module.imports.get_mut(*id);
+                    footer.push_str("\nmodule.exports.");


The use-cases all expect CommonJS?

Good point. Technically it also makes sense for ecmascript modules but I can't think of a use case.

check whether and how to support ecmascript modules

I decided not to use ecmascript modules for now as I don't see a use case for this. The right way of mixing ecmascript modules with wasm should be importing wasm as esm.

Systemcluster · 2024-08-12T14:59:58Z

crates/cli-support/src/js/mod.rs

@@ -142,7 +142,7 @@ impl<'a> Context<'a> {
            self.globals.push_str(c);
        }
        let global = match self.config.mode {
-            OutputMode::Node { module: false } => {
+            OutputMode::Isomorphic | OutputMode::Node { module: false } => {


Why the name Isomorphic?

It was introduced in rustwasm/wasm-pack#1334 and I didn't question it, it was just a catchy name. The idea seems to be that the code should run unchanged (without polyfiils or other modification) in any context (node, browser, ...). But this goal is beyond what I am doing here. Here I just inline the assets that need (and effectively cannot atm) to be processed by a bundler.
I would rather call it node-universal. Basically because that's basically what it should do: You can build your Rust code with wasm-pack and include it in your typescript application (e.g. react, aws lambda, node express) without worrying about the bundling.
What do you think?

Sorry for disturbing you here.

I previously facilitated the sharing the same WebAssembly file across of cjs/esm/bundler through manual adjustments in this JohnnyMorganz/StyLua#848 .
I also shared my concept in #3790, which supports various environments, including browsers, Node, Deno, Bun, and bundlers.

I'm not certain about your reference to include it in your typescript application (e.g. react, but include CJS into the web project is challenging. We faced this issue in swc-project/swc-playground#42, and finallay, we had to create a separate esm to fulfill the requirements.

Inline WASM is viable, but I don't believe it can serve as a direct alternative to the current web/node/deno/bundler option. Instead, this option should complement them, being usable independently and applicable across all environments.

Sorry for disturbing you here.
Thank you for your valuable insights!

I previously facilitated the sharing the same WebAssembly file across of cjs/esm/bundler through manual adjustments in this JohnnyMorganz/StyLua#848 . I also shared my concept in #3790, which supports various environments, including browsers, Node, Deno, Bun, and bundlers.

I'm not certain about your reference to include it in your typescript application (e.g. react, but include CJS into the web project is challenging. We faced this issue in swc-project/swc-playground#42, and finallay, we had to create a separate esm to fulfill the requirements.

Maybe I am missing something here (I am actually using wasm only in AWS Lambdas for now). Is there an issue with using the npm package generated by the Isomorphic target in an arbitrary react app?

Inline WASM is viable, but I don't believe it can serve as a direct alternative to the current web/node/deno/bundler option. Instead, this option should complement them, being usable independently and applicable across all environments.

I totally agree. My focus is also on the inline part not on the "usability across targets". I have been watching the asset handling issue in esbuild, webpack, rollup, wasm-pack, wasm-bindgen and never came closer to the solution than writing custom plugins plus a bunch of scripts. So as an intermediate solution I would propose inline wasm until every target just speaks wasm as esm module either directly or with any bundler of my choice without even having to change configuration.

Summing this up, I think I should just focus on an "inline node" target in this PR, moving the portability issues to the future when someone actually needs this.

lgarron · 2024-08-30T22:27:00Z

I'm sorry this isn't a very substantial contribution, but the name "isomorphic" is rather confusing to me. If I understand, the idea is to make the code identical for browsers and other JS engines (which I'd really love to see), whereas "isomorphic" refers in general to things that are equivalent but not necessarily identical? Would it make more sense to call the code something more like "cross-compatible"?

cryptoquick · 2024-08-30T23:08:45Z

"cross-compatible" is not very specific. I would prefer not to invent a new industry term. This behavior is generally known within the developer community as isomorphic builds. Consider it a term of art.

lgarron · 2024-08-31T06:20:24Z

Hmm, it seems you're right: https://en.wikipedia.org/wiki/Isomorphic_JavaScript

That also states:

The naming of the term 'Isomorphic JavaScript' has been a matter of controversy.[1] The term 'isomorphic' was first coined by Charlie Robbins from Nodejitsu, in one of the company's blog posts.[2] Spike Brehm, a software engineer from Airbnb, wrote another blog post using the same term.[3] However, others have proposed to use the term Universal JavaScript instead.[1][4][5]

I certainly think "universal" is much less of a misnomer, but thanks for the context!

torfmaster · 2024-08-31T19:55:08Z

I changed the target name to "inline"

because it is less controversial
it is more precise. I don't claim this code will work with web browsers at all

It would be great to have an "universal" target (or if you want to call it like this: isomorphic). But the required changes would be much bigger than this tiny pull request.

torfmaster · 2024-09-04T20:07:28Z

Out of curiosity I also checked browser compatibility: after a tiny fix basic functionality (meaning you can go the happy path and npm i a wasm project in your react app) seems to be there in browsers as well. A PoC is here: https://github.com/torfmaster/inline-wasm-react.

torfmaster marked this pull request as draft August 12, 2024 14:31

This was referenced Aug 12, 2024

wasm-pack should generate isomorphic code rustwasm/wasm-pack#1334

Open

One generated wasm to rule them all #2265

Open

Systemcluster reviewed Aug 12, 2024

View reviewed changes

torfmaster force-pushed the feature/isomorphic branch 3 times, most recently from 216acf3 to f9f1eff Compare August 20, 2024 19:18

torfmaster force-pushed the feature/isomorphic branch from f9f1eff to c57b5cf Compare August 31, 2024 19:50

torfmaster changed the title ~~PoC for isomorphic target~~ PoC for inline target Aug 31, 2024

torfmaster marked this pull request as ready for review August 31, 2024 19:59

torfmaster force-pushed the feature/isomorphic branch from c57b5cf to ed3294f Compare September 1, 2024 20:29

torfmaster added 2 commits September 1, 2024 23:05

feat: add experimental inline node target

8e2a73c

fix: do not write wasm for inline mode

cb8a1f1

torfmaster force-pushed the feature/isomorphic branch from ed3294f to cb8a1f1 Compare September 1, 2024 21:05

feat: use uint8array to facilitate browser use

323f2fa

daxpedda added the needs review Needs a review by a maintainer. label Sep 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PoC for inline target #4065

PoC for inline target #4065

torfmaster commented Aug 12, 2024 •

edited

Loading

Systemcluster left a comment •

edited

Loading

Systemcluster Aug 12, 2024

torfmaster Aug 13, 2024 •

edited

Loading

Systemcluster Aug 12, 2024

torfmaster Aug 13, 2024 •

edited

Loading

torfmaster Aug 31, 2024

Systemcluster Aug 12, 2024

torfmaster Aug 13, 2024

magic-akari Aug 14, 2024 •

edited

Loading

torfmaster Aug 14, 2024 •

edited

Loading

lgarron commented Aug 30, 2024 •

edited

Loading

cryptoquick commented Aug 30, 2024

lgarron commented Aug 31, 2024

torfmaster commented Aug 31, 2024

torfmaster commented Sep 4, 2024

PoC for inline target #4065

Are you sure you want to change the base?

PoC for inline target #4065

Conversation

torfmaster commented Aug 12, 2024 • edited Loading

Background

Relation to other issues

Testing

Systemcluster left a comment • edited Loading

Choose a reason for hiding this comment

Systemcluster Aug 12, 2024

Choose a reason for hiding this comment

torfmaster Aug 13, 2024 • edited Loading

Choose a reason for hiding this comment

Systemcluster Aug 12, 2024

Choose a reason for hiding this comment

torfmaster Aug 13, 2024 • edited Loading

Choose a reason for hiding this comment

torfmaster Aug 31, 2024

Choose a reason for hiding this comment

Systemcluster Aug 12, 2024

Choose a reason for hiding this comment

torfmaster Aug 13, 2024

Choose a reason for hiding this comment

magic-akari Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

torfmaster Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

lgarron commented Aug 30, 2024 • edited Loading

cryptoquick commented Aug 30, 2024

lgarron commented Aug 31, 2024

torfmaster commented Aug 31, 2024

torfmaster commented Sep 4, 2024

torfmaster commented Aug 12, 2024 •

edited

Loading

Systemcluster left a comment •

edited

Loading

torfmaster Aug 13, 2024 •

edited

Loading

torfmaster Aug 13, 2024 •

edited

Loading

magic-akari Aug 14, 2024 •

edited

Loading

torfmaster Aug 14, 2024 •

edited

Loading

lgarron commented Aug 30, 2024 •

edited

Loading