[RFC] Way to ease publishing of rust-based plugins

[kdy1]

While trying to find an easy for publishing a rust-based plugin, I found that it's almost impossible and I finally found that the only way to make it comfortable is building a service.

This image illustrates what I wanted to avoid if possible.

So, I'm considering a service which does all required tasks. It will be free for open-source projects, and it will not be free for if it's not FOSS. The service will make your swc plugin source code into binaries for each platform. If companies decide to use their own CI, there's no restriction.

How do you think about this? I want to hear many opinions from various people.

If you have any other way to ease publishing, please tell me.

[vjpr]

I'm not sure too many companies are using private Babel plugins, and I can't think of any of mine that I would want private. I have some that are inside a monorepo, but if I needed them compiled I would just publish them.

If you broadened the scope to Rust N-API plugins, then I can see value here because the 'napi-rs' GitHub actions/workflow seems quite complex. If companies want to improve performance of their services, writing Rust napi plugins is a great way to do so.

Things I would like:

1. Monorepo support.

Searches codebase for plugins and adds builds for each. Vercel charges for this.

2. Ability to run full workflow locally.

The biggest pain point with cloud CIs is having to commit+push+wait for tweaks until things work. So I'd like a cli that runs everything locally.

3. Caching

Rust builds are super slow. Building swc is very slow. I think incremental Rust compilation is not available yet, but maybe making linking against precompiled 'rlibs' easy - if I am understanding how this works correctly.

I really like 'napi-rs' approach of publishing binaries to npm as optional deps.

[jantimon]

I really like the idea of an cli...
The challenge for both ways (cloud service with or without cli) is the cross platform compilation

As an example the npm sharp package supports a wide variety of plattforms - and still a lot of their issues are installation / compatibility related.

[kdy1]

I will create cli regardless of this service.

But it will not support cross-compiling. It's simply not possible.
Cloud service will use the cli I mentioned but will invoke in various operating systems and architectures.

[vjpr]

This is simply impossible. You can build a binary for your platform, but not others.

I was thinking via VMs, and networked physical machines (e.g. From x64 to M1, using a networked Mac Mini M1) to achieve non-native platforms. Cross-compilation is also possible (but can be painful) in some cases (win -> linux, mac -> linux, mac m1 -> mac x64). So if a dev had access over network to an M1 machine and Windows machine, they could just enter the IP addresses, and the CLI would build on them.

This would be necessary if specific platform apis were being used (mac v win v linux), because things tested on your local machine wouldn't be guaranteed to work on other platforms.

However, if the scope is kept to only swc plugins (as opposed to n-api rust plugins as I mentioned), then there will probably not be any specific platform apis being used (unless a plugin wanted to do some kind of file-watching for example). So being able to test/debug each platform locally wouldn't matter as much.

[kdy1]

It would be an enormous amount of work. Isn't it?
What you mentioned seems similar to https://www.incredibuild.com/

[vjpr]

For local dev purposes, it's just like: rsync code to host, and then trigger test/build scripts remotely on host. Maybe add a daemon that you connect to. It's a bit of work. I like to be able to test things locally before pushing, and if everything passes locally, then it passes in CI. Waiting 5 mins for a CI to break on on of several targets is a pain, and then you need to setup a dev env for that target locally anyway. Personally, I run a separate machine on my local network with Windows to test/debug cross-platform packages. I will prob add an M1 Mini eventually for ARM.

But as I said, for swc plugins, there are not platform-specific changes and a traditional cloud system will be fine.

[jantimon]

Thank you @kdy1 for starting this discussion

Here are some questions that came to my mind:

• Have you already tried to react out to https://github.com/rust-lang/rust?
• Could NPM (Microsoft) be of help?
• In babel every plugin user can start debugging by placing a console.log statement in the node_module folder - how would this look like for this approach?
• How much speed impact do you expect between the current JS api and the up coming RUST api?
• Do you think a https://astexplorer.net/ integration would be possible for SWC?

[kdy1]

Have you already tried to react out to https://github.com/rust-lang/rust?

I didn't get it, but if you mean if I asked about cross-compilation to the rust core team, no.

Could NPM (Microsoft) be of help?

Yes, the users of plugin will simply use npm and there will not be a build script. It will use similar technique like @swc/core.
The service is only about cross-compiling, and built binaries will be deployed to npm with some security config user specified.

In babel every plugin user can start debugging by placing a console.log statement in the node_module folder - how would this look like for this approach?

Users can use rust-based testing system to do similar thing, and the cli will support building dynamic library for current platform.

How much speed impact do you expect between the current JS api and the up coming RUST api?

It depends. I expect about (10 ~ 15) * (cpu core).

Do you think a https://astexplorer.net/ integration would be possible for SWC?

Yes.

[kdy1]

Btw, thank you for asking questions and giving some thoughts.
I really need lots of opinions from various people.

[stefee]

Could x-compilation be achieved using a open-source GitHub Action or CircleCI Orb?

[stefee]

I’m not very experience with GitHub Actions, sorry - I think what I mean to ask about is a composite action https://docs.github.com/en/actions/creating-actions/creating-a-composite-action

I am more experienced with CircleCI actually. If it has good Rust support (or you could use Docker executor) then that might be an option to consider too for your use case.

[kdy1]

I think it's possible, although it would be quite complex.

[stefee]

I think it is necessarily complex internally, like how it is already in the CI workflow you linked, but it can be made quite simple for a plugin author by moving into shared CI module.

[stefee]

In your original post

This image illustrates what I wanted to avoid if possible.

I would appreciate if you can elaborate more on the problem you are highlighting here.

[kdy1]

I meant complex ci config file, but I decided to go with a github action.

[ddprrt]

@jantimon pointed me to this issue quite a while ago, and I haven't found time to reply until now. So sorry if this is already outdated and of no interest anymore 🙈

Have you considered wasm32-wasi as a target for Rust-based plugins that load it in a WASM runtime? It would allow for cross-platform distribution and you'd get the extra benefit of sandboxing and proper isolation. This is a simple "Hello World" using the wasmtime crate

use wasmtime::{Engine, Linker, Module, Store};
use wasmtime_wasi::WasiCtxBuilder;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // This is the WASM equivalent of booting up a new V8 platform. Usually one engine is enough
    let engine = Engine::default(); 

    let mut linker = Linker::new(&engine);
    wasmtime_wasi::add_to_linker(&mut linker, |s| s)?;

    let wasi = WasiCtxBuilder::new()
        .inherit_stdio()
        .inherit_args()?
        .build();
    // The "Store" defines the interfaes to the outside. E.g. access to file system, stdio, if necessary. In this case it's WASI
    let mut store = Store::new(&engine, wasi); 

    // Creating a new instance or "isolate", linked with the imports to WASI
    // The &str "main" notes the entry point we want to call
    let module = Module::from_file(&engine, "hello-wasm.wasm")?; 
    linker.module(&mut store, "main", &module)?;

    linker
        .get_default(&mut store, "main")?
        .typed::<(), (), _>(&store)?
        .call(&mut store, ())?;

    Ok(())
}

There are simpler ways if no WASI context is necessary. I collected some of them in a presentation I did a while ago. Theoretically, this would also be a way to create plugins in other languages that compile to wasm32-wasi. And Node-based tools could also be using Rust-based plugins then.

I can't say if introducing a WASM runtime will meet your performance goals, though. I've been using it in a project where we need to modify data on request/response, but I have no details or numbers. It's definitely better than "good enough performance" 😅

[kdy1]

How can I pass data to it and get back the result?

I tried using wasmer and it required configuring compiler. There were several compilers and the llvm-compiler requires system-wide installation of llvm.

Isn't this the case for wasmtime?

If it's simple enough, I think wasm-based plugin system would be good enough.

[ddprrt]

You'd need a runtime that supports WebAssembly Interface Types, like wasmtime. This allows you to map WASM type to Rust/JavaScript/Python/.. types.

This is an example the WASM folks use to illustrate this.

Let's have e.g a Markdown conversion plugin written in Rust, compiled to WASM

use pulldown_cmark::{html, Parser};
use wasm_bindgen::prelude::*;

// This creates the interface for the outside
#[wasm_bindgen]
pub fn render(input: &str) -> String {
    let parser = Parser::new(input);
    let mut html_output = String::new();
    html::push_html(&mut html_output, parser);
    return html_output;
}

And you want to use it in your Rust app:

// this macro defines the interface type (same as render in our WASM module)
// it also creates a struct that boots up an engine, store, linker...
#[wasmtime_rust::wasmtime]
trait WasmMarkdown {
    fn render(&mut self, input: &str) -> String;
}

fn main() -> Result<(), anyhow::Error> {
    // load it like any module
    let mut markdown = WasmMarkdown::load_file("markdown.wasm")?;
    // IO between the WASM Module and Rust works. Both know about the interfaces
    println!("{}", markdown.render("# Hello, Rust!"));
    Ok(())
}

The example above is very simplified, but I think you get the idea :-) There's also been some movement in the Interface types proposal so it might be outdated. If you give me some time I can come up with a demo repo that features some up to date examples.

[kdy1]

Thank you so much for the detailed explanation!
I can't be sure at the moment, but wasm is something I always wanted to use.
Requesting plugin authors to cross-compile their plugin is not what I want to do.

[kdy1]

I profiled it.

Due to abi issue, my small plugin system uses serde_json to pass data.
It uses almost all times, and this ruins profiling result, but I got some notable points anyway.

---

I invoked each plugin 100 times.

To make load time ignorable, I invoked each plugin 10000 times.

---

Points

• Performance of serde_json compiled to wasm is much better than I expected.

  • It was similar to the performance of native.

• Loading of a wasm is quite slow.

  • It can be precompiled and/or cached.
  • As it can be parallelized, the loading time is not linear to the number of wasm plugins.

Although I think I'll add support for wasm-based plugins in the future, the first load time issue is too big IMO.

[kwonoj]

I believe this is superseded by #3540 now we waonly requires single wasm binary for the plugin instead of per-platform compliation.