Add support for `wasi:config/runtime` virtualization #82

scothis · 2024-08-09T19:54:41Z

The wasi:config/runtime interface is a way to pass configuration to other modules with a standard interface. Similar to wasi:cli/environment the core data model is a set of key-value pairs.

The proposed API and CLI support in WASI-Virt mirrors the support for environment variables that currently exists.

The config runtime proposal is currently stage 2 and does not a have a release. Because of that, I understand it may not be desirable to land support inside wasi-virt at this time. However, if there is a willingness to move forward, I can refine this PR to DRY-up the similarities between env and config, and flesh out docs, tests, etc.

guybedford · 2024-08-12T16:14:30Z

Thanks so much for looking into this. Nice to see it implemented as a separate adapter component, to verify - if it's not used, are we able to DCE the adapter similarly to the environment subsystem?

I will aim to review in the next few days.

scothis · 2024-08-15T00:28:32Z

to verify - if it's not used, are we able to DCE the adapter similarly to the environment subsystem?

Currently following the same pattern used for wasi:cli/environment.

The config adapter is only injected into the module when the config option is enabled.

https://github.com/bytecodealliance/WASI-Virt/pull/82/files#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R156-R158

...and stripped out when not active

https://github.com/bytecodealliance/WASI-Virt/pull/82/files#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R229-R233

Something that needs to be handled better is --allow-all, which adds the wasi:config/runtime import to the resulting module even if the wrapped module doesn't import that interface. Hosts that don't provide the interface (like wasi:http/proxy and wasi:cli/command) will no longer be able to satisfy the module imports.

Have you thought about limiting the imported interfaces on the module to not exceed the imports on the wrapped module? I can't think of a reason to expose an import that will never be consumed.

guybedford · 2024-08-16T01:09:50Z

Have you thought about limiting the imported interfaces on the module to not exceed the imports on the wrapped module? I can't think of a reason to expose an import that will never be consumed.

This makes a lot of sense - a DCE of the virtual module to the known module it is adapting, when the virtual module is not being generated on its own but in this composition.

Would you be interested in doing that here or are you thinking of a follow-up perhaps?

scothis · 2024-08-16T02:50:40Z

A separate PR that lands first makes sense. I'll be offline for a few days, but can take a look next week.

guybedford

Something that needs to be handled better is --allow-all, which adds the wasi:config/runtime import to the resulting module even if the wrapped module doesn't import that interface. Hosts that don't provide the interface (like wasi:http/proxy and wasi:cli/command) will no longer be able to satisfy the module imports.

I wonder if we should change this to support "WASI sets", something like --allow-all --allow-core? Where this would be in the all set but not the core set?

This PR looks great to me though to land, we just need to ensure we have a test case that covers at least standard configuration use cases. Let me know if I can assist further at all.

The `wasi:config/runtime` interface is a way to pass configuration to other components with a standard interface. Similar to `wasi:cli/environment` the core data model is a set of key-value pairs. The proposed API and CLI support in WASI-Virt mirrors the support for environment variables that currently exists. The config runtime proposal is currently stage 2 and does not a have a release. Signed-off-by: Scott Andrews <scott@andrews.me>

scothis · 2024-08-28T19:14:46Z

tests/virt.rs

+        let result = Ok(self.props.clone());
+        let result = std::future::ready(result);
+        let result = Box::new(result);
+        let result = Pin::new(result);


there's probably a more idiomatic way to construct the result. Async rust is a bit beyond me at the moment.

scothis · 2024-08-28T19:15:29Z

Rebased on main and added tests/doc.

guybedford

This seems to be the right approach to me. If you can confirm it's definitely not being pulled in for other subsystems I think we're good to land.

src/bin/wasi-virt.rs

README.md

scothis · 2024-08-28T21:12:26Z

confirm it's definitely not being pulled in for other subsystems

I'm exactly sure what you're looking for, so here's a few examples:

no host access

wasi-virt -o virt.wasm

virt.wasm:

package root:component;

world root {
  export wasi:cli/environment@0.2.0;
  export wasi:config/runtime@0.2.0-draft;
  export wasi:cli/exit@0.2.0;
  export wasi:random/insecure-seed@0.2.0;
  export wasi:random/insecure@0.2.0;
  export wasi:random/random@0.2.0;
  export wasi:io/error@0.2.0;
  export wasi:io/poll@0.2.0;
  export wasi:io/streams@0.2.0;
  export wasi:clocks/monotonic-clock@0.2.0;
  export wasi:clocks/wall-clock@0.2.0;
  export wasi:sockets/network@0.2.0;
  export wasi:sockets/ip-name-lookup@0.2.0;
  export wasi:sockets/tcp@0.2.0;
  export wasi:sockets/udp@0.2.0;
  export wasi:sockets/instance-network@0.2.0;
  export wasi:sockets/tcp-create-socket@0.2.0;
  export wasi:sockets/udp-create-socket@0.2.0;
  export wasi:http/types@0.2.0;
  export wasi:http/incoming-handler@0.2.0;
  export wasi:http/outgoing-handler@0.2.0;
  export wasi:cli/stdin@0.2.0;
  export wasi:cli/stdout@0.2.0;
  export wasi:cli/stderr@0.2.0;
  export wasi:cli/terminal-input@0.2.0;
  export wasi:cli/terminal-output@0.2.0;
  export wasi:cli/terminal-stdin@0.2.0;
  export wasi:cli/terminal-stdout@0.2.0;
  export wasi:cli/terminal-stderr@0.2.0;
  export wasi:filesystem/types@0.2.0;
  export wasi:filesystem/preopens@0.2.0;
}

stdio only

wasi-virt --allow-stdio -o virt.wasm

virt.wasm:

package root:component;

world root {
  import wasi:io/error@0.2.0;
  import wasi:io/poll@0.2.0;
  import wasi:io/streams@0.2.0;
  import wasi:cli/stdin@0.2.0;
  import wasi:cli/stdout@0.2.0;
  import wasi:cli/stderr@0.2.0;
  import wasi:cli/terminal-input@0.2.0;
  import wasi:cli/terminal-output@0.2.0;
  import wasi:cli/terminal-stdin@0.2.0;
  import wasi:cli/terminal-stdout@0.2.0;
  import wasi:cli/terminal-stderr@0.2.0;

  export wasi:cli/environment@0.2.0;
  export wasi:config/runtime@0.2.0-draft;
  export wasi:cli/exit@0.2.0;
  export wasi:random/insecure-seed@0.2.0;
  export wasi:random/insecure@0.2.0;
  export wasi:random/random@0.2.0;
  export wasi:io/error@0.2.0;
  export wasi:io/poll@0.2.0;
  export wasi:io/streams@0.2.0;
  export wasi:clocks/monotonic-clock@0.2.0;
  export wasi:clocks/wall-clock@0.2.0;
  export wasi:sockets/network@0.2.0;
  export wasi:sockets/ip-name-lookup@0.2.0;
  export wasi:sockets/tcp@0.2.0;
  export wasi:sockets/udp@0.2.0;
  export wasi:sockets/instance-network@0.2.0;
  export wasi:sockets/tcp-create-socket@0.2.0;
  export wasi:sockets/udp-create-socket@0.2.0;
  export wasi:http/types@0.2.0;
  export wasi:http/incoming-handler@0.2.0;
  export wasi:http/outgoing-handler@0.2.0;
  export wasi:cli/stdin@0.2.0;
  export wasi:cli/stdout@0.2.0;
  export wasi:cli/stderr@0.2.0;
  export wasi:cli/terminal-input@0.2.0;
  export wasi:cli/terminal-output@0.2.0;
  export wasi:cli/terminal-stdin@0.2.0;
  export wasi:cli/terminal-stdout@0.2.0;
  export wasi:cli/terminal-stderr@0.2.0;
  export wasi:filesystem/types@0.2.0;
  export wasi:filesystem/preopens@0.2.0;
}

runtime config only

wasi-virt --allow-config -o virt.wasm

virt.wasm:

package root:component;

world root {
  import wasi:config/runtime@0.2.0-draft;

  export wasi:cli/environment@0.2.0;
  export wasi:config/runtime@0.2.0-draft;
  export wasi:cli/exit@0.2.0;
  export wasi:random/insecure-seed@0.2.0;
  export wasi:random/insecure@0.2.0;
  export wasi:random/random@0.2.0;
  export wasi:io/error@0.2.0;
  export wasi:io/poll@0.2.0;
  export wasi:io/streams@0.2.0;
  export wasi:clocks/monotonic-clock@0.2.0;
  export wasi:clocks/wall-clock@0.2.0;
  export wasi:sockets/network@0.2.0;
  export wasi:sockets/ip-name-lookup@0.2.0;
  export wasi:sockets/tcp@0.2.0;
  export wasi:sockets/udp@0.2.0;
  export wasi:sockets/instance-network@0.2.0;
  export wasi:sockets/tcp-create-socket@0.2.0;
  export wasi:sockets/udp-create-socket@0.2.0;
  export wasi:http/types@0.2.0;
  export wasi:http/incoming-handler@0.2.0;
  export wasi:http/outgoing-handler@0.2.0;
  export wasi:cli/stdin@0.2.0;
  export wasi:cli/stdout@0.2.0;
  export wasi:cli/stderr@0.2.0;
  export wasi:cli/terminal-input@0.2.0;
  export wasi:cli/terminal-output@0.2.0;
  export wasi:cli/terminal-stdin@0.2.0;
  export wasi:cli/terminal-stdout@0.2.0;
  export wasi:cli/terminal-stderr@0.2.0;
  export wasi:filesystem/types@0.2.0;
  export wasi:filesystem/preopens@0.2.0;
}

all

wasi-virt --allow-all -o virt.wasm

virt.wasm:

package root:component;

world root {
  import wasi:cli/environment@0.2.0;
  import wasi:config/runtime@0.2.0-draft;
  import wasi:io/error@0.2.0;
  import wasi:io/poll@0.2.0;
  import wasi:io/streams@0.2.0;
  import wasi:clocks/monotonic-clock@0.2.0;
  import wasi:sockets/network@0.2.0;
  import wasi:sockets/ip-name-lookup@0.2.0;
  import wasi:sockets/tcp@0.2.0;
  import wasi:sockets/udp@0.2.0;
  import wasi:http/types@0.2.0;
  import wasi:http/outgoing-handler@0.2.0;
  import wasi:cli/stdin@0.2.0;
  import wasi:cli/stdout@0.2.0;
  import wasi:cli/stderr@0.2.0;
  import wasi:cli/terminal-input@0.2.0;
  import wasi:cli/terminal-output@0.2.0;
  import wasi:cli/terminal-stdin@0.2.0;
  import wasi:cli/terminal-stdout@0.2.0;
  import wasi:cli/terminal-stderr@0.2.0;
  import wasi:clocks/wall-clock@0.2.0;
  import wasi:filesystem/types@0.2.0;
  import wasi:filesystem/preopens@0.2.0;

  export wasi:cli/environment@0.2.0;
  export wasi:config/runtime@0.2.0-draft;
  export wasi:io/error@0.2.0;
  export wasi:io/poll@0.2.0;
  export wasi:io/streams@0.2.0;
  export wasi:clocks/monotonic-clock@0.2.0;
  export wasi:sockets/ip-name-lookup@0.2.0;
  export wasi:sockets/tcp@0.2.0;
  export wasi:sockets/udp@0.2.0;
  export wasi:http/types@0.2.0;
  export wasi:http/outgoing-handler@0.2.0;
  export wasi:cli/stdin@0.2.0;
  export wasi:cli/stdout@0.2.0;
  export wasi:cli/stderr@0.2.0;
  export wasi:cli/terminal-input@0.2.0;
  export wasi:cli/terminal-output@0.2.0;
  export wasi:cli/terminal-stdin@0.2.0;
  export wasi:cli/terminal-stdout@0.2.0;
  export wasi:cli/terminal-stderr@0.2.0;
  export wasi:filesystem/types@0.2.0;
  export wasi:filesystem/preopens@0.2.0;
}

composed

The same component is wrapped for each of the examples below.

component.wasm:

package root:component;

world root {
  import wasi:logging/logging;
  import wasi:config/runtime@0.2.0-draft;

  export wasi:cli/run@0.2.0;
}

no host access

wasi-virt -o virt.wasm component.wasm

virt.wasm:

package root:component;

world root {
  import wasi:logging/logging;

  export wasi:cli/run@0.2.0;
}

runtime config only

wasi-virt --allow-config -o virt.wasm component.wasm

virt.wasm:

package root:component;

world root {
  import wasi:logging/logging;
  import wasi:config/runtime@0.2.0-draft;

  export wasi:cli/run@0.2.0;
}

all

wasi-virt --allow-all -o virt.wasm component.wasm

virt.wasm:

package root:component;

world root {
  import wasi:logging/logging;
  import wasi:config/runtime@0.2.0-draft;

  export wasi:cli/run@0.2.0;
}

all (without filtering added in #83)

wasi-virt --allow-all -o virt.wasm component.wasm

virt.wasm:

package root:component;

world root {
  import wasi:logging/logging;
  import wasi:cli/environment@0.2.0;
  import wasi:config/runtime@0.2.0-draft;
  import wasi:io/error@0.2.0;
  import wasi:io/poll@0.2.0;
  import wasi:io/streams@0.2.0;
  import wasi:clocks/monotonic-clock@0.2.0;
  import wasi:sockets/network@0.2.0;
  import wasi:sockets/ip-name-lookup@0.2.0;
  import wasi:sockets/tcp@0.2.0;
  import wasi:sockets/udp@0.2.0;
  import wasi:http/types@0.2.0;
  import wasi:http/outgoing-handler@0.2.0;
  import wasi:cli/stdin@0.2.0;
  import wasi:cli/stdout@0.2.0;
  import wasi:cli/stderr@0.2.0;
  import wasi:cli/terminal-input@0.2.0;
  import wasi:cli/terminal-output@0.2.0;
  import wasi:cli/terminal-stdin@0.2.0;
  import wasi:cli/terminal-stdout@0.2.0;
  import wasi:cli/terminal-stderr@0.2.0;
  import wasi:clocks/wall-clock@0.2.0;
  import wasi:filesystem/types@0.2.0;
  import wasi:filesystem/preopens@0.2.0;

  export wasi:cli/run@0.2.0;
}

- clarify config is runtime config in doc Signed-off-by: Scott Andrews <scott@andrews.me>

guybedford

Ahh, thanks for the summary, it seems this comment got overlooked still:

I wonder if we should change this to support "WASI sets", something like --allow-all --allow-core? Where this would be in the all set but not the core set?

I think we should separate the concept of "wasi core" here somehow, via something like --draft-proposals or otherwise through the options to act as another layer to avoid this being output for standard workflows that don't specifically have access to this subsystem.

The risk specifically being that a virt component wouldn't run on Wasmtime since it would expect these imports under the flags in use today.

scothis · 2024-08-28T21:37:33Z

What if we drop it from --allow-all so that explicit opt-in is required. More options can be introduced in the future.

The interface would still show up on the exports of the virt component, but that should be safe.

guybedford · 2024-08-28T21:47:59Z

The interface would still show up on the exports of the virt component, but that should be safe.

I believe we don't ever include subsystems that aren't explicitly included, are you sure this would still be the case? Cutting the bytes is important I think when features aren't needed.

scothis · 2024-08-29T01:25:49Z

I believe we don't ever include subsystems that aren't explicitly included, are you sure this would still be the case?

I followed the pattern from environment. When the subsystem isn't enabled the interface still exists but the functions are replaced.

WASI-Virt/src/lib.rs

Lines 273 to 278 in c451261

    
           // env, exit & random subsystems are fully independent 
        
           if self.env.is_some() { 
        
               resolve.merge_worlds(env_world, base_world)?; 
        
           } else { 
        
               strip_env_virt(&mut module)?; 
        
           }

WASI-Virt/src/virt_env.rs

Lines 233 to 250 in c451261

    
           /// Strip exported functions that implement the WASI CLI environment functionality 
        
           pub(crate) fn strip_env_virt(module: &mut Module) -> Result<()> { 
        
               stub_env_virt(module)?; 
        
               for fn_name in WASI_ENV_FNS { 
        
                   let Ok(fid) = module 
        
                       .exports 
        
                       .get_func(format!("wasi:cli/environment@0.2.0#{fn_name}")) 
        
                   else { 
        
                       bail!("Expected CLI function {fn_name}") 
        
                   }; 
        
                   module.replace_exported_func(fid, |(body, _)| { 
        
                       body.unreachable(); 
        
                   })?; 
        
               } 
        
               Ok(()) 
        
           }

Cutting the bytes is important I think when features aren't needed.

I generated a set of virt components with this branch and from main@c451261. The latent capability adds 3K over the equivlent., actually using --allow-config adds 1K. The delta between --allow-config and --allow-env is -1K. The delta between --allow-config and --allow-all is 88K.

219K branch-allow-all.wasm
131K branch-allow-config.wasm
132K branch-allow-env.wasm
130K branch-none.wasm
216K main-allow-all.wasm
130K main-allow-env.wasm
127K main-none.wasm

guybedford · 2024-08-29T17:04:29Z

Thanks for confirming - so the exports are still written even when the subsystem is unused, but that isn't an issue for compatibility because exports are always optional for importers. And from a size perspective, we get exactly the same DCE situation as for env. The case where an unused subsystem generates exports are always stub exports anyway and this is the standard approach already, and this PR fully implements strip and stub virt operations to align with that.

Appreciated for sharing the numbers as well to verify the DCE!

guybedford reviewed Aug 19, 2024

View reviewed changes

scothis mentioned this pull request Aug 27, 2024

Limit imported interfaces for composed components #83

Merged

scothis force-pushed the wasi-config-runtime branch from b1d6040 to e6b55aa Compare August 28, 2024 19:10

scothis force-pushed the wasi-config-runtime branch from e6b55aa to 8313bf9 Compare August 28, 2024 19:12

scothis commented Aug 28, 2024

View reviewed changes

scothis marked this pull request as ready for review August 28, 2024 19:15

scothis requested a review from guybedford August 28, 2024 19:15

guybedford approved these changes Aug 28, 2024

View reviewed changes

src/bin/wasi-virt.rs Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

Review feedback

49a3e70

- clarify config is runtime config in doc Signed-off-by: Scott Andrews <scott@andrews.me>

scothis requested a review from guybedford August 28, 2024 21:14

guybedford requested changes Aug 28, 2024

View reviewed changes

guybedford merged commit 6f9f14f into bytecodealliance:main Aug 29, 2024
2 checks passed

scothis deleted the wasi-config-runtime branch August 29, 2024 17:52

scothis mentioned this pull request Aug 29, 2024

Mark wasi:config/runtime as experimental #86

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for `wasi:config/runtime` virtualization #82

Add support for `wasi:config/runtime` virtualization #82

scothis commented Aug 9, 2024

guybedford commented Aug 12, 2024

scothis commented Aug 15, 2024

guybedford commented Aug 16, 2024

scothis commented Aug 16, 2024

guybedford left a comment

scothis Aug 28, 2024

scothis commented Aug 28, 2024

guybedford left a comment

scothis commented Aug 28, 2024

guybedford left a comment

scothis commented Aug 28, 2024

guybedford commented Aug 28, 2024

scothis commented Aug 29, 2024

guybedford commented Aug 29, 2024

Add support for wasi:config/runtime virtualization #82

Add support for wasi:config/runtime virtualization #82

Conversation

scothis commented Aug 9, 2024

guybedford commented Aug 12, 2024

scothis commented Aug 15, 2024

guybedford commented Aug 16, 2024

scothis commented Aug 16, 2024

guybedford left a comment

Choose a reason for hiding this comment

scothis Aug 28, 2024

Choose a reason for hiding this comment

scothis commented Aug 28, 2024

guybedford left a comment

Choose a reason for hiding this comment

scothis commented Aug 28, 2024

no host access

stdio only

runtime config only

all

composed

no host access

runtime config only

all

all (without filtering added in #83)

guybedford left a comment

Choose a reason for hiding this comment

scothis commented Aug 28, 2024

guybedford commented Aug 28, 2024

scothis commented Aug 29, 2024

guybedford commented Aug 29, 2024

Add support for `wasi:config/runtime` virtualization #82

Add support for `wasi:config/runtime` virtualization #82