Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add VatWorker for Node.js-based child process #1374

Closed
warner opened this issue Aug 5, 2020 · 1 comment
Closed

add VatWorker for Node.js-based child process #1374

warner opened this issue Aug 5, 2020 · 1 comment
Assignees
Labels
enhancement New feature or request SwingSet package: SwingSet

Comments

@warner
Copy link
Member

warner commented Aug 5, 2020

We're rearranging the way the swingset kernel interacts with vats. #1339 introduced a "Vat Manager", which provides a unified data-centric interface for all kinds of vats, where serializable VatDeliveryObject and VatSyscallObjects can be passed back and forth. It factored out all the kernel/vat translation code into a common layer, and defined a VatManager interface with a .deliver() method. Once the Vat Manager is created, the kernel only interacts with the manager. The manager then owns a "Vat Worker", which communicates exclusively with the manager and hosts the actual vat code.

#1366 will add a second kind of worker, which launches a new Node.js thread (the Worker object) to host the vat code, and then uses postMessage (and its "structured clone" mechanism) to send deliveries and syscalls over the thread boundary. The original approach is now known as a "local worker". When adding a vat, a new managerType option lets you choose between local and nodeWorker.

This ticket is about adding a third "node-process" type, which spawns a new child process to host the vat code. The delivery/syscall messages will be serialized to bytes, framed in netstrings, and sent over a pipe. The child process will do a blocking read from the parent, which allows us to stall the child while the kernel performs device invocations. This will allow the node-process worker to use device reads, which the node-thread worker cannot (unless/until we involve exotic tricks with Atomics and SharedArrayBuffer).

The child process will start from a supervisor file that lives in the swingset tree, just like the node-thread worker spawns a thread which starts from src/kernel/vatManager/nodeWorkerSupervisor.js. This supervisor program will import ses, liveslots and import-bundle. It starts by installing SES, then establishes the pipe-message handlers, then waits for instructions. The first instruction will be to load a bundle with import-bundle, then extract the buildRootObject method and use it to build a liveslots instance. Then subsequent instructions will contain deliveries to pass into liveslots, which dispatches them into the vat's objects. Syscalls are handled by sending requests back over the channel to the kernel.

The kernel side of the node-process worker will look nearly identical to @dckc 's #1299 xs-process worker: it uses the same serialization and messaging. The only difference should be the program that it launches.

@warner warner added enhancement New feature or request SwingSet package: SwingSet labels Aug 5, 2020
@warner warner self-assigned this Aug 5, 2020
warner added a commit that referenced this issue Aug 7, 2020
This adds a new `managerType` named `'subprocess-node'`, which runs Node.js
in a subprocess, and sends it netstring-wrapped JSON-encoded Delivery/Syscall
objects to drive the vat.

It currently has the same limitations as the thread-based worker:

* vatPowers is missing transformTildot
* metering is not implemented at all
* delivery transcripts (and replay) are not yet implemented

In addition, it does not yet use a blocking read for syscall responses, so it
can not yet invoke devices. However, unlike the thread worker, blocking reads
should be possible for a subprocess, so eventually we will have this ability.

refs #1374
warner added a commit that referenced this issue Aug 7, 2020
This adds a new `managerType` named `'subprocess-node'`, which runs Node.js
in a subprocess, and sends it netstring-wrapped JSON-encoded Delivery/Syscall
objects to drive the vat.

It currently has the same limitations as the thread-based worker:

* vatPowers is missing transformTildot
* metering is not implemented at all
* delivery transcripts (and replay) are not yet implemented

In addition, it does not yet use a blocking read for syscall responses, so it
can not yet invoke devices. However, unlike the thread worker, blocking reads
should be possible for a subprocess, so eventually we will have this ability.

refs #1374
warner added a commit that referenced this issue Aug 7, 2020
This adds a new `managerType` named `'subprocess-node'`, which runs Node.js
in a subprocess, and sends it netstring-wrapped JSON-encoded Delivery/Syscall
objects to drive the vat.

It currently has the same limitations as the thread-based worker:

* vatPowers is missing transformTildot
* metering is not implemented at all
* delivery transcripts (and replay) are not yet implemented

In addition, it does not yet use a blocking read for syscall responses, so it
can not yet invoke devices. However, unlike the thread worker, blocking reads
should be possible for a subprocess, so eventually we will have this ability.

refs #1374
@warner
Copy link
Member Author

warner commented Jan 22, 2021

This exists now.

@warner warner closed this as completed Jan 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request SwingSet package: SwingSet
Projects
None yet
Development

No branches or pull requests

1 participant