Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add netstring+Stream utility #1807

Closed
warner opened this issue Sep 19, 2020 · 0 comments · Fixed by #1808
Closed

add netstring+Stream utility #1807

warner opened this issue Sep 19, 2020 · 0 comments · Fixed by #1808
Assignees
Labels
enhancement New feature or request SwingSet package: SwingSet

Comments

@warner
Copy link
Member

warner commented Sep 19, 2020

What is the Problem Being Solved?

The kernel-to-subprocess protocol uses netstrings to frame each message. In #1797 I discovered that the library we were using for this fails to handle partial chunks in the stream decoder, which breaks the protocol any time the delivery message is larger than the OS kernel's pipe buffer (typically about 8kb, but it varies by OS, which is why the test passed on Linux but failed on macOS).

I haven't been able to find any correct libraries that offer a Stream interface, so I have to write my own.

Description of the Design

Four functions:

  • encode: takes a Buffer, returns a Buffer
  • decode: takes a Buffer, returns a list of Buffers (for the completed netstrings) plus a leftover Buffer
  • createWriter: creates a Transform stream that takes Buffers (each of which is a single payload) and emits Buffers (to be concatenated out to a byte pipe)
  • createReader: creates a Transform stream that takes Buffers (which are chunks that do not necessarily align to netstring boundaries, but when all are received and concatenated, will form one or more netstrings) and emits Buffers (one payload each)

Security Considerations

The usual memory-consumption attack exists (the sender just writes 99999.. forever, and the recipient keeps buffering the supposed length prefix forever in hopes of seeing the termination colon). We control both sides of the pipe, however, so I'm not going to try to address it.

Test Plan

Unit tests.

@warner warner added enhancement New feature or request SwingSet package: SwingSet labels Sep 19, 2020
@warner warner self-assigned this Sep 19, 2020
warner added a commit that referenced this issue Sep 19, 2020
For use by the kernel-worker protocol.

refs #1797
closes #1807
warner added a commit that referenced this issue Sep 19, 2020
For use by the kernel-worker protocol.

refs #1797
closes #1807
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request SwingSet package: SwingSet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant