Skip to content

Commit

Permalink
Add support for std*: TransformStream option (#938)
Browse files Browse the repository at this point in the history
  • Loading branch information
ehmicky committed Mar 30, 2024
1 parent 6ca44fb commit 45824d6
Show file tree
Hide file tree
Showing 15 changed files with 478 additions and 64 deletions.
21 changes: 19 additions & 2 deletions docs/transform.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,9 +173,13 @@ const {stdout} = await execa('./command.js', {stdout: {transform, final}});
console.log(stdout); // Ends with: 'Number of lines: 54'
```

## Node.js Duplex/Transform streams
## Duplex/Transform streams

A Node.js [`Duplex`](https://nodejs.org/api/stream.html#class-streamduplex) or [`Transform`](https://nodejs.org/api/stream.html#class-streamtransform) stream can be used instead of a generator function. A `{transform}` plain object must be passed. The [`objectMode`](#object-mode) transform option can be used, but not the [`binary`](#encoding) nor [`preserveNewlines`](#newlines) options.
A [`Duplex`](https://nodejs.org/api/stream.html#class-streamduplex) stream, Node.js [`Transform`](https://nodejs.org/api/stream.html#class-streamtransform) stream or web [`TransformStream`](https://developer.mozilla.org/en-US/docs/Web/API/TransformStream) can be used instead of a generator function.

Like generator functions, web `TransformStream` can be passed either directly or as a `{transform}` plain object. But `Duplex` and `Transform` must always be passed as a `{transform}` plain object.

The [`objectMode`](#object-mode) transform option can be used, but not the [`binary`](#encoding) nor [`preserveNewlines`](#newlines) options.

```js
import {createGzip} from 'node:zlib';
Expand All @@ -185,6 +189,13 @@ const {stdout} = await execa('./run.js', {stdout: {transform: createGzip()}});
console.log(stdout); // `stdout` is compressed with gzip
```

```js
import {execa} from 'execa';

const {stdout} = await execa('./run.js', {stdout: new CompressionStream('gzip')});
console.log(stdout); // `stdout` is compressed with gzip
```

## Combining

The [`stdin`](../readme.md#stdin), [`stdout`](../readme.md#stdout-1), [`stderr`](../readme.md#stderr-1) and [`stdio`](../readme.md#stdio-1) options can accept an array of values. While this is not specific to transforms, this can be useful with them too. For example, the following transform impacts the value printed by `inherit`.
Expand All @@ -199,6 +210,12 @@ This also allows using multiple transforms.
await execa('echo', ['hello'], {stdout: [transform, otherTransform]});
```

Or saving to files.

```js
await execa('./run.js', {stdout: [new CompressionStream('gzip'), {file: './output.gz'}]});
```

## Async iteration

In some cases, [iterating](../readme.md#iterablereadableoptions) over the subprocess can be an alternative to transforms.
Expand Down
19 changes: 13 additions & 6 deletions index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@ type DuplexTransform = {
objectMode?: boolean;
};

type WebTransform = {
transform: TransformStream;
objectMode?: boolean;
};

type CommonStdioOption<IsSync extends boolean = boolean> =
| BaseStdioOption
| 'ipc'
Expand All @@ -47,7 +52,9 @@ type CommonStdioOption<IsSync extends boolean = boolean> =
| IfAsync<IsSync,
| GeneratorTransform
| GeneratorTransformFull
| DuplexTransform>;
| DuplexTransform
| WebTransform
| TransformStream>;

type InputStdioOption<IsSync extends boolean = boolean> =
| Uint8Array
Expand Down Expand Up @@ -126,13 +133,13 @@ type IsObjectOutputOptions<OutputOptions extends StdioOption> = IsObjectOutputOp
: OutputOptions
>;

type IsObjectOutputOption<OutputOption extends StdioSingleOption> = OutputOption extends GeneratorTransformFull
type IsObjectOutputOption<OutputOption extends StdioSingleOption> = OutputOption extends GeneratorTransformFull | WebTransform
? BooleanObjectMode<OutputOption['objectMode']>
: OutputOption extends DuplexTransform
? DuplexObjectMode<OutputOption>
: false;

type BooleanObjectMode<ObjectModeOption extends GeneratorTransformFull['objectMode']> = ObjectModeOption extends true ? true : false;
type BooleanObjectMode<ObjectModeOption extends boolean | undefined> = ObjectModeOption extends true ? true : false;

type DuplexObjectMode<OutputOption extends DuplexTransform> = OutputOption['objectMode'] extends boolean
? OutputOption['objectMode']
Expand Down Expand Up @@ -357,7 +364,7 @@ type CommonOptions<IsSync extends boolean = boolean> = {
This can be an [array of values](https://github.com/sindresorhus/execa#redirect-stdinstdoutstderr-to-multiple-destinations) such as `['inherit', 'pipe']` or `[filePath, 'pipe']`.
This can also be a generator function or a [`Duplex`](https://nodejs.org/api/stream.html#class-streamduplex) to transform the input. [Learn more.](https://github.com/sindresorhus/execa/tree/main/docs/transform.md)
This can also be a generator function or a [`Duplex`](https://nodejs.org/api/stream.html#class-streamduplex) or a [web `TransformStream`](https://developer.mozilla.org/en-US/docs/Web/API/TransformStream) to transform the input. [Learn more.](https://github.com/sindresorhus/execa/tree/main/docs/transform.md)
@default `inherit` with `$`, `pipe` otherwise
*/
Expand All @@ -377,7 +384,7 @@ type CommonOptions<IsSync extends boolean = boolean> = {
This can be an [array of values](https://github.com/sindresorhus/execa#redirect-stdinstdoutstderr-to-multiple-destinations) such as `['inherit', 'pipe']` or `[filePath, 'pipe']`.
This can also be a generator function or a [`Duplex`](https://nodejs.org/api/stream.html#class-streamduplex) to transform the output. [Learn more.](https://github.com/sindresorhus/execa/tree/main/docs/transform.md)
This can also be a generator function or a [`Duplex`](https://nodejs.org/api/stream.html#class-streamduplex) or a [web `TransformStream`](https://developer.mozilla.org/en-US/docs/Web/API/TransformStream) to transform the output. [Learn more.](https://github.com/sindresorhus/execa/tree/main/docs/transform.md)
@default 'pipe'
*/
Expand All @@ -397,7 +404,7 @@ type CommonOptions<IsSync extends boolean = boolean> = {
This can be an [array of values](https://github.com/sindresorhus/execa#redirect-stdinstdoutstderr-to-multiple-destinations) such as `['inherit', 'pipe']` or `[filePath, 'pipe']`.
This can also be a generator function or a [`Duplex`](https://nodejs.org/api/stream.html#class-streamduplex) to transform the output. [Learn more.](https://github.com/sindresorhus/execa/tree/main/docs/transform.md)
This can also be a generator function or a [`Duplex`](https://nodejs.org/api/stream.html#class-streamduplex) or a [web `TransformStream`](https://developer.mozilla.org/en-US/docs/Web/API/TransformStream) to transform the output. [Learn more.](https://github.com/sindresorhus/execa/tree/main/docs/transform.md)
@default 'pipe'
*/
Expand Down
86 changes: 86 additions & 0 deletions index.test-d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ const duplexNotObject = {transform: duplexStream as Duplex & {readableObjectMode
const duplexObjectProperty = {transform: duplexStream, objectMode: true as const};
const duplexNotObjectProperty = {transform: duplexStream, objectMode: false as const};
const duplexTransform = {transform: new Transform()};
const webTransformInstance = new TransformStream();
const webTransform = {transform: webTransformInstance};
const webTransformObject = {transform: webTransformInstance, objectMode: true as const};
const webTransformNotObject = {transform: webTransformInstance, objectMode: false as const};

type AnySyncChunk = string | Uint8Array | undefined;
type AnyChunk = AnySyncChunk | string[] | unknown[];
Expand Down Expand Up @@ -674,6 +678,10 @@ try {
expectType<unknown[]>(objectTransformLinesStdoutResult.stdout);
expectType<[undefined, unknown[], string[]]>(objectTransformLinesStdoutResult.stdio);

const objectWebTransformStdoutResult = await execa('unicorns', {stdout: webTransformObject});
expectType<unknown[]>(objectWebTransformStdoutResult.stdout);
expectType<[undefined, unknown[], string]>(objectWebTransformStdoutResult.stdio);

const objectDuplexStdoutResult = await execa('unicorns', {stdout: duplexObject});
expectType<unknown[]>(objectDuplexStdoutResult.stdout);
expectType<[undefined, unknown[], string]>(objectDuplexStdoutResult.stdio);
Expand All @@ -686,6 +694,10 @@ try {
expectType<unknown[]>(objectTransformStdoutResult.stdout);
expectType<[undefined, unknown[], string]>(objectTransformStdoutResult.stdio);

const objectWebTransformStderrResult = await execa('unicorns', {stderr: webTransformObject});
expectType<unknown[]>(objectWebTransformStderrResult.stderr);
expectType<[undefined, string, unknown[]]>(objectWebTransformStderrResult.stdio);

const objectDuplexStderrResult = await execa('unicorns', {stderr: duplexObject});
expectType<unknown[]>(objectDuplexStderrResult.stderr);
expectType<[undefined, string, unknown[]]>(objectDuplexStderrResult.stdio);
Expand All @@ -698,6 +710,10 @@ try {
expectType<unknown[]>(objectTransformStderrResult.stderr);
expectType<[undefined, string, unknown[]]>(objectTransformStderrResult.stdio);

const objectWebTransformStdioResult = await execa('unicorns', {stdio: ['pipe', 'pipe', webTransformObject]});
expectType<unknown[]>(objectWebTransformStdioResult.stderr);
expectType<[undefined, string, unknown[]]>(objectWebTransformStdioResult.stdio);

const objectDuplexStdioResult = await execa('unicorns', {stdio: ['pipe', 'pipe', duplexObject]});
expectType<unknown[]>(objectDuplexStdioResult.stderr);
expectType<[undefined, string, unknown[]]>(objectDuplexStdioResult.stdio);
Expand All @@ -710,6 +726,10 @@ try {
expectType<unknown[]>(objectTransformStdioResult.stderr);
expectType<[undefined, string, unknown[]]>(objectTransformStdioResult.stdio);

const singleObjectWebTransformStdoutResult = await execa('unicorns', {stdout: [webTransformObject]});
expectType<unknown[]>(singleObjectWebTransformStdoutResult.stdout);
expectType<[undefined, unknown[], string]>(singleObjectWebTransformStdoutResult.stdio);

const singleObjectDuplexStdoutResult = await execa('unicorns', {stdout: [duplexObject]});
expectType<unknown[]>(singleObjectDuplexStdoutResult.stdout);
expectType<[undefined, unknown[], string]>(singleObjectDuplexStdoutResult.stdio);
Expand All @@ -722,6 +742,10 @@ try {
expectType<unknown[]>(singleObjectTransformStdoutResult.stdout);
expectType<[undefined, unknown[], string]>(singleObjectTransformStdoutResult.stdio);

const manyObjectWebTransformStdoutResult = await execa('unicorns', {stdout: [webTransformObject, webTransformObject]});
expectType<unknown[]>(manyObjectWebTransformStdoutResult.stdout);
expectType<[undefined, unknown[], string]>(manyObjectWebTransformStdoutResult.stdio);

const manyObjectDuplexStdoutResult = await execa('unicorns', {stdout: [duplexObject, duplexObject]});
expectType<unknown[]>(manyObjectDuplexStdoutResult.stdout);
expectType<[undefined, unknown[], string]>(manyObjectDuplexStdoutResult.stdio);
Expand All @@ -734,6 +758,10 @@ try {
expectType<unknown[]>(manyObjectTransformStdoutResult.stdout);
expectType<[undefined, unknown[], string]>(manyObjectTransformStdoutResult.stdio);

const falseObjectWebTransformStdoutResult = await execa('unicorns', {stdout: webTransformNotObject});
expectType<string>(falseObjectWebTransformStdoutResult.stdout);
expectType<[undefined, string, string]>(falseObjectWebTransformStdoutResult.stdio);

const falseObjectDuplexStdoutResult = await execa('unicorns', {stdout: duplexNotObject});
expectType<string>(falseObjectDuplexStdoutResult.stdout);
expectType<[undefined, string, string]>(falseObjectDuplexStdoutResult.stdio);
Expand All @@ -746,6 +774,10 @@ try {
expectType<string>(falseObjectTransformStdoutResult.stdout);
expectType<[undefined, string, string]>(falseObjectTransformStdoutResult.stdio);

const falseObjectWebTransformStderrResult = await execa('unicorns', {stderr: webTransformNotObject});
expectType<string>(falseObjectWebTransformStderrResult.stderr);
expectType<[undefined, string, string]>(falseObjectWebTransformStderrResult.stdio);

const falseObjectDuplexStderrResult = await execa('unicorns', {stderr: duplexNotObject});
expectType<string>(falseObjectDuplexStderrResult.stderr);
expectType<[undefined, string, string]>(falseObjectDuplexStderrResult.stdio);
Expand All @@ -758,6 +790,10 @@ try {
expectType<string>(falseObjectTransformStderrResult.stderr);
expectType<[undefined, string, string]>(falseObjectTransformStderrResult.stdio);

const falseObjectWebTransformStdioResult = await execa('unicorns', {stdio: ['pipe', 'pipe', webTransformNotObject]});
expectType<string>(falseObjectWebTransformStdioResult.stderr);
expectType<[undefined, string, string]>(falseObjectWebTransformStdioResult.stdio);

const falseObjectDuplexStdioResult = await execa('unicorns', {stdio: ['pipe', 'pipe', duplexNotObject]});
expectType<string>(falseObjectDuplexStdioResult.stderr);
expectType<[undefined, string, string]>(falseObjectDuplexStdioResult.stdio);
Expand All @@ -770,6 +806,14 @@ try {
expectType<string>(falseObjectTransformStdioResult.stderr);
expectType<[undefined, string, string]>(falseObjectTransformStdioResult.stdio);

const topObjectWebTransformStdoutResult = await execa('unicorns', {stdout: webTransformInstance});
expectType<string>(topObjectWebTransformStdoutResult.stdout);
expectType<[undefined, string, string]>(topObjectWebTransformStdoutResult.stdio);

const undefinedObjectWebTransformStdoutResult = await execa('unicorns', {stdout: webTransform});
expectType<string>(undefinedObjectWebTransformStdoutResult.stdout);
expectType<[undefined, string, string]>(undefinedObjectWebTransformStdoutResult.stdio);

const undefinedObjectDuplexStdoutResult = await execa('unicorns', {stdout: duplex});
expectType<string | unknown[]>(undefinedObjectDuplexStdoutResult.stdout);
expectType<[undefined, string | unknown[], string]>(undefinedObjectDuplexStdoutResult.stdio);
Expand Down Expand Up @@ -1279,6 +1323,16 @@ execa('unicorns', {stdin: [duplexTransform]});
expectError(execaSync('unicorns', {stdin: [duplexTransform]}));
expectError(execa('unicorns', {stdin: {...duplex, objectMode: 'true'}}));
expectError(execaSync('unicorns', {stdin: {...duplex, objectMode: 'true'}}));
execa('unicorns', {stdin: webTransformInstance});
expectError(execaSync('unicorns', {stdin: webTransformInstance}));
execa('unicorns', {stdin: [webTransformInstance]});
expectError(execaSync('unicorns', {stdin: [webTransformInstance]}));
execa('unicorns', {stdin: webTransform});
expectError(execaSync('unicorns', {stdin: webTransform}));
execa('unicorns', {stdin: [webTransform]});
expectError(execaSync('unicorns', {stdin: [webTransform]}));
expectError(execa('unicorns', {stdin: {...webTransform, objectMode: 'true'}}));
expectError(execaSync('unicorns', {stdin: {...webTransform, objectMode: 'true'}}));
execa('unicorns', {stdin: unknownGenerator});
expectError(execaSync('unicorns', {stdin: unknownGenerator}));
execa('unicorns', {stdin: [unknownGenerator]});
Expand Down Expand Up @@ -1392,6 +1446,16 @@ execa('unicorns', {stdout: [duplexTransform]});
expectError(execaSync('unicorns', {stdout: [duplexTransform]}));
expectError(execa('unicorns', {stdout: {...duplex, objectMode: 'true'}}));
expectError(execaSync('unicorns', {stdout: {...duplex, objectMode: 'true'}}));
execa('unicorns', {stdout: webTransformInstance});
expectError(execaSync('unicorns', {stdout: webTransformInstance}));
execa('unicorns', {stdout: [webTransformInstance]});
expectError(execaSync('unicorns', {stdout: [webTransformInstance]}));
execa('unicorns', {stdout: webTransform});
expectError(execaSync('unicorns', {stdout: webTransform}));
execa('unicorns', {stdout: [webTransform]});
expectError(execaSync('unicorns', {stdout: [webTransform]}));
expectError(execa('unicorns', {stdout: {...webTransform, objectMode: 'true'}}));
expectError(execaSync('unicorns', {stdout: {...webTransform, objectMode: 'true'}}));
execa('unicorns', {stdout: unknownGenerator});
expectError(execaSync('unicorns', {stdout: unknownGenerator}));
execa('unicorns', {stdout: [unknownGenerator]});
Expand Down Expand Up @@ -1505,6 +1569,16 @@ execa('unicorns', {stderr: [duplexTransform]});
expectError(execaSync('unicorns', {stderr: [duplexTransform]}));
expectError(execa('unicorns', {stderr: {...duplex, objectMode: 'true'}}));
expectError(execaSync('unicorns', {stderr: {...duplex, objectMode: 'true'}}));
execa('unicorns', {stderr: webTransformInstance});
expectError(execaSync('unicorns', {stderr: webTransformInstance}));
execa('unicorns', {stderr: [webTransformInstance]});
expectError(execaSync('unicorns', {stderr: [webTransformInstance]}));
execa('unicorns', {stderr: webTransform});
expectError(execaSync('unicorns', {stderr: webTransform}));
execa('unicorns', {stderr: [webTransform]});
expectError(execaSync('unicorns', {stderr: [webTransform]}));
expectError(execa('unicorns', {stderr: {...webTransform, objectMode: 'true'}}));
expectError(execaSync('unicorns', {stderr: {...webTransform, objectMode: 'true'}}));
execa('unicorns', {stderr: unknownGenerator});
expectError(execaSync('unicorns', {stderr: unknownGenerator}));
execa('unicorns', {stderr: [unknownGenerator]});
Expand Down Expand Up @@ -1596,6 +1670,10 @@ expectError(execa('unicorns', {stdio: duplex}));
expectError(execaSync('unicorns', {stdio: duplex}));
expectError(execa('unicorns', {stdio: duplexTransform}));
expectError(execaSync('unicorns', {stdio: duplexTransform}));
expectError(execa('unicorns', {stdio: webTransformInstance}));
expectError(execaSync('unicorns', {stdio: webTransformInstance}));
expectError(execa('unicorns', {stdio: webTransform}));
expectError(execaSync('unicorns', {stdio: webTransform}));
expectError(execa('unicorns', {stdio: new Writable()}));
expectError(execaSync('unicorns', {stdio: new Writable()}));
expectError(execa('unicorns', {stdio: new Readable()}));
Expand Down Expand Up @@ -1653,6 +1731,8 @@ execa('unicorns', {
{file: './test'},
duplex,
duplexTransform,
webTransformInstance,
webTransform,
new Writable(),
new Readable(),
new WritableStream(),
Expand Down Expand Up @@ -1683,6 +1763,8 @@ expectError(execaSync('unicorns', {stdio: [unknownGenerator]}));
expectError(execaSync('unicorns', {stdio: [{transform: unknownGenerator}]}));
expectError(execaSync('unicorns', {stdio: [duplex]}));
expectError(execaSync('unicorns', {stdio: [duplexTransform]}));
expectError(execaSync('unicorns', {stdio: [webTransformInstance]}));
expectError(execaSync('unicorns', {stdio: [webTransform]}));
expectError(execaSync('unicorns', {stdio: [new WritableStream()]}));
expectError(execaSync('unicorns', {stdio: [new ReadableStream()]}));
expectError(execaSync('unicorns', {stdio: [emptyStringGenerator()]}));
Expand All @@ -1708,6 +1790,8 @@ execa('unicorns', {
[{file: './test'}],
[duplex],
[duplexTransform],
[webTransformInstance],
[webTransform],
[new Writable()],
[new Readable()],
[new WritableStream()],
Expand Down Expand Up @@ -1742,6 +1826,8 @@ expectError(execaSync('unicorns', {stdio: [[unknownGenerator]]}));
expectError(execaSync('unicorns', {stdio: [[{transform: unknownGenerator}]]}));
expectError(execaSync('unicorns', {stdio: [[duplex]]}));
expectError(execaSync('unicorns', {stdio: [[duplexTransform]]}));
expectError(execaSync('unicorns', {stdio: [[webTransformInstance]]}));
expectError(execaSync('unicorns', {stdio: [[webTransform]]}));
expectError(execaSync('unicorns', {stdio: [[new WritableStream()]]}));
expectError(execaSync('unicorns', {stdio: [[new ReadableStream()]]}));
expectError(execaSync('unicorns', {stdio: [[['foo', 'bar']]]}));
Expand Down
7 changes: 6 additions & 1 deletion lib/stdio/async.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import {createReadStream, createWriteStream} from 'node:fs';
import {Buffer} from 'node:buffer';
import {Readable, Writable} from 'node:stream';
import {Readable, Writable, Duplex} from 'node:stream';
import mergeStreams from '@sindresorhus/merge-streams';
import {isStandardStream, incrementMaxListeners} from '../utils.js';
import {handleInput} from './handle.js';
Expand All @@ -18,6 +18,11 @@ const forbiddenIfAsync = ({type, optionName}) => {
const addProperties = {
generator: generatorToDuplexStream,
nodeStream: ({value}) => ({stream: value}),
webTransform({value: {transform, writableObjectMode, readableObjectMode}}) {
const objectMode = writableObjectMode || readableObjectMode;
const stream = Duplex.fromWeb(transform, {objectMode});
return {stream};
},
duplex: ({value: {transform}}) => ({stream: transform}),
native() {},
};
Expand Down
1 change: 1 addition & 0 deletions lib/stdio/direction.js
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ const guessStreamDirection = {

return isNodeWritableStream(value, {checkOpen: false}) ? undefined : 'input';
},
webTransform: anyDirection,
duplex: anyDirection,
native(value) {
const standardStreamDirection = getStandardStreamDirection(value);
Expand Down
Loading

0 comments on commit 45824d6

Please sign in to comment.