K6 process hangs on larger test files due to Go buffer size issue

[Lordnibbler]

When using

os.Pipe()

in Go, pipes have a fixed buffer size. If the amount of data written to the pipe exceeds the buffer's capacity and there is no concurrent reader, the write operation will block until space becomes available in the buffer. This behavior can cause your program to hang if the buffer is full and the reading end (the other side of the pipe) is not actively reading the data.

This manifests as the k6 process hanging if you have somewhat large k6 javascript/typescript test files to be processed by xk6-ts loader.go file

To reproduce this simply, you can create a contrived k6 test file which is quite long (the contents of which are unimportant, I use an empty function here), simulating what it might be like in a larger codebase with multiple files and imports:

// helloworld.js

export const options = {
    scenarios: {
        per_vu_scenario: {
            executor: "per-vu-iterations",
            vus: 10,
            iterations: 10,
            startTime: "10s",
        },
    },
};

function noop() {}

export default function () {
    noop();
    noop();
    noop();
    // truncating for brevity, but repeat this function call 1500 times or so to make the file sufficiently long.
    // this must be explicit function calls, not a loop calling the function 1500 times.
}

Build k6 with xk6-ts, and run this example:

xk6 build v0.50.0 --output "./k6" --with github.com/szkiba/xk6-ts@latest
./k6 run helloworld.js

And k6 will hang indefinitely.

To fix this, use concurrent Goroutines for reading and writing. Ensure that reading from and writing to the pipe happen concurrently. This approach prevents the buffer from filling up by continuously emptying it while writing. I propose a fix similar to this as a starting point, but I am not a golang expert, so was hoping @szkiba could assist

func redirectStdin() {
    if os.Getenv("XK6_TS") == "false" {
        return
    }

    isRun, scriptIndex := isRunCommand(os.Args)
    if !isRun {
        return
    }

    filename := os.Args[scriptIndex]
    if filename == "-" {
        return
    }

    opts := &k6pack.Options{
        Filename:  filename,
        SourceMap: os.Getenv("XK6_TS_SOURCEMAP") != "false",
    }

    source, err := os.ReadFile(filepath.Clean(filename))
    if err != nil {
        logrus.WithError(err).Fatal()
        return
    }

	packStarted := time.Now()

    jsScript, err := k6pack.Pack(string(source), opts)
    if err != nil {
        logrus.WithError(err).Fatal()
        return
    }

    if os.Getenv("XK6_TS_BENCHMARK") == "true" {
	duration := time.Since(packStarted)
	logrus.WithField("extension", "xk6-ts").WithField("duration", duration).Info("Bundling completed in ", duration)
    }

    reader, writer, err := os.Pipe()
    if err != nil {
        logrus.WithError(err).Fatal()
        return
    }

    // Set up os.Stdin to be the reader before any writing happens
    originalStdin := os.Stdin
    os.Stdin = reader

    // cleanup after function exit
    defer func() {
        os.Stdin = originalStdin
        // Ensure the writer and reader is closed when done
        writer.Close() 
        reader.Close()
    }()

    // Write to the pipe in a goroutine
    go func() {
        // Close writer after writing to signal EOF
        defer writer.Close() 
        _, writeErr := writer.Write(jsScript)
        if writeErr != nil {
            logrus.WithError(writeErr).Error("Failed to write JS script to pipe")
        }
    }()
}

[szkiba]

@Lordnibbler thank you for reporting this (and thank you for this good quality report). I'll fix it soon.

[szkiba]

@Lordnibbler thanks again, v0.2.4 contains the fix

[Lordnibbler]

Thanks for the quick turnaround, this solved the issue for us!