Merge pull request #22 from szkiba/v0.4.0

V0.4.0
grafana · Oct 10, 2023 · ad6da2c · ad6da2c
2 parents 8bebbb1 + f853056
commit ad6da2c
Show file tree

Hide file tree

Showing 39 changed files with 2,515 additions and 777 deletions.
diff --git a/README.md b/README.md
@@ -12,18 +12,19 @@ k6x is a [k6](https://k6.io) launcher that automatically provides k6 with the [e
 
 The build step uses a [Docker Engine](https://docs.docker.com/engine/) (even a [remote one](#remote-docker)), so no other local tools (go, git, docker cli, etc.) are needed, just k6x. If [Go language toolkit is installed](https://go.dev/doc/install), the build step uses it instead of Docker Engine. In this case, Docker Engine is not needed and build will be faster.
 
-**asciicast (native builder)**
+The (experimental) [builder service](#builder-service) makes the use of extensions even easier, no tools need to be installed except for k6x. The [builder service](#builder-service) builds the k6 binary on the fly.
 
-[![asciicast](https://asciinema.org/a/610337.svg)](https://asciinema.org/a/610337)
+**asciicast**
 
-**asciicast (docker builder)**
-
-[![asciicast](https://asciinema.org/a/611493.svg)](https://asciinema.org/a/611493)
+[![asciicast](https://asciinema.org/a/612785.svg)](https://asciinema.org/a/612785)
 
 ## Prerequisites
 
 - Either a [Go language toolkit](https://go.dev/doc/install) or a properly configured [Docker Engine](https://docs.docker.com/engine/install/) environment (e.g. `DOCKER_HOST` environment variable)
 
+> **Warning**
+> When using the [builder service](#builder-service), you no longer need either the [Docker Engine](https://docs.docker.com/engine/install/) or the [Go language toolkit](https://go.dev/doc/install) to use k6x.
+
 ## Usage
 
 The `k6x` command is a drop-in replacement for the `k6` command, using exactly the same syntax. For example, running the test script (with `top` output extension):
@@ -70,14 +71,17 @@ The pragma syntax can also be used for output extensions:
 
 Any number of `"use k6"` pragmas can be used.
 
+> **Note**
+> The use of pragmas is completely optional, it is only necessary if you want to specify version constraints.
+
 Read the version constraints syntax in the [Version Constraints](#version-constraints) section of the [Appendix](#appendix)
 
 > **Warning**
 > Version constraints can reduce the efficiency of the cache, so it is advisable to use them only in justified cases. Even then, it is worth using permissive, upwardly open version constraints instead of specifying a specific version.
 
 ### Cache
 
-Reusable artifacts (k6 binary, HTTP responses) are stored in the subdirectory `k6x` under the directory defined by the `XDG_CACHE_HOME` environment variable. The default of `XDG_CACHE_HOME` depends on the operating system (Windows: `%LOCALAPPDATA%\cache`, Linux: `~/.cache`, macOS: `~/Library/Caches`)
+Reusable artifacts (k6 binary, HTTP responses) are stored in the subdirectory `k6x` under the directory defined by the `XDG_CACHE_HOME` environment variable. The default of `XDG_CACHE_HOME` depends on the operating system (Windows: `%LOCALAPPDATA%\cache`, Linux: `~/.cache`, macOS: `~/Library/Caches`). The default cache directory can be changed using the `K6X_CACHE_DIR` environment variable or the `--cache-dir` flag.
 
 The directory where k6x stores the compiled k6 binary can be specified in the `K6X_BIN_DIR` environment variable. If it is missing, the `.k6x` directory is used if it exists in the current working directory, otherwise the k6 binary is stored in the cache directory described above. In addition, the location of the directory used to store k6 can also be specified using the `--bin-dir` command line option. See the [Flags](#flags) section for more information.
 
@@ -96,18 +100,41 @@ The k6 subcommands are extended with some global command line flags related to b
   ```
   k6x run --bin-dir ./custom-k6 script.js
   ```
+- `--cache-dir path`  set cache base directory (it will overwrite the value of `K6X_CACHE_DIR`)
+
 - `--with dependency`  you can specify additional dependencies and version constraints, the form of the `dependency` is the same as that used in the `"use k6 with"` pragma (practically the same as the string after the `use k6 with`)
 
   ```
   k6x run --with k6/x/mock script.js
   ```
-
-- `--builder list` a comma-separated list of builders (default: `native,docker`), available builders:
+
+- `--filter expr` [jmespath](https://jmespath.org/) syntax extension registry [filter](#filtering) (default: `[*]`)
+
+   ```
+   k6x run --filter "[?contains(tiers,'Official')]" script.js
+   ```
+
+- `--builder list` a comma-separated list of builders (default: `service,native,docker`), available builders:
+  - `service` this builder uses the builder service if it is specified (in the `K6X_BUILDER_SERVICE` environment variable), otherwise the next builder will be used without error
   - `native` this builder uses the installed go compiler if available, otherwise the next builder is used without error
   - `docker` this builder uses Docker Engine, which can be local or remote (specified in `DOCKER_HOST` environment variable)
 
+    ```
+    k6x run --buider docker script.js
+    ```
+
+- `--replace name=path` replaces the module path, where `name` is the dependency/module name and `path` is a remote module path (version should be appended with `@`) or an absolute local file-system path (a path starting with `.` can also be used, which will be resolved to an absolute path). It implies the use of the `native` builder (`--builder native`) and clean flag (`--clean`)
+
+  *with local file-system path*
+
+  ```
+  k6x --replace k6/x/faker=../xk6-faker run script.js
+  ```
+
+  *with remote path*
+
   ```
-  k6x run --buider docker script.js
+  k6x --replace k6/x/faker=github.com/my-user/xk6-faker@latest run script.js
   ```
 
 ### Subcommands
@@ -122,7 +149,8 @@ Some new subcommands will also appear, which are related to building the k6 bina
   Flags:
     -o, --out name  output extension name
     --bin-dir path  folder for custom k6 binary (default: .)
-    --builder list  comma separated list of builders (default: native,docker)
+    --filter expr   jmespath syntax extension registry filter (default: [*])
+    --builder list  comma separated list of builders (default: service,native,docker)
     -h, --help      display this help
   ```
 
@@ -138,9 +166,36 @@ Some new subcommands will also appear, which are related to building the k6 bina
     -h, --help      display this help  
   ```
 
+- `service` start the [builder service](#builder-service)
+  ```
+  Usage:
+    k6x service [flags]
+
+  Flags:
+    --addr address  listen address (default: 127.0.0.1:8787)
+    --filter expr   jmespath syntax extension registry filter (default: [*])
+    --builder list  comma separated list of builders
+
+    -h, --help      display this help
+  ```
+
+- `preload` preload the build cache with popular extensions
+  ```
+  Usage:
+    k6x preload [flags]
+
+  Flags:
+    --platform list    comma separated list of platforms (default: linux/amd64,linux/arm64,windows/amd64,windows/arm64,darwin/amd64,darwin/arm64)
+    --stars number     minimum number of repository stargazers (default: 5)
+    --with dependency  dependency and version constraints (default: latest version of k6 and registered extensions)
+    --filter expr      jmespath syntax extension registry filter (default: [*])
+    --builder list     comma separated list of builders (default: service,local,docker)
+    -h, --help         display this help
+  ```
+
 ### Help
 
-The new subcommands (`build`, `deps`) display help in the usual way, with the `--help` or `-h` command line option.
+The new subcommands (`build`, `deps`, `service`, `preload`) display help in the usual way, with the `--help` or `-h` command line option.
 
 The k6 subcommands (`version`, `run` etc) also display help with the `--help` or `-h` command line option, so in this case the new k6x launcher flags are displayed before the normal k6 help.
 
@@ -160,6 +215,24 @@ The build step is done using the go compiler included in the image. The partial
 
 The k6x docker builder (`--builder docker`) also uses this docker image. It creates a local volume called `k6x-cache` and mounts it to the `/cache` path. Thanks to this, the docker build runs almost at the same speed as the native build (apart from the first build).
 
+### Filtering
+
+In certain runtime environments, the use of arbitrary extensions is not allowed. There is a need to limit the extensions that can be used.
+
+This use case can be solved most flexibly by narrowing down the extension registry. The content of the [extension registry](https://github.com/grafana/k6-docs/blob/main/src/data/doc-extensions/extensions.json) can be narrowed using a [jmespath](https://jmespath.org/) syntax filter expression. Extensions can be filtered based on any property. 
+
+*allow only officially supported extensions*
+
+```
+k6x --filter "[?contains(tiers,'Official')]" run script.js
+```
+
+*allow only cloud enabled extensions*
+
+```
+k6x --filter "[?cloudEnabled == true]" run script.js
+```
+
 ## Appendix
 
 ### How It Works
@@ -258,3 +331,76 @@ major change is API breaking. For example,
 * `^0.0` is equivalent to `>=0.0.0 <0.1.0`
 * `^0` is equivalent to `>=0.0.0 <1.0.0`
 
+### Builder Service
+
+The k6x builder service is an HTTP service that generates a k6 binary with the extensions specified in the request. The service is included in the k6x binary, so it can be started using the `k6x service` command.
+
+The k6x builder service can be used independently, from the command line (e.g. using `curl` or `wget` commands), from a `web browser`, or from different subcommands of the k6x launcher as a builder called `service`.
+
+#### Usage from the command line
+
+The k6 binary can be easily built using wget , curl or other command-line http client by retrieving the appropriate builder service URL:
+
+*using wget*
+
+```
+wget --content-disposition https://example.com/linux/amd64/k6@v0.46.0,dashboard@v0.6.0,k6/x/faker@v0.2.2,top@v0.1.1
+```
+
+*using curl*
+
+```
+curl -OJ https://example.com/linux/amd64/k6@v0.46.0,dashboard@v0.6.0,k6/x/faker@v0.2.2,top@v0.1.1
+```
+
+#### Usage from k6x
+
+The builder service can be used from k6x using the `--builder service` flag:
+
+```
+k6x run --builder service script.js
+```
+
+k6x expects the address of the builder service in the environment variable called `K6X_BUILDER_SERVICE`. There is currently no default, it must be specified.
+
+#### Simplified command line usage
+
+In order to simplify use from the command line, the service also accepts version dependencies in any order. In this case, after unlocking the latest versions and sorting, the response will be an HTTP redirect.
+
+*using wget*
+
+```
+wget --content-disposition https://example.com/linux/amd64/top,k6/x/faker,dashboard
+```
+
+*using curl*
+
+```
+curl -OJL https://example.com/linux/amd64/top,k6/x/faker,dashboard
+```
+
+#### How Build Service Works
+
+The service serves `HTTP GET` requests, with a well-defined path structure:
+
+```
+htps://example.com/goos/goarch/dependency-list
+```
+
+Where `goos` is the usual operating system name in the go language (e.g. `linux`, `windows`, `darwin`), `goarch` is the usual processor architecture in the go language (e.g. `amd64`, `arm64`). The `dependency-list` is a comma-separated list of dependencies, in the following form:
+
+```
+name@version
+```
+
+Where `name` is the name of the dependency and `version` is the version number according to [semver](https://semver.org/) (with an optional leading `v` character). The first item in the list is always the dependency named `k6`, and the other items are sorted alphabetically by name. For example:
+
+```
+https://example.com/linux/amd64/k6@v0.46.0,dashboard@v0.6.0,k6/x/faker@v0.2.2,top@v0.1.1
+```
+
+Based on the platform parameters (`goos`, `goarch`) and dependencies, the service prepares the k6 binary.
+
+Since the response (the k6 binary) depends only on the request path, it can be easily cached. The service therefore sets a sufficiently long caching period (at least three month) in the response, as well as the usual cache headers (e.g. `ETag`). By placing a caching proxy in front of the service, it can be ensured that the actual k6 binary build takes place only once for each parameter combination.
+
+The advantage of the solution is that the k6 binary is created on the fly, only for the parameter combinations that are actually used. Since the service preserves the go cache between builds, a specific build happens quickly enough.
diff --git a/go.mod b/go.mod
@@ -9,17 +9,20 @@ require (
 	github.com/docker/cli v24.0.6+incompatible
 	github.com/docker/docker v24.0.6+incompatible
 	github.com/evanw/esbuild v0.19.2
-	github.com/fatih/color v1.9.0
+	github.com/fatih/color v1.15.0
 	github.com/google/go-github/v55 v55.0.0
 	github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79
+	github.com/jmespath/go-jmespath v0.4.0
 	github.com/magefile/mage v1.15.0
-	github.com/mattn/go-colorable v0.1.12
+	github.com/mattn/go-colorable v0.1.13
+	github.com/mattn/go-isatty v0.0.18
 	github.com/princjef/mageutil v1.0.0
 	github.com/sirupsen/logrus v1.9.0
 	github.com/spf13/afero v1.9.5
 	github.com/spf13/pflag v1.0.5
-	github.com/stretchr/testify v1.7.0
+	github.com/stretchr/testify v1.8.2
 	go.k6.io/xk6 v0.9.2
+	golang.org/x/net v0.10.0
 	golang.org/x/sys v0.11.0
 )
 
@@ -36,21 +39,22 @@ require (
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/google/btree v1.0.0 // indirect
 	github.com/google/go-querystring v1.1.0 // indirect
-	github.com/mattn/go-isatty v0.0.14 // indirect
-	github.com/mattn/go-runewidth v0.0.7 // indirect
+	github.com/mattn/go-runewidth v0.0.14 // indirect
 	github.com/moby/term v0.5.0 // indirect
 	github.com/morikuni/aec v1.0.0 // indirect
 	github.com/opencontainers/go-digest v1.0.0 // indirect
 	github.com/opencontainers/image-spec v1.0.2 // indirect
 	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/rivo/uniseg v0.4.3 // indirect
 	golang.org/x/crypto v0.12.0 // indirect
 	golang.org/x/mod v0.8.0 // indirect
-	golang.org/x/net v0.10.0 // indirect
+	golang.org/x/sync v0.2.0 // indirect
 	golang.org/x/term v0.11.0 // indirect
 	golang.org/x/text v0.12.0 // indirect
+	golang.org/x/time v0.3.0 // indirect
 	golang.org/x/tools v0.6.0 // indirect
-	gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
 	gotest.tools/v3 v3.5.1 // indirect
 )