Attention: The code in this repository is intended for experimental use only and is not fully tested, documented, or supported by SingleStore. Visit the SingleStore Forums to ask questions about this repository.
This repository provides utilities and documentation intended to help you streamline development of Wasm UDFs and TVFs for SingleStoreDB. It consists of development containers, a tutorial, and a collection of example Wasm programs.
This README describes the technical details on how to get started using this toolkit. Please also check out our tutorial, which is great place to start once you are ready to write your first function, and will walk you through specific examples.
To use the tools in this repo, you will need to have Docker installed on your system. Most of these tools can be installed locally as well.
In this repo, you will find two development containers -- one standalone and one designed specifically for VS Code. These containers attempt to strike a balance between providing a fairly comprehensive set of Wasm-oriented development tools, while still being mindful of image sizes. Still, the container sizes currently range between 2-3 GB.
The following tools are available within each container:
The Clang compiler and toolchain. The exact compiler version may differ between containers; see below for specifics.
The GCC compiler and toolchain. The exact compiler version may differ between containers; see below for specifics.
The Rust compiler with the stable toolchain. It has been configured with compilation targets for the default platform, wasm32-wasi, and wasm32-unknown-unknown. The exact compiler version may differ between containers; see below for specifics.
A popular Wasm compiler and runtime.
Utilities to support the WASI toolchain.
A language binding generator for the WIT IDL.
A utility to help test Wasm functions locally without the need to create a separate driver program. Please see its dedicated Git Repository for more information.
A utility that allows you to easily import your locally-built Wasm function into SingleStoreDB as a UDF or TVF. Please see its dedicated Git Repository for more information.
The VS Code container includes an experimental remote debugging tool that can be run as an external function from SingleStoreDB. For instructions on how to use this utility, please see this document.
Finally, both containers give you access to password-less sudo
so that you can further customize instances of them if you prefer.
This section suggests a possible workflow for developing a Wasm UDF or TVF for use in SingleStoreDB.
Before you start, you'll need to clone this repository. For the purposes of example, we'll assume you are cloning to $HOME/singlestore-wasm-toolkit
.
git clone https://github.com/singlestore-labs/singlestore-wasm-toolkit.git
One-Time Setup
-
Install VS Code.
-
In VS Code, press Ctrl-Shift-X. This will take you to the Extensions Marketplace. Type
Remote - Containers
in the search field and click the Install button. -
In VS Code, press F1 and type
Remote Containers: Install Docker
. -
By default, VS Code will mount just this repo in the container file system. This is only really useful if you want to work with the examples. If you'd like to work on other directories in this environment, then you will also need to do the following:
- Edit the
.devcontainer/devcontainer.json
file. - Search for the
mounts
section and uncomment it. - Change the value after
source=
to the local directory you want to mount. - Change the value after
target=
to the path where you want the directory mounted in the container. If you intend to develop in this directory, things will flow more smoothly if it is a child of the/workspaces
directory. For example:target=/workspaces/mycode
. - You can add a new entry for each additional path you want mounted.
- Edit the
Beginning Development
-
Start VS Code.
-
Press
F1
and search forRemote-Containers: Open Folder in Container
. -
Navigate to the
$HOME/singlestore-wasm-toolkit
(or the place where you cloned this repo). Click Open. -
If this is your first time opening the container, it will now be built. This can take quite a while, so please be patient -- it only needs to happen once.
-
If you intend to work on a project outside of this repo, then you will add this code to the current project. Here's how:
- First make sure you've completed Step 4 in the Preparation section, above.
- Press F1 and search for
Workspaces: Add Folder to Workspace
. - In the drop-down list, if you followed the convention suggested in Step 4 of Preparation, you should see the name of the folder you mounted into the
/workspaces
directory. If you chose a different location, then navigate to that folder instead. When you are ready, make sure the proper folder is selected and click the OK button. - The project will reload automatically.
- Your new folder can be found in the project navigator window. If you want to save this workspace configuration for later, then press F1, search for
Workspaces: Save Workspace As
, and pick a name.
-
Edit the source code as necessary.
-
When you are ready to compile the Wasm module, press Ctrl-` (that's the backtick character) to open a console window in the IDE. You should see a prompt prefixed with
dev-shell
. Be sure to change (cd
) to the directory of the code you intend to compile.
-
From this repo, run the script
scripts/dev-shell
. It takes a single argument with the location of the local source directory where you intend to work. For example:scripts/dev-shell /home/pete/mycoolproject
. -
If this is your first time running this container, it will be downloaded from the GitHub Docker registry. This may take a few minutes, depending on your internet connection.
-
You should now see a prompt prefixed with
dev-shell
. The directory you supplied in the argument will be mounted to~/src
in the container, and will be the current working directory when the prompt appears. -
Edit your source code locally, using whatever tools your prefer. You only need to use the dev-shell when you are ready to compile, test, or deploy.
For both container types, iterative development proceeds the same. We'll use the split
example included in this repo as a case study to describe a possible workflow.
-
To start:
- In the container, ensure that you are in the root directory of the code you intend to compile. For our demonstration, we will be in
~/src/examples/rust/split
. - Ensure that you have created a WIT file to describe the common interface of your Wasm function and UDF/TVF. The WIT syntax is described here. For the
split
example, we have placed this file at the root of the example's source tree.
- In the container, ensure that you are in the root directory of the code you intend to compile. For our demonstration, we will be in
-
Compile your program using the appropriate compiler. Clang/GCC and Rust are both available in the container. The process of compiling is described in more detail for C/C++ here, and for Rust here.
To compilesplit
for Rust in optimized mode, we would use this command:cargo wasi build --lib --release
-
Now, test your program using the
writ
tool. It should already be included in your$PATH
. Specific instructions on its usage are available in its repository. As an example, we might use it to test one input ofsplit
like this:% writ --wit split.wit target/wasm32-wasi/release/split.wasm split_str 'hello_you_fine_folks' '_' [ { "str": "hello", "idx": 0 }, { "str": "you", "idx": 6 }, { "str": "fine", "idx": 10 }, { "str": "folks", "idx": 15 } ]
-
When you are satisfied with your program, you can import it into SingleStoreDB using the
pushwasm
tool, also included in your default$PATH
. Usage instructions can be found in its source repository. For example, we could use it like this to create a Wasm TVF from oursplit
module:% pushwasm --tvf --prompt mysql://admin@svc-0e3c0e37-dml.singlestore.com:3306/testing --wit split.wit target/wasm32-wasi/release/split.wasm split_str Wasm TVF 'split_str' was created successfully.
-
The last step is verify that the Wasm UDF or TVF works in your database. Using the SQL Editor in the SingleStore Customer Portal, you can check that the function has been created using the
SHOW FUNCTIONS
statement. Then, you might try running using your UDF or TVF on a table. Here's an example using thesplit
TVF:SELECT * from example_table t, split_str(t.name, ' ');
To learn about the process of developing a Wasm UDF or TVF in more detail, please have a look at our tutorial.
Information about setting up the remote debugging tool can be found here.
For instructions on using the writ
tool, please go here.
For instructions on using the pushwasm
tool, please go here.
Information about the Rust examples can be found here.
The SingleStoreDB Wasm UDF/TVF documentation is here.