With firebuild, you can build and deploy secure VMs directly from Dockerfiles
and Docker images in just few minutes.
The concept of firebuild
is to leverage as much of the existing Docker world as possible. There are thousands of Docker images out there. Docker images are awesome because they encapsulate the software we want to run in our workloads, they also encapsulate dependencies. Dockerfiles are what Docker images are built from. Dockeriles are the blueprints of the modern infrastructure. There are thousands of them for almost anything one can imagine and new ones are very easy to write.
With firebuild it is possible to:
- build root file systems directly from Dockerfiles
- tag and version root file systems
- run and manage microvms on a single host
- define run profiles
Build and start HashiCorp Consul 1.9.4 on Firecracker with three simple steps:
- build a base operating system image
- build Consul image
- start the application
sudo $GOPATH/bin/firebuild baseos \
--profile=standard \
--dockerfile $(pwd)/baseos/_/alpine/3.12/Dockerfile
sudo $GOPATH/bin/firebuild rootfs \
--profile=standard \
--dockerfile=git+https://github.com/hashicorp/docker-consul.git:/0.X/Dockerfile \
--cni-network-name=machine-builds \
--ssh-user=alpine \
--vmlinux-id=vmlinux-v5.8 \
--tag=combust-labs/consul:1.9.4
sudo $GOPATH/bin/firebuild run \
--profile=standard \
--name=consul1 \
--from=combust-labs/consul:1.9.4 \
--cni-network-name=machines \
--vmlinux-id=vmlinux-v5.8
Find the IP of the consul1
VM and query Consul:
VMIP=$(sudo $GOPATH/bin/firebuild inspect \
--profile=standard \
--vmm-id=consul1 | jq '.NetworkInterfaces[0].StaticConfiguration.IPConfiguration.IP' -r)
$ curl http://${VMIP}:8500/v1/status/leader
"127.0.0.1:8300"
mkdir -p $GOPATH/src/github.com/combust-labs/firebuild
cd $GOPATH/src/github.com/combust-labs/firebuild
go install
The binary will be placed in $GOPATH/bin/firebuild
.
# create required directories, these need to exist before the profile can be created:
sudo mkdir -p /firecracker/rootfs
sudo mkdir -p /firecracker/vmlinux
sudo mkdir -p /srv/jailer
sudo mkdir -p /var/lib/firebuild
# create a profile:
sudo $GOPATH/bin/firebuild profile-create \
--profile=standard \
--binary-firecracker=$(readlink /usr/bin/firecracker) \
--binary-jailer=$(readlink /usr/bin/jailer) \
--chroot-base=/srv/jailer \
--run-cache=/var/lib/firebuild \
--storage-provider=directory \
--storage-provider-property-string="rootfs-storage-root=/firecracker/rootfs" \
--storage-provider-property-string="kernel-storage-root=/firecracker/vmlinux" \
--tracing-enable
Kernel images will be stored in /firecracker/vmlinux
, root file systems will be stored in /firecracker/rootfs
.
The examples use the 5.8 Linux kernel image which is built using the configuration from the baseos/kernel/5.8.config
file in this repository. To build the kernel:
export KERNEL_VERSION=v5.8
mkdir -p /tmp/linux && cd /tmp/linux
git clone https://github.com/torvalds/linux.git .
git checkout ${KERNEL_VERSION}
wget -O .config https://github.com/raw/combust-labs/firebuild/master/baseos/kernel/5.8.config
make vmlinux -j32 # adapt to the number of cores you have
Once built, copy the kernel to the storage:
mv /tmp/linux/vmlinux /firecracker/vmlinux/vmlinux-${KERNEL_VERSION}
firebuild
assumes CNI availability. Installing the plugins is very straightforward. Create /opt/cni/bin/
directory and download the plugins:
mkdir -p /opt/cni/bin
curl -O -L https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-amd64-v0.9.1.tgz
tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v0.9.1.tgz
Firecracker also requires the tc-redirect-tap
plugin. Unfortunately, this one does not offer downloadable binaries and has to be built from sources.
mkdir -p $GOPATH/src/github.com/awslabs/tc-redirect-tap
cd $GOPATH/src/github.com/awslabs/tc-redirect-tap
git clone https://github.com/awslabs/tc-redirect-tap.git .
make install
Feel free to change the ipam.subnet
or set multiple ones. host-local
IPAM CNI plugin documentation.
cat <<EOF > /etc/cni/conf.d/machine-builds.conflist
{
"name": "machine-builds",
"cniVersion": "0.4.0",
"plugins": [
{
"type": "bridge",
"name": "builds-bridge",
"bridge": "builds0",
"isDefaultGateway": true,
"ipMasq": true,
"hairpinMode": true,
"ipam": {
"type": "host-local",
"subnet": "192.168.128.0/24",
"resolvConf": "/etc/resolv.conf"
}
},
{
"type": "firewall"
},
{
"type": "tc-redirect-tap"
}
]
}
EOF
The maximum socket path in the Linux Kernel is 107
characters + \0
:
struct sockaddr_un {
__kernel_sa_family_t sun_family; /* AF_UNIX */
char sun_path[UNIX_PATH_MAX]; /* pathname */
};
The --chroot-base
value must have a maximum length of 31
characters. The constant jailer path suffix used by firebuild
is 76
characters:
- constant
/firecracker-v0.22.4-x86_64/
(automatically generated by the jailer) - VM ID is always
20
characters long - constant
/root/run/firecracker.socket
assumed by the jailer
Example: /firecracker-v0.22.4-x86_64/sifuqm4rq2runxparjcx/root/run/firecracker.socket
.
Using more than 31
characters for the --chroot-base
value, regardless if in the profile setting or using the command --chroot-base
flag, will lead to a very obscure error. Firecracker will report an error similar to:
INFO[0006] Called startVMM(), setting up a VMM on /mnt/sdd1/firebuild/jailer/firecracker-v0.22.4-x86_64/6b41ecc3783c4f38a743c9c8af4bbe0f/root/run/firecracker.socket
WARN[0009] Failed handler "fcinit.StartVMM": Firecracker did not create API socket /mnt/sdd1/firebuild/jailer/firecracker-v0.22.4-x86_64/6b41ecc3783c4f38a743c9c8af4bbe0f/root/run/firecracker.socket: context deadline exceeded
{"@level":"error","@message":"Firecracker VMM did not start, build failed","@module":"rootfs","@timestamp":"2021-03-14T19:20:49.856228Z","reason":"Failed to start machine: Firecracker did not create API socket /mnt/sdd1/firebuild/jailer/firecracker-v0.22.4-x86_64/6b41ecc3783c4f38a743c9c8af4bbe0f/root/run/firecracker.socket: context deadline exceeded","veth-name":"vethHvfZiskhLkQ","vmm-id":"6b41ecc3783c4f38a743c9c8af4bbe0f"}
{"@level":"info","@message":"cleaning up jail directory","@module":"rootfs","@timestamp":"2021-03-14T19:20:49.856407Z","veth-name":"vethHvfZiskhLkQ","vmm-id":"6b41ecc3783c4f38a743c9c8af4bbe0f"}
{"@level":"info","@message":"cleaning up temp build directory","@module":"rootfs","@timestamp":"2021-03-14T19:20:49.856458Z"}
WARN[0010] firecracker exited: signal: killed
In the above example, the path is 114
characters long. Changing the chroot to /mnt/sdd1/fc/jail
would solve the problem.
firebuild
uses the Docker metaphor. An image of an application is built FROM
a base. An application image can be built FROM alpine:3.13
, for example. Or FROM debian:buster-slim
, or FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3
and dozens others.
In order to fulfill those semantics, a base operating system image must be built before the application root file system can be created.
Build a base Debian Buster slim:
sudo $GOPATH/bin/firebuild baseos \
--profile=standard \
--dockerfile $(pwd)/baseos/_/debian/buster-slim/Dockerfile
Because the baseos
root file system is built completely with Docker, there is no need to configure the kernel storage.
This does not belong here, structure better: It's possible to tag the baseos
output using the --tag=
argument, for example:
sudo $GOPATH/bin/firebuild baseos \
--profile=standard \
--dockerfile $(pwd)/baseos/_/debian/buster-slim/Dockerfile \
--tag=custom/os:latest
The upstream Dockerfile
is built FROM debian:buster-slim
, that's the baseos
built in the previous step:
sudo $GOPATH/bin/firebuild rootfs \
--profile=standard \
--dockerfile=git+https://github.com/docker-library/postgres.git:/13/Dockerfile \
--cni-network-name=machine-builds \
--vmlinux-id=vmlinux-v5.8 \
--mem=512 \
--tag=combust-labs/postgres:13
For example:
cat <<EOF > /etc/cni/conf.d/machines.conflist
{
"name": "machines",
"cniVersion": "0.4.0",
"plugins": [
{
"type": "bridge",
"name": "machines-bridge",
"bridge": "machines0",
"isDefaultGateway": true,
"ipMasq": true,
"hairpinMode": true,
"ipam": {
"type": "host-local",
"subnet": "192.168.127.0/24",
"resolvConf": "/etc/resolv.conf"
}
},
{
"type": "firewall"
},
{
"type": "tc-redirect-tap"
}
]
}
EOF
Once the root file system is built, start the VM:
sudo $GOPATH/bin/firebuild run \
--profile=standard \
--name=postgres1 \
--from=combust-labs/postgres:13 \
--cni-network-name=machines \
--vmlinux-id=vmlinux-v5.8 \
--mem=512 \
--env="POSTGRES_PASSWORD=some-password"
To avoid passing the password on the command line, you can use --env-file
flag instead. The database is running, to verify:
Fine the IP address of the Postgres VM:
VMIP=$(sudo $GOPATH/bin/firebuild inspect \
--profile=standard \
--vmm-id=postgres1 | jq '.NetworkInterfaces[0].StaticConfiguration.IPConfiguration.IP' -r)
$ nc -zv ${VMIP} 5432
Connection to 192.168.127.94 5432 port [tcp/postgresql] succeeded!
If SSH access to the VM is required, this command can be used instead:
sudo $GOPATH/bin/firebuild run \
--profile=standard \
--name=postgres2 \
--from=combust-labs/postgres:13 \
--cni-network-name=machines \
--vmlinux-id=vmlinux-v5.8 \
--mem=512 \
--env="POSTGRES_PASSWORD=some-password" \
--ssh-user=debian \
--identity-file=path/to/the/identity.pub
--daemonize
: when specified, runs the VM in a daemonized mode--env-file
: full path to the environment file, multiple OK--env
: environment variable to deploy to configure the VM with, multiple OK, format--env=VAR_NAME=value
--hostname
: hostname to apply to the VM which the VM uses to resolve itself--name
: name of the virtual machine, if empty, random string will be used, maxmimum 20 characters, onlya-zA-Z0-9
ranges are allowed--ssh-user
: username to get access to the VM via SSH with, these are defined in thebaseos
Dockerfiles and follow the EC2 pattern:alpine
for Alpine images anddebian
for Debian image; together with--identity-file
allows access to the running VM via SSH--identity-file
: full path to the publish SSH key to deploy to the running VM
The final environment variables are written to /etc/profile.d/run-env.sh
file. All files specified with --env-file
are merged first in the order of occurrcence, variables specified with --env
are merged last.
Sometimes having just the Dockerfile
is not sufficient to execute a rootfs
build. A good example is this Jaeger all-in-one Dockerfile
. The Dockerfile
depends on the binary artifact built via Makefile
prior to Docker build. In this case, it's possible to build the VM rootfs directly from the Docker image:
sudo $GOPATH/bin/firebuild rootfs \
--profile=standard \
--docker-image=jaegertracing/all-in-one:1.22 \
--docker-image-base=alpine:3.13 \
--cni-network-name=machine-builds \
--vmlinux-id=vmlinux-v5.8 \
--mem=512 \
--tag=combust-labs/jaeger-all-in-one:1.22
The --docker-image-base
is required because the underlying operating system the image was built from cannot be established from the Docker manifest.
To access the Jaeger Query UI via the host:
sudo iptables -t filter -A FORWARD \
-m comment --comment "jaeger:1.22" \
-p tcp -d 192.168.127.100 --dport 16686 \
-m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
sudo iptables -t nat -A PREROUTING \
-m comment --comment "jaeger:1.22" \
-p tcp -i eno1 --dport 16686 \
-j DNAT \
--to-destination 192.168.127.100:16686
Where the exact IP address can be obtained using the firebuild inspect --profile=... --vmm-id=...
command and the destination IP and interface depend on your configuration, you can use ip link
to find the up broadcast
interfaces and relevant IP address. Tool intergration will be added at a later stage.
The builder pulls the requested Docker image with Docker. It then open the Docker image via the Docker save
command and looks up the manifest.json
and the Docker image config json
explicitly stated in the manifest. When config is fetched, a temporary Dockerfile is built from the Docker config history. Any ADD
and COPY
commands for resources other than first /
are used to extract files from the saved source image. When resources are exported, the build further continues exactly the same way as in case of the Dockerfile
build.
A VM started with the --daemonize
flag can be stopped in three ways:
- by executing the
kill
tool command, this is a clean stop which will take care of all the necessry clean up - by executing
reboot
from inside of the VM SSH connection; unclean stop, manual purge of the CNI cache, jailer directory, run cache and the veth link is needed - by executing the cURL HTTP against the VM socket file; unclean stop, manual purge of the CNI cache, jailer directory, run cache and the veth link is needed
To get the VM ID, look closely at the output of the run ... --detached
command:
{
"@level":"info",
"@message":"VMM running as a daemon",
"@module":"run",
"@timestamp":"2021-03-09T19:55:41.684488Z",
"cache-dir":"/var/lib/firebuild/831b7068f7924584b384260e8d262834",
"ip-address":"192.168.127.3",
"ip-net":"192.168.127.3/24",
"jailer-dir":"/srv/jailer/firecracker-v0.22.4-x86_64/831b7068f7924584b384260e8d262834",
"pid":17904,
"veth-name":"vethydMSApKfoDu",
"vmm-id":"831b7068f7924584b384260e8d262834"
}
Copy the VM ID from the output and run:
sudo $GOPATH/bin/firebuild kill --profile=standard --vmm-id=${VMMID}
If a VM exits in any other way than via kill
command, following data continues residing on the host:
- jail directory with all contents
- run cache directory with all contents
- CNI interface with CNI cache directory
To remove this data, run the purge
command.
sudo $GOPATH/bin/firebuild purge --profile=standard
sudo $GOPATH/bin/firebuild ls --profile=standard
Example output:
2021-03-12T01:46:21.752Z [INFO] ls: vmm: id=df45b6e14538456286e4a4bc1f9bf6e2 running=true pid=20658 image=tests/postgres:13 started="2021-03-12 01:46:11 +0000 UTC" ip-address=192.168.127.9
It's possible to reference a Dockerfile
residing in the git repository available under a HTTP(s) URL. Here's an example:
sudo $GOPATH/bin/firebuild rootfs \
--profile=standard \
--dockerfile=git+https://github.com/hashicorp/docker-consul.git:/0.X/Dockerfile#master \
--cni-network-name=machine-builds \
--vmlinux-id=vmlinux-v5.8 \
--tag=combust-labs/consul:1.9.4
The URL format is:
git+http(s)://host:port/path/to/repo.git:/path/to/Dockerfile[#<commit-hash | branch-name | tag-name>]
And will be processed as:
- path
/path/to/repo.git:/path/to/Dockerfile
will be split by:
and must contain both sides/path/to/repo.git
is the git repository path/path/to/Dockerfile
is the path to theDockerfile
in the repository, must point to a file after clone and checkout
- optional
#fragment
may be a comit hash, a branch name or a tag name- if no
#fragment
is given, the program will use the default cloned branch, check the remote to find out what is it
- if no
- the cloned repository will have a single remote and the first remote wil be used
http://
andhttps://
for direct paths to theDockerfile
, these can handle single file only and do not attempt loading any resources handled byADD
/COPY
commands, the server must be capable of responding toHEAD
andGET
http requests, more details inCaveats when building from the URL
further in this document- special
git+http://
andgit+https://
, documented above - standard
ssh://
,git://
andgit+ssh://
URL formats with the expectation that the path meets the criteria from thegit+http(s):// URL
section above
The build
command will resolve the resources referenced in ADD
and COPY
commands even when loading the Dockerfile
via the URL. The context root in this case will be established by removing the file name from the URL. An example:
- consider the URL
https://github.com/raw/hashicorp/docker-consul/master/0.X/Dockerfile
- the
Dockerfile
name will be removed from the URL and the context ishttps://github.com/raw/hashicorp/docker-consul/master/0.X
- assuming that the
Dockerfile
containsADD ./docker-entrypoint.sh ...
, the resolver will try loadinghttps://github.com/raw/hashicorp/docker-consul/master/0.X/docker-entrypoint.sh
There are following limitations when loading the resources like that via URL:
- if the
ADD
orCOPY
points to a directory, the command will fail because there is no unified way of loading directories via HTTP, the resolver will not even attempt this, it will most likely fail on theHTTP GET
request - the file permissions will not be carried over because there is no method to infer file mode from a HTTP response
The build program does not support:
ONBUILD
commandsHEALTHCHECK
commandsSTOPSIGNAL
commands
firebuild
supports multi-stage Dockerfile
builds. An example with grepplabs Kafka Proxy.
Build v0.2.8
using git repository link:
sudo $GOPATH/bin/firebuild rootfs \
--profile=standard \
--dockerfile=git+https://github.com/grepplabs/kafka-proxy.git:/Dockerfile#v0.2.8 \
--cni-network-name=machine-builds \
--vmlinux-id=vmlinux-v5.8 \
--tag=combust-labs/kafka-proxy:0.2.8
TODO: eat your own dog food, start with firebuild.
Start Jaeger, for example:
docker run --rm -ti \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 14250:14250 \
-p 9411:9411 \
jaegertracing/all-in-one:1.22
And configure respective commands with:
... --tracing-enable \
--tracing-collector-host-port=... \
The default value of the --tracing-collector-host-port
is 127.0.0.1:6831
. To enable tracer log output, set --tracing-log-enable
flag.
Unless explcitly stated: AGPL-3.0 License.