From 815407860882072e9278bee9045848582c9a1882 Mon Sep 17 00:00:00 2001 From: Markus Boehme Date: Thu, 14 Jul 2022 17:29:46 +0200 Subject: [PATCH] docs: Add quickstart guide for local QEMU/KVM VMs Commit e2b58d60 ("tools: Add start-local-vm script") added a wrapper script to simplify running a locally built Bottlerocket image in a local VM using QEMU and KVM. Provide a quickstart guide analogous to the existing ones for EKS, ECS, and VMware to help make use of it. The guide currently assumes anyone following along to be running Fedora. Changes to the guide for other distros will be minimal, if they are needed at all. However, changes in start-local-vm are likely to be required. Leave this to a dedicated later change. Signed-off-by: Markus Boehme --- BUILDING.md | 7 +- QUICKSTART-LOCAL.md | 169 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 175 insertions(+), 1 deletion(-) create mode 100644 QUICKSTART-LOCAL.md diff --git a/BUILDING.md b/BUILDING.md index dd08712edab..466984248a1 100644 --- a/BUILDING.md +++ b/BUILDING.md @@ -174,7 +174,12 @@ cargo make -e PUBLISH_REGIONS=your-region-here -e BUILDSYS_VARIANT=my-variant-he ## Use your image -See the [setup guide for Kubernetes](QUICKSTART-EKS.md) or the [setup guide for Amazon ECS](QUICKSTART-ECS.md) for information on running Bottlerocket images. +See any of the setup guides tailored to the various execution environments for information on running Bottlerocket images: + +* [Setup guide for Kubernetes](QUICKSTART-EKS.md) +* [Setup guide for Amazon ECS](QUICKSTART-ECS.md) +* [Setup guide for VMware](QUICKSTART-VMWARE.md) +* [Setup guide for QEMU/KVM](QUICKSTART-LOCAL.md) ## Publish your image diff --git a/QUICKSTART-LOCAL.md b/QUICKSTART-LOCAL.md new file mode 100644 index 00000000000..acbc19f3f55 --- /dev/null +++ b/QUICKSTART-LOCAL.md @@ -0,0 +1,169 @@ +# Testing Bottlerocket in a local virtual machine + +This quickstart will walk through launching a Bottlerocket VM guest on a local machine using QEMU and KVM. +The VM will not join an ECS or Kubernetes cluster. +This way of running Bottlerocket is therefore best used for testing purposes when developing Bottlerocket components that do not need to integrate with any orchestrators or to just get a feel for what a Bottlerocket node looks from the inside. + + +## Prerequisites + +We assume you are following along on a machine running Fedora. +If you are using a cloud VM, ensure you can use hardware-assisted virtualization. +For example, on Amazon EC2 this requires the use of a .metal instance type. + +You need a clone of the main Bottlerocket repository and a build of the metal-dev variant. +Please refer to [`BUILDING.md`](https://github.com/bottlerocket-os/bottlerocket/blob/develop/BUILDING.md) for instructions on how to build a Bottlerocket image and ensure you pass `-e BUILDSYS_VARIANT=metal-dev` to `cargo make`. + +The use of QEMU requires extra packages which you may install using this dnf invocation: + +``` +sudo dnf install qemu +``` + +If you'd (optionally) like to make use of the control container, you'll need an AWS account and AWS CLI. + + +## Configuring Bottlerocket + +Bottlerocket is configured [via an API](https://github.com/bottlerocket-os/bottlerocket/#using-the-api-client) or, if running in a cloud VM, [via user data](https://github.com/bottlerocket-os/bottlerocket/#using-user-data) upon boot. +For running a local VM, neither mechanism can be used to apply configuration on first boot: Bottlerocket is not yet running, making its API server unavailable, and the goal to have Bottlerocket running locally precludes use of the user data mechanism. +As an alternative, the `start-local-vm` wrapper script included in the `tools` directory of the main repository allows to inject configuration into well-known locations of the built image for Bottlerocket to find on boot. + + +### Set up networking + +The `start-local-vm` wrapper configures QEMU to provide one virtual network interface to the VM. +To enable this interface, create a file named `net.toml` containing the following TOML snippet: + +``` +version = 1 + +[enp0s16] +dhcp4 = true +``` + +This will prompt [netdog](https://github.com/bottlerocket-os/bottlerocket/blob/develop/sources/api/netdog/README.md) to set up `enp0s16` as the primary network interface with IPv4 networking configured via DHCP. +No dedicated DHCP server needs to be running on the host as QEMU will act as one on the virtual network interface. +Note that for virtual machines launched with `start-local-vm`, the primary network interface will always be named `enp0s16`. +The name will differ when running on bare metal or in a cloud environment. + + +### Accessing your Bottlerocket guest via host containers + +When running a Bottlerocket development variant such as metal-dev locally, you can directly interact with the system via the serial console that `start-local-vm` connects you to. +For remote access to your running Bottlerocket VMs, you will need to provide additional configuration to enable host containers. +The Bottlerocket metal images don't include any host containers enabled by default. +But don't worry! +You can use our [admin](https://github.com/bottlerocket-os/bottlerocket-admin-container) and/or [control](https://github.com/bottlerocket-os/bottlerocket-control-container) containers, they just need to be configured first. +Information about the roles these host containers play can be found [here](https://github.com/bottlerocket-os/bottlerocket/#exploration). + + +#### Admin container + +If you would like to use the admin container, you will need to create some base64 encoded user data which will be passed to the container at runtime. +Full details are covered in the [admin container documentation](https://github.com/bottlerocket-os/bottlerocket-admin-container#authenticating-with-the-admin-container). +If we assume you have a public key at `${HOME}/.ssh/id_rsa.pub`, the below will add the correct user data to your `user-data.toml`. + +``` +PUBKEY_FILE="${HOME}/.ssh/id_rsa.pub" +PUBKEY=$(< "${PUBKEY_FILE}") +ADMIN_USER_DATA="$(echo '{"ssh": {"authorized-keys": ["'"${PUBKEY}"'"]}}' | base64 -w 0)" + +cat <>user-data.toml +[settings.host-containers.admin] +enabled = true +user-data = "${ADMIN_USER_DATA}" +source = "public.ecr.aws/bottlerocket/bottlerocket-admin:v0.9.0" +EOF +``` + + +#### Control container + +Enabling the control container is very similar to the admin container; you will create some base64 encoded user data that will be passed to the container at runtime. +This user data includes an activation ID and code retrieved from AWS SSM. +Full details can be found in the [control container documentation](https://github.com/bottlerocket-os/bottlerocket-control-container#connecting-to-aws-systems-manager-ssm). + +You'll first need an AWS account, and AWS CLI installed. +Then you'll create a service role in that account to [grant AWS STS trust to the AWS Systems Manager service](https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-service-role.html). + +``` +cat < ssmservice-trust.json +{ + "Version": "2012-10-17", + "Statement": { + "Effect": "Allow", + "Principal": { + "Service": "ssm.amazonaws.com" + }, + "Action": "sts:AssumeRole" + } +} +EOF + +# Create the role using the above policy +aws iam create-role \ + --role-name SSMServiceRole \ + --assume-role-policy-document file://ssmservice-trust.json + +# Attach the policy enabling the role to create session tokens +aws iam attach-role-policy \ + --role-name SSMServiceRole \ + --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore +``` + +Once the above is created, we can use the role to create an activation code and ID. + +``` +export SSM_ACTIVATION="$(aws ssm create-activation \ + --iam-role SSMServiceRole \ + --registration-limit 100 \ + --region us-west-2 \ + --output json)" +``` + +Using the above activation data we can create our user data to pass to the control container + +``` +SSM_ACTIVATION_ID="$(jq -r '.ActivationId' <<< ${SSM_ACTIVATION})" +SSM_ACTIVATION_CODE="$(jq -r '.ActivationCode' <<< ${SSM_ACTIVATION})" +CONTROL_USER_DATA="$(echo '{"ssm": {"activation-id": "'${SSM_ACTIVATION_ID}'", "activation-code": "'${SSM_ACTIVATION_CODE}'", "region": "us-west-2"}}' | base64 -w0)" + +cat <>user-data.toml +[settings.host-containers.control] +enabled = true +user-data = "${CONTROL_USER_DATA}" +source = "public.ecr.aws/bottlerocket/bottlerocket-control:v0.6.1" +EOF +``` + + +## Launch! + +We have now prepared all configuration we might need. +Assuming you are in the root of the main Bottlerocket repository, you can run + +``` +./tools/start-local-vm --variant metal-dev --arch $(uname -m) --inject-file net.toml --inject-file user-data.toml +``` + +to start a local VM with the Bottlerocket image you built earlier. + +The `--inject-file` options add the listed files to the private partition of the image, where Bottlerocket's various services will find them on boot. +The final configuration files ending up in the image need to be named like in the examples above. +If you named yours differently, you can ensure they have the right name in the image by using a colon as the separator of local file name and file name in the image, e.g. `--inject-file admin-container-only.toml:user-data.toml`. +If you did not enable any host containers and thus have no `user-data.toml` you need to leave this option off. + +Once the VM launches, boot output will be visible in the terminal. +The `start-local-vm` script connects you to the serial console of the VM, which can also be used to interact with the system if you are running a development variant such as metal-dev. +When prompted to login, any username will do. + +The virtual serial console will capture most keyboard input, such as Ctrl-C. +If you want to terminate the VM, you can either instruct it to `systemctl poweroff` from within or exit QEMU via the Ctrl-A X shortcut. + +By default, the `start-local-vm` wrapper will forward the host's TCP port 2222 to the VM's port 22. +If you enabled the admin host container, the SSH server running in it will therefore be available by connecting to localhost's port 2222: + +``` +ssh -p 2222 ec2-user@localhost +```