Kayobe Configuration for "A Universe from Nothing: Containerised OpenStack deployment using Kolla, Ansible and Kayobe"
This repository may be used as a workshop to configure, deploy and get hands-on with OpenStack Kayobe.
It provides a configuration and walkthrough for the Kayobe project based on the configuration provided by the kayobe-config repository. It deploys a containerised OpenStack environment using Kolla, Ansible and Kayobe.
Select the Git branch of this repository for the OpenStack release you are interested in, and follow the README.
For this workshop, we require the use of a single server, configured as a seed hypervisor. This server should be a bare metal node or VM running Rocky 9, with the following minimum requirements:
- 32GB RAM
- 80GB disk
We will also need SSH access to the seed hypervisor, and passwordless sudo configured for the login user.
On the seed hypervisor, we will deploy three VMs:
- 1 seed
- 1 controller
- 1 compute node
The seed runs a standalone Ironic service. The controller and compute node are 'virtual bare metal' hosts, and we will use the seed to provision them with an OS. Next we'll deploy OpenStack services on the controller and compute node.
At the end you'll have a miniature OpenStack cluster that you can use to test out booting an instance using Nova, access the Horizon dashboard, etc.
There are four parts to this guide:
Preparation has instructions to prepare the seed hypervisor for the exercise, and fetching the necessary source code.
Deploying a Seed includes all instructions necessary to download and install the Kayobe prerequisites on a plain Rocky 9 cloud image, including provisioning and configuration of a seed VM. Optionally, snapshot the instance after this step to reduce setup time in the future.
A Universe from a Seed contains all instructions necessary to deploy from a host running a seed VM. An image suitable for this can be created via Optional: Creating a Seed Snapshot.
Once the control plane has been deployed see Next Steps for some ideas for what to try next.
This shows how to prepare the seed hypervisor for the exercise. It assumes
you have created a seed hypervisor instance fitting the requirements
above and have already logged in (e.g. ssh rocky@<ip>
).
# Install git and tmux.
if $(which dnf 2>/dev/null >/dev/null); then
sudo dnf -y install git tmux
else
sudo apt update
sudo apt -y install git tmux
fi
# Disable the firewall.
sudo systemctl is-enabled firewalld && sudo systemctl stop firewalld && sudo systemctl disable firewalld
# Disable SELinux both immediately and permanently.
if $(which setenforce 2>/dev/null >/dev/null); then
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
fi
# Prevent sudo from making DNS queries.
echo 'Defaults !fqdn' | sudo tee /etc/sudoers.d/no-fqdn
# Optional: start a new tmux session in case we lose our connection.
tmux
# Start at home.
cd
# Clone Kayobe.
git clone https://opendev.org/openstack/kayobe.git -b stable/2023.1
cd kayobe
# Clone the Tenks repository.
git clone https://opendev.org/openstack/tenks.git
# Clone this Kayobe configuration.
mkdir -p config/src
cd config/src/
git clone https://github.com/stackhpc/a-universe-from-nothing.git kayobe-config -b stable/2023.1
# Configure host networking (bridge, routes & firewall)
./kayobe-config/configure-local-networking.sh
# Install kayobe.
cd ~/kayobe
./dev/install-dev.sh
This shows how to create an image suitable for deploying Kayobe.
It assumes you have created a seed hypervisor instance fitting the requirements
above and have already logged in (e.g. ssh rocky@<ip>
), and performed the
necessary Preparation.
cd ~/kayobe
# Activate the Kayobe environment, to allow running commands directly.
source ~/kayobe-venv/bin/activate
source config/src/kayobe-config/kayobe-env
# Bootstrap the Ansible control host.
kayobe control host bootstrap
# Configure the seed hypervisor host.
kayobe seed hypervisor host configure
# Provision the seed VM.
kayobe seed vm provision
# Configure the seed host, and deploy a local registry.
kayobe seed host configure
# Pull, retag images, then push to our local registry.
./config/src/kayobe-config/pull-retag-push-images.sh
# Deploy the seed services.
kayobe seed service deploy
# Deploying the seed restarts networking interface,
# run configure-local-networking.sh again to re-add routes.
./config/src/kayobe-config/configure-local-networking.sh
# Optional: Shutdown the seed VM if creating a seed snapshot.
sudo virsh shutdown seed
If required, add any additional SSH public keys to /home/rocky/.ssh/authorized_keys
If necessary, take a snapshot of the hypervisor instance at this point to speed up this process in the future.
You are now ready to deploy a control plane using this host or snapshot.
This shows how to deploy a control plane from a VM image that contains a pre-deployed seed VM, or a host that has run through the steps in Deploying a Seed.
Having a snapshot image saves us some time if we need to repeat the deployment. If working from a snapshot, create a new instance with the same dimensions as the Seed image and log into it. Otherwise, continue working with the instance from Deploying a Seed.
# Optional: start a new tmux session in case we lose our connection.
tmux
# Set working directory
cd ~/kayobe
# Configure non-persistent networking, if the node has rebooted.
./config/src/kayobe-config/configure-local-networking.sh
Make sure that the seed VM (running Bifrost and supporting services) is present and running.
# Check if the seed VM is present and running.
sudo virsh list --all
# Start up the seed VM if it is shut off.
sudo virsh start seed
We use the TENKS project to model some 'bare metal' VMs for the controller and compute node. Here we set up our model development environment, alongside the seed VM.
# NOTE: Make sure to use ./tenks, since just ‘tenks’ will install via PyPI.
export TENKS_CONFIG_PATH=config/src/kayobe-config/tenks.yml
./dev/tenks-deploy-overcloud.sh ./tenks
# Activate the Kayobe environment, to allow running commands directly.
source dev/environment-setup.sh
# Inspect and provision the overcloud hardware:
kayobe overcloud inventory discover
kayobe overcloud hardware inspect
kayobe overcloud introspection data save
kayobe overcloud provision
Configure and deploy OpenStack to the control plane (following Kayobe host configuration documentation):
kayobe overcloud host configure
kayobe overcloud container image pull
kayobe overcloud service deploy
source config/src/kayobe-config/etc/kolla/public-openrc.sh
kayobe overcloud post configure
At this point it should be possible to access the Horizon GUI via the
server's public IP address, using port 80 (achieved through port
forwarding to the controller VM). Use the admin credentials from
OS_USERNAME
and OS_PASSWORD
to get in.
The following script will register some resources (keys, flavors, networks, images, etc) in OpenStack to enable booting up a tenant VM:
source config/src/kayobe-config/etc/kolla/public-openrc.sh
./config/src/kayobe-config/init-runonce.sh
Following the instructions displayed by the above script, boot a VM. You'll need to have activated the ~/os-venv virtual environment.
source ~/os-venv/bin/activate
openstack server create --image cirros \
--flavor m1.tiny \
--key-name mykey \
--network demo-net demo1
# Assign a floating IP to the server to make it accessible.
openstack floating ip create public1
fip=$(openstack floating ip list -f value -c 'Floating IP Address' --status DOWN | head -n 1)
openstack server add floating ip demo1 $fip
# Check SSH access to the VM.
ssh cirros@$fip
# If the ssh command above fails you may need to reconfigure the local
networking setup again:
~/kayobe/config/src/kayobe-config/configure-local-networking.sh
Note: when accessing the VNC console of an instance via Horizon,
you will be sent to the internal IP address of the controller,
192.168.33.2
, which will fail. Open the console-only display link
in new broser tab and replace this IP in the address bar with
the public IP of the hypervisor host.
That's it, you're done!
Here's some ideas for things to explore with the deployment:
- Access Control Plane Components: take a deep dive into the internals by Exploring the Deployment.
- Deploy OpenSearch and OpenSearch Dashboards: see Enabling Centralised Logging to get logs aggregated from across our OpenStack control plane.
Once each of the VMs becomes available, they should be accessible
via SSH as the rocky
or stack
user at the following IP addresses:
Host | IP |
---|---|
seed | 192.168.33.5 |
controller0 | 192.168.33.3 |
compute0 | 192.168.33.6 |
The control plane services are run in Docker containers, so try using the docker CLI to inspect the system.
# List containers
docker ps
# List images
docker images
# List volumes
docker volume ls
# Inspect a container
docker inspect <container name>
# Execute a process in a container
docker exec -it <container> <command>
The kolla container configuration is generated under /etc/kolla
on
the seed and overcloud hosts - each container has its own directory
that is bind mounted into the container.
Log files are stored in the kolla_logs
docker volume, which is
mounted at /var/log/kolla
in each container. They can be accessed
on the host at /var/lib/docker/volumes/kolla_logs/_data/
.
Verify that Tenks has created controller0
and compute0
VMs:
sudo virsh list --all
Verify that virtualbmc is running:
/usr/local/bin/vbmc list
+-------------+---------+--------------+------+
| Domain name | Status | Address | Port |
+-------------+---------+--------------+------+
| compute0 | running | 192.168.33.4 | 6231 |
| controller0 | running | 192.168.33.4 | 6230 |
+-------------+---------+--------------+------+
VirtualBMC config is here (on the VM hypervisor host):
/root/.vbmc/controller0/config
Note that the controller and compute node are registered in Ironic, in the bifrost container. Once kayobe is deployed and configured the compute0 and controller0 will be controlled by bifrost and not virsh commands.
ssh stack@192.168.33.5
docker exec -it bifrost_deploy bash
export OS_CLOUD=bifrost
baremetal node list
+--------------------------------------+-------------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+-------------+---------------+-------------+--------------------+-------------+
| d7184461-ac4b-4b9e-b9ed-329978fc0648 | compute0 | None | power on | active | False |
| 1a40de56-be8a-49e2-a903-b408f432ef23 | controller0 | None | power on | active | False |
+--------------------------------------+-------------+---------------+-------------+--------------------+-------------+
exit
In Kolla-Ansible, centralised logging is easily enabled and results in the deployment of OpenSearch services and configuration to forward all OpenStack service logging. Be cautious as OpenSearch will consume a significant portion of available resources on a standard deployment.
To enable the service, one flag must be changed in
~/kayobe/config/src/kayobe-config/etc/kayobe/kolla.yml
:
-#kolla_enable_central_logging:
+kolla_enable_central_logging: yes
This will deploy opensearch
and opensearch_dashboards
containers, and
configure logging via fluentd
so that logging from all deployed Docker
containers will be routed to OpenSearch.
Before this can be applied, it is necessary to download the missing images to the seed VM. Pull, retag and push the centralised logging images:
~/kayobe/config/src/kayobe-config/pull-retag-push-images.sh ^opensearch
To deploy the logging stack:
kayobe overcloud container image pull
kayobe overcloud service deploy
As simple as that...
The new containers can be seen running on the controller node:
$ ssh stack@192.168.33.3 docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fad79f29afbc 192.168.33.5:4000/openstack.kolla/opensearch-dashboards:2023.1-rocky-9 "dumb-init --single-…" 6 hours ago Up 6 hours (healthy) opensearch_dashboards
64df77adc709 192.168.33.5:4000/openstack.kolla/opensearch:2023.1-rocky-9 "dumb-init --single-…" 6 hours ago Up 6 hours (healthy) opensearch
...
We can see the log indexes in OpenSearch:
curl -X GET "192.168.33.3:9200/_cat/indices?v"
To access OpenSearch Dashboards, we must first forward connections from our
public interface to the OpenSearch Dashboards service running on our
controller0
VM.
The easiest way to do this is to add OpenSearch Dashboards's default port (5601) to our
configure-local-networking.sh
script in ~/kayobe/config/src/kayobe-config/
:
--- a/configure-local-networking.sh
+++ b/configure-local-networking.sh
@@ -20,7 +20,7 @@ seed_hv_private_ip=$(ip a show dev $iface | grep 'inet ' | awk '{ print $2 }' |
# Forward the following ports to the controller.
# 80: Horizon
# 6080: VNC console
-forwarded_ports="80 6080"
+forwarded_ports="80 6080 5601"
Then rerun the script to apply the change:
config/src/kayobe-config/configure-local-networking.sh
We can now connect to OpenSearch Dashboards using our hypervisor host public IP and port 5601.
The username is opensearch
and the password we can extract from the
Kolla-Ansible passwords (in production these would be vault-encrypted
but they are not here).
grep opensearch_dashboards config/src/kayobe-config/etc/kolla/passwords.yml
Once you're in, OpenSearch Dashboards needs some further setup which is not automated.
Set the log index to flog-*
and you should be ready to go.
Barbican is the OpenStack secret management service. It is an example of a simple service we can use to illustrate the process of adding new services to our deployment.
As with the Logging service above, enable Barbican by modifying the flag in
~/kayobe/config/src/kayobe-config/etc/kayobe/kolla.yml
as follows:
-#kolla_enable_barbican:
+kolla_enable_barbican: yes
This instructs Kolla to install the barbican api, worker & keystone-listener containers. Pull down barbican images:
~/kayobe/config/src/kayobe-config/pull-retag-push-images.sh barbican
To deploy the Barbican service:
# Activate the venv if not already active
cd ~/kayobe
source dev/environment-setup.sh
kayobe overcloud container image pull
kayobe overcloud service deploy
Once Barbican has been deployed it can be tested using the barbicanclient plugin to the OpenStack CLI. This should be installed and tested in the OpenStack venv:
# Deactivate existing venv context if necessary
deactivate
# Activate the OpenStack venv
. ~/os-venv/bin/activate
# Install barbicanclient
pip install python-barbicanclient -c https://releases.openstack.org/constraints/upper/2023.1
# Source the OpenStack environment variables
source ~/kayobe/config/src/kayobe-config/etc/kolla/public-openrc.sh
# Store a test secret
openstack secret store --name mysecret --payload foo=bar
# Copy the 'Secret href' URI for later use
SECRET_URL=$(openstack secret list --name mysecret -f value --column 'Secret href')
# Get secret metadata
openstack secret get ${SECRET_URL}
# Get secret payload
openstack secret get ${SECRET_URL} --payload
Congratulations, you have successfully installed Barbican on Kayobe.
- Kayobe documentation: https://docs.openstack.org/kayobe/latest/
- Source: https://github.com/stackhpc/a-universe-from-nothing
- Bugs: https://github.com/stackhpc/a-universe-from-nothing/issues
- IRC: #openstack-kolla