Tips for core sysadmins

This guide is only for sysadmins in the core devs OFN team. Here's a couple of things we can use to make server administration a lot smoother.

Ansible Vault password management

This allows automatic inclusion of the ansible-vault password in operations involving secrets, without having to paste it into the command line all the time from Bitwarden.

Setup

• Create a script in ~/ofn-vault-pass.sh and add the following lines:

#!/bin/bash

sudo cat /root/ofn-vault-pass

• Make sure this script is executable

sudo chmod u+x ~/ofn-vault-pass.sh

• Create a file owned by root, under /root/ofn-vault-pass and add the ansible-vault password used to encrypyt/decrypt secrets (see Bitwarden). The file contents should be a single line and contain only the password, in plain text.
• Add this line to your ~/bash.rc file: export ANSIBLE_VAULT_PASSWORD_FILE="$HOME/ofn-vault-pass.sh"
• Log out and back in again, or pick up the bashrc changes with: source ~/.bashrc (first time only)

You will now be asked for your sudo password when provisioning or deploying, and the script will send the password to Ansible whenever it's needed. 🎉

You can also use ansible-vault <encrypt|decrypt> <some-encrypted-file> at any time to quickly encrypt/decrypt secrets files when viewing or editing them. 🎉

Ensure up-to-date secrets when provisioning

There is a playbook to load the secret files into the host_vars directory, to save needing to explicitly include each secrets file. Always run this before provisioning any playbook using secrets, to avoid accidentally pushing old values to the server.

Setup

• Make sure your Github account has your ssh key.
• Run: ansible-playbook playbooks/fetch_secrets.yml
• Run the provision playbook as normal (without secrets), eg ansible-playbook playbooks/provision.yml --limit uk-staging

You can also provision multiple servers at once, for example with --limit europe or --limit all-staging, and the correct secrets will be automatically applied to each server. 🎉

Using Ansible's Ad-Hoc Commands

You can use Ansible's ad-hoc commands to run commands across a group of servers all at once, in the format: ansible <server group> -u <remote user> -a <command> (from the ofn-install directory).

For example, to check the status of certbot certificates on all production servers:

ansible all-prod -u ofn-admin -a "sudo certbot certificates"