otaru

Over-Engineering at Its Finest.

Bare-Metal Home Lab for Kubernetes and Technical Playground.

• otaru
  • Architecture
  • Hardware
  • Cluster Components
  • IaaS, PaaS, and SaaS
  • Bootstrap Cluster
  • Oopsy
  • Repository Configuration

Architecture

Hardware

ID	Device	HAT	Role	/dev/mmcblk0	/dev/nvme0n1
etcd	Intel NUC Mini PC Core i3-3217U 8GB	-	-	-	-
raspberrypi-00	Raspberry Pi 4 Model B 8GB	Waveshare PoE HAT (B)	Master	SanDisk Max Endurance 32 GB	-
raspberrypi-01	Raspberry Pi 4 Model B 8GB	Waveshare PoE HAT (B)	Master	SanDisk Max Endurance 32 GB	-
raspberrypi-02	Raspberry Pi 4 Model B 8GB	Waveshare PoE HAT (B)	Master	SanDisk Max Endurance 32 GB	-
raspberrypi-03	Raspberry Pi 5 8GB	Raspberry Pi Active Cooler + Pineberry Pi HatDrive! Bottom	Master	SanDisk Max Endurance 32 GB	Samsung 980 PRO NVMe™ M.2 SSD 2TB (MZ-V8P2T0BW)

Cluster Components

Category	Name	Description
Application	CyberChef	The Cyber Swiss Army Knife by GCHQ
Application	Home Assistant	Home Automation
Application	Jellyfin	Home Media System
Application	Kubernetes Service Patcher	An operator to update the kubernetes service type to LoadBalancer.
Application	SFTPGo	SFTP for Jellyfin
Application	冗PowerBot	Telegram bot tracks and counts individual message counts in groups.
CI/CD	Argo CD	GitOps, drift detection, and reconciliation
Connectivity	Cilium Gateway	Cilium Ingress Controller with Virtual IP Layer 2 announcement and TLS termination
Connectivity	Cilium	Cilium is a networking, observability, and security solution with an eBPF-based dataplane
Connectivity	Cloudflare Tunnel	Cloudflare Zero Trust Edge
Connectivity	Gateway API Kubernetes	Virtual IP and Layer 2 announcement for kubernetes service's External IP
Connectivity	Gateway API	Kubernetes standard CRDs for managing network traffic.
Connectivity	httpbin	Generic health check service
Monitoring	Kubernetes Metrics Server	Scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines
Scheduling	Descheduler	Evicts pods for optimal cluster node utilisation
Scheduling	KEDA	Event Driven Autoscaler
Scheduling	Reloader	Watch changes in ConfigMap and Secret and do rolling upgrades
Security	1Password Connect	Proxy service for 1Password; acts as a secret provider
Security	External Secrets Operator	Extracts secrets from a secret provider
Security	cert-manager	Manages TLS certificates via Let's Encrypt and ACME protocol
Storage	Longhorn	Distributed block storage system; backup and restore from/to remote destinations

IaaS, PaaS, and SaaS

Category	Name	Service	Description
CI/CD	Github	Actions	Run Terragrunt
Connectivity	Cloudflare	Access	Edge Access Control
Connectivity	Cloudflare	DNS	Authoritative DNS Service
Connectivity	Cloudflare	Tunnel	Edge Connectivity
Connectivity	Cloudflare	WARP	VPN to Internal Network
Monitoring	Healthchecks.io	Healthchecks.io	Health Check - Heartbeat
Monitoring	UptimeRobot	UptimeRobot	Health Check
Security	1Password	Connect	Secrets Automation
Security	Let's Encrypt	Let's Encrypt	Certificate Authority
Storage	AWS	S3	Terraform Remote State
Storage	Backblaze	B2	Volume Backup

Bootstrap Cluster

1. Install Tooling

   brew install ansible cilium go-jsonnet helm kubectl terraform terragrunt && ansible-galaxy collection install -r ansible/requirements.yaml

2. Add SSH Keys to known_hosts

   for i in {00..03}; do ssh-keygen -R "raspberrypi-$i.local"; done && for i in {00..03}; do ssh-keyscan "raspberrypi-$i.local" >> ~/.ssh/known_hosts; done

3. Set Up 1Password Credentials

   Follow the 1Password Connect Doc to create 1password-credentials.json and save the access token to the file token.

   ❯ tree $(pwd) -L 1
   /path/to/project/otaru
   ├── 1password-credentials.json
   ├── 1password-credentials.json.sample
   ├── ...
   ├── token
   └── token.sample

4. Bootstrap Cluster

   make

Oopsy

Update host packages and reboot the entire cluster.

make maintenance

Upgrade k3s kubernetes version and restart workloads.

make upgrade-cluster

Wipe everything and start from scratch.

make nuke-cluster

Rebuild the cluster.

make build-cluster

Restart all workloads.

make restart-all

Repository Configuration

Secrets for GitHub Actions

Key
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
B2_APPLICATION_KEY
B2_APPLICATION_KEY_ID
CLOUDFLARE_ACCOUNT_ID
CLOUDFLARE_API_TOKEN
CLOUDFLARE_TUNNEL_SECRET
CLOUDFLARE_ZONE
CLOUDFLARE_ZONE_ID
CLOUDFLARE_ZONE_SUBDOMAIN
CLOUDFLARE_ZONE_TUNNEL_IP_LIST
GH_ADD_COMMENT_TOKEN
GH_DELETE_UNTAGGED_IMAGES_TOKEN
UPTIME_ROBOT_API_KEY