icp-ce-with-loadbalancers

Terraform ICP IBM Cloud

This Terraform example configurations uses the IBM Cloud provider to provision virtual machines on IBM Cloud Infrastructure (SoftLayer) and Terraform Module ICP Deploy to prepare VSIs and deploy IBM Cloud Private version 3.1.0 or later. This Terraform template automates best practices learned from installing ICP on IBM Cloud Infrastructure.

Deployment overview

This template creates an environment where

• Cluster is deployed on private network and is accessed through load balancers
• SSH access is enabled on public network
• Most ICP services disabled (some can be activated via terraform.tfvars settings as described below)
• Minimal VM sizes
• No separate boot node
• No management node (can be enabled via terraform.tfvars settings as described below)
• No Vulnerability Advisor node and vulnerability advisor service disabled by default

Architecture Diagram

Pre-requisites

• Working copy of Terraform
  • As of this writing, IBM Cloud Terraform provider is not in the main Terraform repository and must be installed manually. See these steps. The templates have been tested with Terraform version 0.11.7 and the IBM Cloud provider version 0.11.3.
• The template is tested on VSIs based on Ubuntu 16.04. RHEL is not supported in this automation.

Using the Terraform templates

1. git clone the repository

2. Navigate to the template directory templates/icp-ce-minimal-with-loadbalancers

3. Create a terraform.tfvars file to reflect your environment. Please see variables.tf and below tables for variable names and descriptions. Here is an example terraform.tfvars file:

sl_username = "<my username>"
sl_api_key  = "<my api key>"
datacenter  = "dal13"
key_name    = ["my-ssh-key"]

1. Run terraform init to download depenencies (modules and plugins)

2. Run terraform plan to investigate deployment plan

3. Run terraform apply to start deployment.

Automation Notes

What does the automation do

1. Create the virtual machines as defined in variables.tf and terraform.tfvars
   • Use cloud-init to add a user icpdeploy with a randomly generated ssh-key
   • Configure a separate hard disk to be used by docker
2. Create security groups and rules for cluster communication as declared in security_group.tf
3. Create load balancers for Proxy and Control plane
4. Handover to the icp-deploy terraform module as declared in the icp-deploy.tf file

What does the icp deploy module do

1. It uses the provided ssh key which has been generated for the icpdeploy user to ssh from the terraform controller to all cluster nodes to install ICP prerequisites
2. It generates a new ssh keypair for ICP Boot(master) node to ICP cluster communication and distributes the public key to the cluster nodes. This key is used by the ICP Ansible installer.
3. It populates the necessary /etc/hosts file on the boot node
4. It generates the ICP cluster hosts file based on information provided in icp-deploy.tf
5. It generates the ICP cluster config.yaml file based on information provided in icp-deploy.tf

Security Groups

The automation leverages Security Groups to lock down public and private access to the cluster.

• Inbound communication to the master and proxy nodes are only permitted on ports from the private subnet that the LBaaS is provisioned on.
• Inbound SSH to the cluster nodes is permitted from all addresses on the internet to ease exploration and investigation.
• All outbound communication is allowed.
• All other communication is only permitted between cluster nodes.

LBaaS

The automation exposes the Master control plane to the Internet on:

• TCP port 8443 (master console)
• TCP port 8500 (private registry)
• TCP port 8600 (private registry)
• TCP port 8001 (Kubernetes API)
• TCP port 9443 (OIDC authentication endpoint)

The automation exposes the Proxy nodes to the internet on:

• TCP port 443 (https)
• TCP port 80 (http)

Terraform configuration

Please see variables.tf for additional parameters.

name	required	value
sl_username	yes	Username for IBM Cloud infrastructure account
sl_api_key	yes	API Key for IBM Cloud infrastructure account
key_name	no	Array of SSH keys to add to root for all created VSI instances. Note that the automation generates its own SSH keys so these are additional keys that can be used for access
datacenter	yes	Datacenter to place all objects in
os_reference_code	yes	OS to install on the VSIs. Use the API to determine valid values. Only Ubuntu 16.04 was tested. Note that the boot node OS can be specified separately (defaults to UBUNTU_16_64 to save licensing costs).
icp_inception_image	yes	The ICP installer image to use. This corresponds to the version of ICP to install. Defaults to 3.1.0
docker_package_location	no	The local path to where the IBM-provided docker installation binary is saved. If not specified and using Ubuntu, will install latest docker-ce off public repo.
private_network_only	no	Specify true to remove the cluster from the public network. If public network access is disabled, note that to allow outbound internet access you will require a Gateway Appliance on the VLAN to do Source NAT. Additionally, the automation requires SSH access to the boot node to provision ICP, so a VPN tunnel may be required. The LBaaS for both the master and the control plane will still be provisioned on the public internet, but the cluster nodes will not have public addresses configured.
private_vlan_router_hostname	no	Private VLAN router to place all VSIs behind. e.g. bcr01a. See Network > IP Management > VLANs in the portal. Leave blank to let the system choose. This option should be used when setting private_network_only to true along with private_vlan_number using a private VLAN that is routed with a Gateway Appliance.
private_vlan_number	no	Private VLAN number to place all VSIs on. e.g. 1211. See Network > IP Management > VLANs in the portal. Leave blank to let the system choose. This option should be used when setting private_network_only to true along with private_vlan_router_hostname, using a private VLAN that is routed with a Gateway Appliance.
public_vlan_router_hostname	no	Public VLAN router to place all VSIs behind. e.g. fcr01a. See Network > IP Management > VLANs in the portal. Leave blank to let the system choose.
public_vlan_number	no	Public VLAN number to place all VSIs on. e.g. 1211. See Network > IP Management > VLANs in the portal. Leave blank to let the system choose.
icppassword	no	ICP administrator password. One will be generated if not set.
deployment	no	Identifier prefix added to the host names of all your infrastructure resources for organising/naming ease

Configuration examples

1. terraform.tfvars which does not add a SSH key and uses all default values. This is the minimum configuration possible.

   sl_username = "<my username>"
   sl_api_key  = "<my api key>"
   datacenter  = "dal13"
   

2. terraform.tfvars which adds a SSH key to the root user and uses all default values.

   sl_username = "<my username>"
   sl_api_key  = "<my api key>"
   datacenter  = "dal13"
   key_name    = ["my-ssh-key"]
   

3. terraform.tfvars which adds a management node and some additional services (metering, monitoring and logging)

   sl_username = "<my username>"
   sl_api_key  = "<my api key>"
   key_name    = ["my-ssh-key"]
   
   # Disable most management services except metering, monitoring and logging
   disabled_management_services = ["istio", "vulnerability-advisor", "storage-glusterfs", "storage-minio", "custom-metrics-adapter", "image-security-enforcement"]
   
   # Enabling metering, monitoring and logging requires additinal resources,
   # so we will enable 1 dedicated management node
   mgmt        = {
     nodes = "1"
   }
   
   

4. terraform.tfvars which adds additional worker nodes, a management node and some additional services (metering, monitoring and logging)

   sl_username = "<my username>"
   sl_api_key  = "<my api key>"
   key_name    = ["my-ssh-key"]
   
   # Disable most management services except metering, monitoring and logging
   disabled_management_services = ["istio", "vulnerability-advisor", "storage-glusterfs", "storage-minio", "custom-metrics-adapter", "image-security-enforcement"]
   
   # Enabling metering, monitoring and logging requires additinal resources,
   # so we will enable 1 dedicated management node
   mgmt        = {
     nodes = "1"
   }
   worker        = {
     nodes = "6"
   }