Skip to content

Kube

Jump to bottom Edit New page
Philippe Coval edited this page Sep 3, 2019 · 27 revisions

KUBE:

webthing-iotjs was made to target MCU but it can also support other platforms like GnuLinux and its containers (ie: Docker).

This page will explain how to create microservices and run them in a cluster.

There are various Kubernetes (K8S) distributions, a few are described from simplest to more hazardous environments.

MICROK8S:

Cannonical is providing to community a simplified K8S snap package, Even if snap is part of Ubuntu, it is also also supported by others GnuLinux distros:

Note that MicroK8s was designed to build a single node cluster (WIP for multinode) :

MICROK8S : DEPLOY AND TEST

project="webthing-iotjs"
org="rzrfreefr"
image="${org}/${project}:latest"
kubectl=microk8s.kubectl
port=8888

sudo apt-get install snapd curl
sudo snap install microk8s --classic --channel=1.14/stable

microk8s.status --wait-ready

$kubectl cluster-info
#| Kubernetes master is running at https://127.0.0.1:16443

$kubectl get services
#| kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP   14h

microk8s.enable dns # Will restart kubelet

$kubectl run "${project}" --image="${image}"

$kubectl describe deployment/$name

# Wait running pod
time $kubectl get all --all-namespaces | grep "pod/$name"  | grep ' Running ' \
    || time $kubectl get all --all-namespaces 
    
pod=$($kubectl get all --all-namespaces \
  | grep -o "pod/${project}.*" | cut -d/ -f2 | awk '{ print $1}' \
  || echo failure) && echo "# pod=${pod}"

# Wait until ready
$kubectl describe pod "$pod" | grep 'Status: * Running' \
   || $kubectl describe pod "$pod"
   
# Try server (direct to pod)
ip=$(microk8s.kubectl describe pod "$pod" | grep 'IP:' | awk '{ print $2 }') && echo "# log: ip=${ip}"
url=http://$ip:${port}/

curl -i $url/1/properties
  || curl -i http://$ip:${port}
#| {"level":42.8888}

# Remove service and uninstall
$kubectl delete deployment/${name}
$kubectl get all --all-namespaces | grep ${name}

microk8s.reset

sudo snap remove microk8s

OK we have verified our base, Next step is to deploy a service.

MICROK8S: SERVICE

Reinstall microk8s

name="webthing-iotjs"
specUrl="https://github.com/raw/rzr/${name}/master/extra/tools/kube/$name.yml"
service_port=30080
# K8S env
unit=microk8s
export PATH="/snap/$unit/current/:/snap/bin/:$PATH"
kubectl="$unit.kubectl"

time $kubectl apply -f "${specUrl}" \
  || curl "$url"

#| deployment.extensions/webthing-iotjs created
#| service/webthing-iotjs created

# Check ports: container port (docker) and kube port (public)
$kubectl get svc ${name} 
#| NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
#| webthing-iotjs   NodePort   10.152.183.83   <none>        8888:30080/TCP   3s

service_port=$($kubectl get svc ${name} -o=jsonpath="{.spec.ports[?(@.port==$port)].nodePort}") && echo $service_port
service_url="http://127.0.0.1:${service_port}" && echo "# log: service_url=${service_url}"
#| log: service_url=http://127.0.0.1:30080

time $kubectl get all --all-namespaces | grep "pod/$name" # wait "Running" status
#| default     pod/webthing-iotjs-FFFFFFFFFF-FFFFF   1/1     Running   0          72s

curl -i ${service_url}/1/properties \
  || curl -i ${service_url}
#| {"level":42}

$kubectl delete deployment ${name}
$kubectl delete service/${name}

Next step is to setup ingress and public backend.

MICROKS: LOCAL INGRESS

name="webthing-iotjs"
specUrl="https://github.com/raw/rzr/${name}/master/extra/tools/kube/$name.yml"
port=8888
nodePort=32018
domain=localhost # May edit /etc/hosts
host=${name}.${domain}
url="http://${host}/"
# K8s env
unit=microk8s
export PATH="/snap/$unit/current/:/snap/bin/:$PATH"
kubectl="$unit.kubectl"

# Creating deployent then service from yaml script

# Creating deployent then service from yaml script
curl "${specUrl}" | sed -e "s|nodePort: .*|nodePort: ${nodePort}|g" \
  | $kubectl apply -f -
  
microk8s.enable ingress
$kubectl get deployment ${name} # Ready 1/1

$kubectl delete ingress.extensions/${name} \
  || $kubectl get ing

cat<<EOF | $kubectl create -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ${name}
spec:
  rules:
  - host: ${host}
    http:
      paths:
      - backend:
          serviceName: ${name}
          servicePort: ${port}
EOF
ping -c1 ${host} || echo "127.0.0.1 $host" | sudo tee -a /etc/hosts

curl -kLi ${url}
#| [{"id":"urn:dev:ops:my-lamp-1234"

# Extra
microk8s.enable dns # Will restart kubelet
$kubectl delete ingress.extensions/${name}

MICROK8S: PUBLIC INGRESS

name="webthing-iotjs"
nodePort=32018
# Common vars
specUrl="https://github.com/raw/rzr/${name}/master/extra/tools/kube/$name.yml"
port=8888
topdomain="volatile.cloudns.org" # must be registered
host="${name}.${topdomain}" # must be registered
publicPort=80
url="http://${host}:${publicPort}"
ingress_name="${name}-ingress"
ingressObject=ingress.extensions/${ingress_name}
# K8s env
unit=microk8s
export PATH="/snap/$unit/current/:/snap/bin/:$PATH"
kubectl="$unit.kubectl"

$kubectl version
$kubectl get ingress.extensions

$kubectl delete ${ingressObject}

$kubectl get all --all-namespaces  | grep ${name} | awk '{ print $2 }' \
  | xargs -n 1 $kubectl delete \
  || $kubectl get all --all-namespaces
# wait Terminating pod:
$kubectl get all --all-namespaces  | grep ${name}

# Creating deployent then service from yaml script
curl "${specUrl}" | sed -e "s|nodePort: .*|nodePort: ${nodePort}|g" \
  | $kubectl apply -f -
        
microk8s.enable ingress # TODO Adapt for other K8S
$kubectl get deployment ${name} # Ready 1/1

$kubectl delete ingress.extensions/${name} \
  || $kubectl get ingress.extensions

    cat<<EOF | $kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ${ingress_name}

spec:
  rules:
  - host: ${host}
    http:
      paths:
      - backend:
          serviceName: ${name}
          servicePort: ${port}
EOF
# wait from 404 to 503 to 200
ping -c 1 ${host} && curl -kLi "${url}" 
#| Server: nginx/1.15.10
#| [{"id":"urn:dev:ops:my-lamp-1234"

$kubectl delete ingress.extensions/${name}

For multiple services it's same just update, run again but replace:

name=webthing-go
nodePort=32019
    
name=iotjs-express
nodePort=32016

Each service will run on separate hostname.

MICROK8S HELPER

From Dockerfile to cluster:

export PATH=${PATH}:/snap/bin
git clone https://github.com/rzr/iotjs-express ; cd iotjs-express
make -C extra/tools/kube delete setup start
make -C extra/tools/kube status client
#| curl -kLi http://iotjs-express.localhost/
#| HTTP/1.1 200 OK
#| Server: nginx/1.15.10
#| (...)
#| x-powered-by: iotjs-express
#| {}

curl -kLi http://iotjs-express.$HOSTNAME.local
#| curl: (6) Could not resolve host: iotjs-express.duo.local

make -C extra/tools/kube status ingress domain=$HOSTNAME

make -C extra/tools/kube status host client domain=$HOSTNAME
curl -kLi http://iotjs-express.$HOSTNAME.local

Rebuild docker image is optional, latest published one can be also used, here public domain is also assigned (it must be registered before):

export PATH=${PATH}:/snap/bin
make -C extra/tools/kube delete apply enable ingress status proxy client \
 username=rzrfreefr \
 domain=volatile.${USER}.cloudns.org

MICROK8S: HTTPS

Work In progress:

domain="example.tld.localhost" # TODO: replace with user one
email="$USER@$domain" # TODO: replace with user one
#
name=iotjs-express
spec_url=https://github.com/raw/rzr/${name}/master/extra/tools/kube/${name}.yml
host=${name}.${domain}
port=8888
kubectl=/snap/bin/microk8s.kubectl
export PATH=${PATH}:/snap/bin

#
sudo microk8s.reset # Will erase all env

$kubectl get all
$kubectl get all --all-namespaces
$kubectl delete all
$kubectl delete namespace cert-manager
# $kubectl delete apiservice --all
sudo reboot

microk8s.enable dns
microk8s.enable ingress

$kubectl apply -f "${spec_url}"

$kubectl delete ingress.networking.k8s.io/${name}
cat<<EOF | $kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ${name}
spec:
  rules:
  - host: ${host}
    http:
      paths:
      - backend:
          serviceName: ${name}
          servicePort: ${port}
EOF
$kubectl describe ingress.networking.k8s.io/${name}

$kubectl get all # ContainerCreating , then Running
curl -ikL http://${host}/ # OK

# Install Cert Manager for LetsEncrypt
$kubectl create namespace cert-manager
$kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true --overwrite
$kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.9.1/cert-manager.yaml
$kubectl get pods --namespace cert-manager 

# Wait: Expected Status: ContainerCreating, Running

# Create staging https
$kubectl delete clusterissuer letsencrypt-staging # ErrRegisterACMEAccount
cat<<EOF | ${kubectl} apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: ${email}
    privateKeySecretRef:
      name: letsencrypt-staging
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx
EOF
$kubectl describe clusterissuer letsencrypt-staging # Type: Ready
# ErrRegisterACMEAccount

ping -c 1 ${domain}

$kubectl delete certificate ${name}
cat<<EOF | ${kubectl} apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: ${name}
spec:
  secretName: ${name}-crt
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
    namespace: cert-manager
  commonName: ${domain}
  dnsNames:
  - ${name}.${domain}
EOF
$kubectl describe certificate ${name}
$kubectl get certificate

$kubectl logs -n cert-manager deploy/cert-manager -f
$kubectl get all

$kubectl delete ingress.networking.k8s.io/${name}
cat<<EOF | $kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ${name}
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging
spec:
  tls:
  - hosts:
    - ${domain}
    secretName: letsencrypt-staging
  rules:
  - host: ${host}
    http:
      paths:
      - backend:
          serviceName: ${name}
          servicePort: ${port}
EOF
$kubectl describe ingress.networking.k8s.io/${name} # TLS
#|   Normal  CreateCertificate  2s    cert-manager  Successfully created Certificate "letsencrypt-staging"

ping -c 1 ${host}
curl -iL http://${host}/ # OK
curl -ikL https://${host}/ # OK

curl -i https://${host}/ # OK
#| curl: (60) SSL certificate problem: unable to get local issuer certificate


$kubectl delete clusterissuer.certmanager.k8s.io/letsencrypt-prod
cat<<EOF | ${kubectl} apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: ${email}
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx
EOF
$kubectl describe clusterissuer.certmanager.k8s.io/letsencrypt-prod

$kubectl get secret 
# $kubectl describe secret letsencrypt-prod

$kubectl delete certificate.certmanager.k8s.io/${name}
cat<<EOF | ${kubectl} apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: ${name}
spec:
  secretName: ${name}-crt
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
    namespace: cert-manager
  commonName: ${domain}
  dnsNames:
  - ${name}.${domain}
EOF

$kubectl get certificate ${name} # wait True
#| iotjs-express   True    iotjs-express-crt   69s

$kubectl describe certificate ${name}
#|  Message:               Certificate is up to date and has not expired

$kubectl describe challenges

$kubectl logs -n cert-manager deploy/cert-manager -f
# $kubectl log -n cert-manager  pod/cert-manager-*

$kubectl get ing

$kubectl delete ingress.networking.k8s.io/${name}
cat<<EOF | $kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ${name}-prod
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - ${host}
    secretName: ${name}-prod
  rules:
  - host: ${host}
    http:
      paths:
      - backend:
          serviceName: ${name}
          servicePort: ${port}
EOF
$kubectl describe ingress.networking.k8s.io/${name}

#     nginx.ingress.kubernetes.io/rewrite-target: /


$kubectl describe ingress.networking.k8s.io/${name}

curl -i https://${host}/ 

curl -vLi  -H "Host: $host" https://localhost
#| * TLSv1.2 (OUT), TLS alert, unknown CA (560):
#| * SSL certificate problem: unable to get local issuer certificate

openssl s_client -connect $host:443 -showcerts
#| (...)
#| 0 s:O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
#| (...)


$kubectl get events
#| (...)
#| 52m         Normal    GenerateSelfSigned    certificate/iotjs-express   Generated temporary self signed certificate
#| (...)

$kubectl get challenges --all-namespaces
#| No resources found.

$kubectl get order
#| iotjs-express-786239982   pending   5m45s

$kubectl describe order
#|  Warning  NoMatchingSolver  47s   cert-manager  Failed to create challenge for domain "example.tld": no configured challenge solvers can be used for this challenge

$kubectl get certificates 
#| NAME            READY   SECRET              AGE
#| iotjs-express   False   iotjs-express-tls   7m6s

$kubectl describe certificates 
#| Message:               Certificate issuance in progress. Temporary certificate issued.

$kubectl log -n cert-manager  pod/cert-manager-78d45b9d8-m8rst


curl -i https://${host}/ #

$kubectl get events

$kubectl describe challenges

Websites prove their identity via certificates. Firefox does not trust this site because it uses a certificate that is not valid for iotjs-express.example.tld. The certificate is only valid for ingress.local.

Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT

kubectl  delete  clusterissuer/letsencrypt-staging # Type: Ready
 
cat<<EOF | ${kubectl} apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: ${domain}
spec:
  secretName: ${domain}-crt
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
    namespace: cert-manager
  commonName: ${domain}
  dnsNames:
  - "*.${domain}"
EOF

$kubectl describe certificate ${domain}

#| Status:                False
#| Type:                  Ready

$kubectl get order
#| example.tld-324294416              pending   119s

$kubectl get clusterissuer.certmanager.k8s.io/letsencrypt-prod

$kubectl logs -n cert-manager deploy/cert-manager -f
#| I0903 14:09:13.202698       1 logger.go:93] Calling HTTP01ChallengeResponse
#| I0903 14:09:13.203411       1 logger.go:73] Calling GetAuthorization


 Warning  Failed     50s   cert-manager  Accepting challenge authorization failed: acme: authorization for identifier example.tld.teuz.eu is invalid

Resources:

MICROK8S: ISTEO (TODO)

microk8s.enable istio

MICROK8S: LINKERD (TODO)

DNS (WIP)

microk8s.status | grep dns
#| dns: disabled
microk8s.enable dns # Will restart kubelet

TODO avoid definition of each domain and only register a portal hostname

MICROK8S RESOURCES:

K3S

K3S is another distribution designed for Edge devices, Make sure to uninstall other K8S to avoid overlap of resources

project="webthing-iotjs"
org="rzrfreefr"
image="${org}/${project}:latest"
kubectl="sudo kubectl"

curl -sfL https://get.k3s.io | sh -
sudo snap remove microk8s
sudo systemctl restart k3s.service
sudo systemctl status k3s.service

$kubectl get nodes
#| ...   Ready    master   51s   v1.14.4-k3s.1

$kubectl run "${project}" --image="${image}"

pod=$($kubectl get all --all-namespaces \
  | grep -o "pod/${project}.*" | cut -d/ -f2 | awk '{ print $1}' \
  || echo failure) && echo pod="$pod"
$kubectl describe pod "$pod"  | grep 'Status:             Running' 
ip=$($kubectl describe pod "$pod" | grep 'IP:' | awk '{ print $2 }') && echo "ip=${ip}"

curl http://$ip:8888
#| [{"name":"My Lamp"," ...

sudo grep server /etc/rancher/k3s/k3s.yaml
#| server: https://localhost:6443

curl -k -i https://localhost:6443
#| HTTP/1.1 401 Unauthorized
#| Content-Type: application/json
#| Www-Authenticate: Basic realm="kubernetes-master"

# token=$(sudo cat /var/lib/rancher/k3s/server/node-token)
# curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=${token} sh -

MINIKUBE

An other K8S system designed for prototyping (WIP)

To get started see README.md of :

Or try the webthing-go version as explained at:

KUBERNETES/ KADMIN

Kadmin is the tool for multi node clustering

kubernetes

RESOURCES

LICENSE: CC-BY-SA-4.0

LOST ?

Check Concept page for overview, Gateway to get started, IotJs page to install runtime to build webthing as explained in Home page.

For further experiments check Social and Sensor, or Extra parts like WebApp (for Tizen or PWA) or MCU info about running on other microcontrollers not supported by TizenRT.

While Home focus mostly on using iotjs to build webthings (on GNU/Linux or TizenRT for ARTIK05X devices).

FEEDBACK: WELCOME

This document is still in draft state, but reviews are always welcome, if you try to replicate it and stuck on missing instructions I would appreciate that you file issues or even better make pull request (just edit in github) that insert "TODO marks" in following chapters, like:

  • TODO: please explain more this chapter and then remove this TODO line

Community contributions are welcome at:

Support is also possible, ask in:

DISCLAIMER:

WARNING: Developement branches could break over time.

Instead of maintaining "quick and dirty" demo code, I decided to split demo in smaller independents parts (which can reused) and I am upstreaming the most I can.

Then support can be done on mainline branches (or released versions).

Note that, Upstreaming can be a slow process, so snapshots links will remain until 100% of code is upstreamed.

Licence:

Reference documentation is at:

INDEX

Clone this wiki locally