-
Notifications
You must be signed in to change notification settings - Fork 0
Kube
webthing-iotjs was made to target MCU but it can also support other platforms like GnuLinux and its containers (ie: Docker).
This page will explain how to create microservices and run them in a cluster.
There are various Kubernetes (K8S) distributions, a few are described from simplest to more hazardous environments.
Cannonical is providing to community a simplified K8S snap package, Even if snap is part of Ubuntu, it is also also supported by others GnuLinux distros:
Note that MicroK8s was designed to build a single node cluster (WIP for multinode) :
project="webthing-iotjs"
org="rzrfreefr"
image="${org}/${project}:latest"
kubectl=microk8s.kubectl
port=8888
sudo apt-get install snapd curl
sudo snap install microk8s --classic --channel=1.14/stable
microk8s.status --wait-ready
$kubectl cluster-info
#| Kubernetes master is running at https://127.0.0.1:16443
$kubectl get services
#| kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 14h
microk8s.enable dns # Will restart kubelet
$kubectl run "${project}" --image="${image}"
$kubectl describe deployment/$name
# Wait running pod
time $kubectl get all --all-namespaces | grep "pod/$name" | grep ' Running ' \
|| time $kubectl get all --all-namespaces
pod=$($kubectl get all --all-namespaces \
| grep -o "pod/${project}.*" | cut -d/ -f2 | awk '{ print $1}' \
|| echo failure) && echo "# pod=${pod}"
# Wait until ready
$kubectl describe pod "$pod" | grep 'Status: * Running' \
|| $kubectl describe pod "$pod"
# Try server (direct to pod)
ip=$(microk8s.kubectl describe pod "$pod" | grep 'IP:' | awk '{ print $2 }') && echo "# log: ip=${ip}"
url=http://$ip:${port}/
curl -i $url/1/properties
|| curl -i http://$ip:${port}
#| {"level":42.8888}
# Remove service and uninstall
$kubectl delete deployment/${name}
$kubectl get all --all-namespaces | grep ${name}
microk8s.reset
sudo snap remove microk8s
OK we have verified our base, Next step is to deploy a service.
Reinstall microk8s
name="webthing-iotjs"
specUrl="https://github.com/raw/rzr/${name}/master/extra/tools/kube/$name.yml"
service_port=30080
# K8S env
unit=microk8s
export PATH="/snap/$unit/current/:/snap/bin/:$PATH"
kubectl="$unit.kubectl"
time $kubectl apply -f "${specUrl}" \
|| curl "$url"
#| deployment.extensions/webthing-iotjs created
#| service/webthing-iotjs created
# Check ports: container port (docker) and kube port (public)
$kubectl get svc ${name}
#| NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
#| webthing-iotjs NodePort 10.152.183.83 <none> 8888:30080/TCP 3s
service_port=$($kubectl get svc ${name} -o=jsonpath="{.spec.ports[?(@.port==$port)].nodePort}") && echo $service_port
service_url="http://127.0.0.1:${service_port}" && echo "# log: service_url=${service_url}"
#| log: service_url=http://127.0.0.1:30080
time $kubectl get all --all-namespaces | grep "pod/$name" # wait "Running" status
#| default pod/webthing-iotjs-FFFFFFFFFF-FFFFF 1/1 Running 0 72s
curl -i ${service_url}/1/properties \
|| curl -i ${service_url}
#| {"level":42}
$kubectl delete deployment ${name}
$kubectl delete service/${name}
Next step is to setup ingress and public backend.
name="webthing-iotjs"
specUrl="https://github.com/raw/rzr/${name}/master/extra/tools/kube/$name.yml"
port=8888
nodePort=32018
domain=localhost # May edit /etc/hosts
host=${name}.${domain}
url="http://${host}/"
# K8s env
unit=microk8s
export PATH="/snap/$unit/current/:/snap/bin/:$PATH"
kubectl="$unit.kubectl"
# Creating deployent then service from yaml script
# Creating deployent then service from yaml script
curl "${specUrl}" | sed -e "s|nodePort: .*|nodePort: ${nodePort}|g" \
| $kubectl apply -f -
microk8s.enable ingress
$kubectl get deployment ${name} # Ready 1/1
$kubectl delete ingress.extensions/${name} \
|| $kubectl get ing
cat<<EOF | $kubectl create -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ${name}
spec:
rules:
- host: ${host}
http:
paths:
- backend:
serviceName: ${name}
servicePort: ${port}
EOF
ping -c1 ${host} || echo "127.0.0.1 $host" | sudo tee -a /etc/hosts
curl -kLi ${url}
#| [{"id":"urn:dev:ops:my-lamp-1234"
# Extra
microk8s.enable dns # Will restart kubelet
$kubectl delete ingress.extensions/${name}
name="webthing-iotjs"
nodePort=32018
# Common vars
specUrl="https://github.com/raw/rzr/${name}/master/extra/tools/kube/$name.yml"
port=8888
topdomain="volatile.cloudns.org" # must be registered
host="${name}.${topdomain}" # must be registered
publicPort=80
url="http://${host}:${publicPort}"
ingress_name="${name}-ingress"
ingressObject=ingress.extensions/${ingress_name}
# K8s env
unit=microk8s
export PATH="/snap/$unit/current/:/snap/bin/:$PATH"
kubectl="$unit.kubectl"
$kubectl version
$kubectl get ingress.extensions
$kubectl delete ${ingressObject}
$kubectl get all --all-namespaces | grep ${name} | awk '{ print $2 }' \
| xargs -n 1 $kubectl delete \
|| $kubectl get all --all-namespaces
# wait Terminating pod:
$kubectl get all --all-namespaces | grep ${name}
# Creating deployent then service from yaml script
curl "${specUrl}" | sed -e "s|nodePort: .*|nodePort: ${nodePort}|g" \
| $kubectl apply -f -
microk8s.enable ingress # TODO Adapt for other K8S
$kubectl get deployment ${name} # Ready 1/1
$kubectl delete ingress.extensions/${name} \
|| $kubectl get ingress.extensions
cat<<EOF | $kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ${ingress_name}
spec:
rules:
- host: ${host}
http:
paths:
- backend:
serviceName: ${name}
servicePort: ${port}
EOF
# wait from 404 to 503 to 200
ping -c 1 ${host} && curl -kLi "${url}"
#| Server: nginx/1.15.10
#| [{"id":"urn:dev:ops:my-lamp-1234"
$kubectl delete ingress.extensions/${name}
For multiple services it's same just update, run again but replace:
name=webthing-go
nodePort=32019
name=iotjs-express
nodePort=32016
Each service will run on separate hostname.
From Dockerfile to cluster:
export PATH=${PATH}:/snap/bin
git clone https://github.com/rzr/iotjs-express ; cd iotjs-express
make -C extra/tools/kube delete setup start
make -C extra/tools/kube status client
#| curl -kLi http://iotjs-express.localhost/
#| HTTP/1.1 200 OK
#| Server: nginx/1.15.10
#| (...)
#| x-powered-by: iotjs-express
#| {}
curl -kLi http://iotjs-express.$HOSTNAME.local
#| curl: (6) Could not resolve host: iotjs-express.duo.local
make -C extra/tools/kube status ingress domain=$HOSTNAME
make -C extra/tools/kube status host client domain=$HOSTNAME
curl -kLi http://iotjs-express.$HOSTNAME.local
Rebuild docker image is optional, latest published one can be also used, here public domain is also assigned (it must be registered before):
export PATH=${PATH}:/snap/bin
make -C extra/tools/kube delete apply enable ingress status proxy client \
username=rzrfreefr \
domain=volatile.${USER}.cloudns.org
Work In progress:
domain="example.tld.localhost" # TODO: replace with user one
email="$USER@$domain" # TODO: replace with user one
#
name=iotjs-express
spec_url=https://github.com/raw/rzr/${name}/master/extra/tools/kube/${name}.yml
host=${name}.${domain}
port=8888
kubectl=/snap/bin/microk8s.kubectl
export PATH=${PATH}:/snap/bin
#
sudo microk8s.reset # Will erase all env
$kubectl get all
$kubectl get all --all-namespaces
$kubectl delete all
$kubectl delete namespace cert-manager
# $kubectl delete apiservice --all
sudo reboot
microk8s.enable dns
microk8s.enable ingress
$kubectl apply -f "${spec_url}"
$kubectl delete ingress.networking.k8s.io/${name}
cat<<EOF | $kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ${name}
spec:
rules:
- host: ${host}
http:
paths:
- backend:
serviceName: ${name}
servicePort: ${port}
EOF
$kubectl describe ingress.networking.k8s.io/${name}
$kubectl get all # ContainerCreating , then Running
curl -ikL http://${host}/ # OK
# Install Cert Manager for LetsEncrypt
$kubectl create namespace cert-manager
$kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true --overwrite
$kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.9.1/cert-manager.yaml
$kubectl get pods --namespace cert-manager
# Wait: Expected Status: ContainerCreating, Running
# Create staging https
$kubectl delete clusterissuer letsencrypt-staging # ErrRegisterACMEAccount
cat<<EOF | ${kubectl} apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: ${email}
privateKeySecretRef:
name: letsencrypt-staging
server: https://acme-staging-v02.api.letsencrypt.org/directory
solvers:
- http01:
ingress:
class: nginx
EOF
$kubectl describe clusterissuer letsencrypt-staging # Type: Ready
# ErrRegisterACMEAccount
ping -c 1 ${domain}
$kubectl delete certificate ${name}
cat<<EOF | ${kubectl} apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: ${name}
spec:
secretName: ${name}-crt
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
namespace: cert-manager
commonName: ${domain}
dnsNames:
- ${name}.${domain}
EOF
$kubectl describe certificate ${name}
$kubectl get certificate
$kubectl logs -n cert-manager deploy/cert-manager -f
$kubectl get all
$kubectl delete ingress.networking.k8s.io/${name}
cat<<EOF | $kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ${name}
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
spec:
tls:
- hosts:
- ${domain}
secretName: letsencrypt-staging
rules:
- host: ${host}
http:
paths:
- backend:
serviceName: ${name}
servicePort: ${port}
EOF
$kubectl describe ingress.networking.k8s.io/${name} # TLS
#| Normal CreateCertificate 2s cert-manager Successfully created Certificate "letsencrypt-staging"
ping -c 1 ${host}
curl -iL http://${host}/ # OK
curl -ikL https://${host}/ # OK
curl -i https://${host}/ # OK
#| curl: (60) SSL certificate problem: unable to get local issuer certificate
$kubectl delete clusterissuer.certmanager.k8s.io/letsencrypt-prod
cat<<EOF | ${kubectl} apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
email: ${email}
privateKeySecretRef:
name: letsencrypt-prod
server: https://acme-v02.api.letsencrypt.org/directory
solvers:
- http01:
ingress:
class: nginx
EOF
$kubectl describe clusterissuer.certmanager.k8s.io/letsencrypt-prod
$kubectl get secret
# $kubectl describe secret letsencrypt-prod
$kubectl delete certificate.certmanager.k8s.io/${name}
cat<<EOF | ${kubectl} apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: ${name}
spec:
secretName: ${name}-crt
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
namespace: cert-manager
commonName: ${domain}
dnsNames:
- ${name}.${domain}
EOF
$kubectl get certificate ${name} # wait True
#| iotjs-express True iotjs-express-crt 69s
$kubectl describe certificate ${name}
#| Message: Certificate is up to date and has not expired
$kubectl describe challenges
$kubectl logs -n cert-manager deploy/cert-manager -f
# $kubectl log -n cert-manager pod/cert-manager-*
$kubectl get ing
$kubectl delete ingress.networking.k8s.io/${name}
cat<<EOF | $kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ${name}-prod
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- ${host}
secretName: ${name}-prod
rules:
- host: ${host}
http:
paths:
- backend:
serviceName: ${name}
servicePort: ${port}
EOF
$kubectl describe ingress.networking.k8s.io/${name}
# nginx.ingress.kubernetes.io/rewrite-target: /
$kubectl describe ingress.networking.k8s.io/${name}
curl -i https://${host}/
curl -vLi -H "Host: $host" https://localhost
#| * TLSv1.2 (OUT), TLS alert, unknown CA (560):
#| * SSL certificate problem: unable to get local issuer certificate
openssl s_client -connect $host:443 -showcerts
#| (...)
#| 0 s:O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
#| (...)
$kubectl get events
#| (...)
#| 52m Normal GenerateSelfSigned certificate/iotjs-express Generated temporary self signed certificate
#| (...)
$kubectl get challenges --all-namespaces
#| No resources found.
$kubectl get order
#| iotjs-express-786239982 pending 5m45s
$kubectl describe order
#| Warning NoMatchingSolver 47s cert-manager Failed to create challenge for domain "example.tld": no configured challenge solvers can be used for this challenge
$kubectl get certificates
#| NAME READY SECRET AGE
#| iotjs-express False iotjs-express-tls 7m6s
$kubectl describe certificates
#| Message: Certificate issuance in progress. Temporary certificate issued.
$kubectl log -n cert-manager pod/cert-manager-78d45b9d8-m8rst
curl -i https://${host}/ #
$kubectl get events
$kubectl describe challenges
Websites prove their identity via certificates. Firefox does not trust this site because it uses a certificate that is not valid for iotjs-express.example.tld. The certificate is only valid for ingress.local.
Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT
kubectl delete clusterissuer/letsencrypt-staging # Type: Ready
cat<<EOF | ${kubectl} apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: ${domain}
spec:
secretName: ${domain}-crt
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
namespace: cert-manager
commonName: ${domain}
dnsNames:
- "*.${domain}"
EOF
$kubectl describe certificate ${domain}
#| Status: False
#| Type: Ready
$kubectl get order
#| example.tld-324294416 pending 119s
$kubectl get clusterissuer.certmanager.k8s.io/letsencrypt-prod
$kubectl logs -n cert-manager deploy/cert-manager -f
#| I0903 14:09:13.202698 1 logger.go:93] Calling HTTP01ChallengeResponse
#| I0903 14:09:13.203411 1 logger.go:73] Calling GetAuthorization
Warning Failed 50s cert-manager Accepting challenge authorization failed: acme: authorization for identifier example.tld.teuz.eu is invalid
Resources:
- https://github.com/jetstack/cert-manager/
- https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html
- https://docs.cert-manager.io/en/latest/tasks/issuers/setup-acme/index.html
- https://certbot.eff.org/lets-encrypt/debianbuster-other
- https://docs.cert-manager.io/en/latest/reference/orders.html#debugging-order-resources
- https://github.com/jetstack/cert-manager/issues/1399
microk8s.enable istio
- https://linkerd.io/
- https://linkerd.io/2/overview/
- https://www.redhat.com/en/topics/microservices/what-is-a-service-mesh
microk8s.status | grep dns
#| dns: disabled
microk8s.enable dns # Will restart kubelet
TODO avoid definition of each domain and only register a portal hostname
- https://microk8s.io/docs/working
- https://ubuntu.com/blog/single-node-kubernetes-on-raspberry-pi-microk8s-ubuntu
- https://github.com/ubuntu/microk8s/issues/235
- https://github.com/ubuntu/microk8s/pull/626
K3S is another distribution designed for Edge devices, Make sure to uninstall other K8S to avoid overlap of resources
project="webthing-iotjs"
org="rzrfreefr"
image="${org}/${project}:latest"
kubectl="sudo kubectl"
curl -sfL https://get.k3s.io | sh -
sudo snap remove microk8s
sudo systemctl restart k3s.service
sudo systemctl status k3s.service
$kubectl get nodes
#| ... Ready master 51s v1.14.4-k3s.1
$kubectl run "${project}" --image="${image}"
pod=$($kubectl get all --all-namespaces \
| grep -o "pod/${project}.*" | cut -d/ -f2 | awk '{ print $1}' \
|| echo failure) && echo pod="$pod"
$kubectl describe pod "$pod" | grep 'Status: Running'
ip=$($kubectl describe pod "$pod" | grep 'IP:' | awk '{ print $2 }') && echo "ip=${ip}"
curl http://$ip:8888
#| [{"name":"My Lamp"," ...
sudo grep server /etc/rancher/k3s/k3s.yaml
#| server: https://localhost:6443
curl -k -i https://localhost:6443
#| HTTP/1.1 401 Unauthorized
#| Content-Type: application/json
#| Www-Authenticate: Basic realm="kubernetes-master"
# token=$(sudo cat /var/lib/rancher/k3s/server/node-token)
# curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=${token} sh -
An other K8S system designed for prototyping (WIP)
To get started see README.md of :
Or try the webthing-go version as explained at:
- https://github.com/rzr/webthing-go/blob/master/docs/kube.md
- https://github.com/rzr/webthing-go/tree/master/docs
Kadmin is the tool for multi node clustering
Check Concept page for overview, Gateway to get started, IotJs page to install runtime to build webthing as explained in Home page.
For further experiments check Social and Sensor, or Extra parts like WebApp (for Tizen or PWA) or MCU info about running on other microcontrollers not supported by TizenRT.
While Home focus mostly on using iotjs to build webthings (on GNU/Linux or TizenRT for ARTIK05X devices).
This document is still in draft state, but reviews are always welcome, if you try to replicate it and stuck on missing instructions I would appreciate that you file issues or even better make pull request (just edit in github) that insert "TODO marks" in following chapters, like:
- TODO: please explain more this chapter and then remove this TODO line
Community contributions are welcome at:
Support is also possible, ask in:
- https://github.com/rzr/webthing-iotjs
- irc://irc.mozilla.org/#iot
WARNING: Developement branches could break over time.
Instead of maintaining "quick and dirty" demo code, I decided to split demo in smaller independents parts (which can reused) and I am upstreaming the most I can.
Then support can be done on mainline branches (or released versions).
Note that, Upstreaming can be a slow process, so snapshots links will remain until 100% of code is upstreamed.
Licence:
Reference documentation is at:
-
Concept:
- Demo Concept and Architecture
-
Gateway:
- Getting started with Mozilla IoT gateway
-
IotJs:
- Install IoT.js needed to run webthings
-
Home:
- Welcome page to build WebThings using IotJs
-
Social:
- Notification service using Mastodon FLOSS
-
TizenRT:
- webthing-iotjs on ARTIK05x
-
Sensor: and Actuator
- Physical interactions
-
Extra hints:
- Docker: About running in container
- MCU: About microcontrollers (not supported by TizenRT)
- WebApp: Alternate browser (Tizen and PWA)
- GnuLinux: Article about Edison and other
- Raspbian: Article about RaspberryPi
- Arduino: Alt For atmel or Esprissif boards
- DigitalTwins : WiP experiments
- TODO: Work in progress