CloudRail Assignment Task: Automate the Deployment and Monitoring of a Web Application Using Open-Source Tools:

Things to keep in mind :

• Application has 2 components which Frontend and Backend
• Frontend is based on Angular Framework and is of Node 8 version and Angular Cli Version 1.7.3
• Backend is based on Nodejs and express and is of Node 8 version
• Application also use MongoDB as a Datatbase. I have used MongoDB Atlas Cluster URI.

CICD Flow used for the Project:

• Source Code is pushed to Github Repository.
• For Automating Continous Integration , I have used Github Action Workflow which works similar to Jenkins, GitLab CI,etc.
• I have created ci.yml file which has all the Stages and Step. File is located in .github/workflows directory.
• The Stages Constitute stages like : Build, Test & Deploy.
• For Continous Deployment , I have used ArgoCD which is a GitOps Controller and Open Source Tool.

Stage 1 in Github Action Workflow : Build & Push

• In this Stage, I am Checking Out the Code Installing Trivy and Setting Up NodeJs Environment.
• Trivy is a free and open source tool to check vulnerabilities and scan containers as well as Images build using Docker.
• Using Docker , I dockerised the Frontend & Backend of the application using Dockerfiles for the respective .
• After Dockerizing the Frontend & backend , Images Are scanned by trivy and pushed to DockerHub Registery.

Stage 2 in Github Action Workflow : Test Frontend & Backend

• In this Stage, I am Testing the Build Dependencies Installed in Frontend & Backend.
• I have Setup the Required Node Verison to install Dependencies and Test.

Stage 3 in Github Action Workflow : Update Manifest & Deploy Application

• In this Stage, I am updating the YML Manifests with the updated Image build during dockerizing.
• After completing the above 2 stages we are updating the Image version on a different repository specifically maintained to store manifest files used for Kubernetes Cluster.
• After Updating the Deployment.yml for both frontend and backend, I pushed the code with the new commit.

Creating AWS EKS Cluster to deploy Application

Pre-requisites:

• an EC2 Instance (Note : If Using Ubuntu EC2 Instance instead of Amazon Linux then Make Sure to have aws-iam-authenticator installed.)

Article to Install aws-iam-authenticator :

https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html

AWS EKS Setup

1. Setup kubectl
   a. Download kubectl
   b. Grant execution permissions to kubectl executable
   c. Move kubectl onto /usr/local/bin
   d. Test that your kubectl installation was successful

   curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/kubectl
   chmod +x ./kubectl
   mv ./kubectl /usr/local/bin 
   kubectl version --short --client

2. Setup eksctl
   a. Download and extract the latest release
   b. Move the extracted binary to /usr/local/bin
   c. Test that your eksclt installation was successful

   curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
   sudo mv /tmp/eksctl /usr/local/bin
   eksctl version

3. Create an IAM Role and attach it to EC2 instance
   Note: create IAM user with programmatic access if your bootstrap system is outside of AWS
   IAM user should have access to
   IAM
   EC2
   VPC
   CloudFormation EKS Administrator

4. Create your cluster and nodes

   eksctl create cluster --name cluster-name  \
   --region region-name \
   --node-type instance-type \
   --nodes-min 2 \
   --nodes-max 2 \ 
   --zones <AZ-1>,<AZ-2>

Installing ArgoCD in EKS cluster to make Continous Deployment

Prerequisites:

• An existing EKS Cluster.
• AWS Load Balancer Controller Installed.

Step 1 : We Create a namespace and Install Manifest for ArgoCD

kubectl create ns argocd
kubectl apply -f https://github.com/raw/argoproj/argo-cd/v2.4.7/manifests/install.yaml -n argocd

Step 2 : Configuring ArgoCD

• Here I have Edited the ArgoCD Server Manifest file from ClusterIP to LoadBalancer to access the ArgoCD UI in Browser.

kubectl edit svc argocd-server -n argocd

• Setting Up the manifest repository in ArgoCD.

Successfully Deployed Our Web Application

• I have used LoadBalancer in my Service yml manifest to access the application outside the cluster

Cluster Monitoring using Prometheus & Grafana

Key Components :

• Prometheus server - Processes and stores metrics data
• Alert Manager - Sends alerts to any systems/channels
• Grafana - Visualize scraped data in UI

Pre Requisites :

• EKS Cluster is setup already
• Install Helm
• EC2 instance to access EKS cluster

Installation Steps

helm repo add stable https://charts.helm.sh/stable
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm search repo prometheus-community
kubectl create namespace prometheus
helm install stable prometheus-community/kube-prometheus-stack -n prometheus
kubectl get pods -n prometheus
kubectl get svc -n prometheus

Edit Prometheus Service (Edit type : LoadBalancer)

kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus

Edit Grafana Service (Edit type : LoadBalancer)

kubectl edit svc stable-grafana -n prometheus

Verify if service is changed to LoadBalancer and also to get the Load Balancer URL.

kubectl get svc -n prometheus

Access Grafana Dashboard

UserName: admin 
Password: prom-operator

• Prometheus UI also used LoadBalancer to access in Browser

Creating Customised Monitoring Dashboard using Prometheus & Grafana

• Prometheus is used to gather the dynamic realtime timeseries metrics of nodes from kubelet and we used it to give data to grafana so that we can used it to visualize
• In Grafana I have used Prometheus as a data source and grafana retrieves data from prometheus using queries.

Our Dynamic Customised Monitoring Dashboard for our cluster

• I have used Memory & CPU metrics for the pods and nodes.
• For checking the realtime scaling and replicas of the deployment we can use other queries to retrieve data.
• Frontend & Backend Containers can also be monitored in the dashboard.