Staging (#7)

* adding instrumentation to docker compose

* updating notes

* enabling instrumentation, adding logging + k8s con

* switching to debug level logging in flask

* adding nginx.conf in image buil

* updating conf to include proxy_pass to flask

* updating expected envvars for RUM init

* enabling rum + calling envvars

* adding base path /api k8s deploy for axios call

* removing hashbang from base url of served vue app

* adding k8s-config deployment files

* adding dockerfile for mongodb container

* removing carryover unused components

* updating notes

* dynamicallu assigning api call routes

* updating backend routes to include /api

* adding dynamic app build env var for vue

* updating styles and correcting logo click behavior

* removing unused component from router

* calling api routes dynamically

* updating notes

* updating nginx server conf in case of page refresh

* adding notes for finding custom theming

* shell script for applying all yaml files in dir

* always pull image + adding envvars to vue

* adding ingress resource

* updating api version for ingress resource

* updating namspace to default

* adding helm values

* updating clustername

* updating clustername

* updating target type for ingress

* updating ingress annotations

* updating namespace

* updating namespace

* updating namespace for ingress to default

* adding elb

* adding ingress controller

* updating type of vue service to LoadBalancer

* updating yaml files

* adding kubeadm init config

* updating init-config

* updating init-config

* updating init file

* enable apm in values.yaml

* updating for dev

* updating notes

* adding recapchta server validation

* updating flask reqs for recaptcha

* adding recaptcha and datadog-ci

* hide recaptcha badge

* adding recaptcha and email validation

* adding favicon

* adding sh script for building/pushing imgs

* adding debounce for window resize event listener

* adding link to faviconn

* adding debounce dep

* removing unused dependency

* adding docker updates

* update flask to read from config

* add config.py to container deploy

* add dev containerized build for vue

* update host for dev build to 0.0.0.0

* update vales for dd-helm to pull latest agent

* adding default nginx config /etc/nginx

* updateing vuenotes

* move docker_push

* adding nginx w/apm dockerfile

* adding trace propogation to nginx

* update main nginx.conf- /etc/nginx

* adding apm specific docker entrypoint

* adding shell script to generate the ddtrace config

* update values to include commented config

* adding envvars to flask deploy + logs source

* adding ust + apm to vue deploy

* adding source annotations to mongo deploy

* removing unnecessary NodePort type from svcs

* adding more notes

* adding headless option to flask config

* adding hl ss and svc

* updating to prevent conflict

* adding tls encryption

* adding host to rule

* udating targetPort for https to nlb to 80

* updating app protocol to http

* updating ingress service to use ssl

* adding ssl redirect annotations

* removing ssl redirect at ingress level

* removing tls at ingress resource

* removing tls updates

* adding default ssl cert to controller deployment

* adding tls options back to ingress

* adding ingressclassname

* kubernetes/ingress-nginx#2724 (comment)

* small updates

* removing unnecessary tls at Ingress

* cleaning up ingress

* removing default ssl opt

* cleaning up nginx-elb

* updating notes

* updating mongo deploy to always pull image

* statefulset image config for mongo

* adding updates to ingress for aws deploy on dev

* removing duplicate init files

* adding notes

* adding multiConnect config to flask config file

* adding working statefulset image

* adding authorization to mongo statefulset

* adding updated mongo init file and script for RS

* adding dd annotations to mongodb

* improve css styling

* improve hamburger animation

* add playfair font as default

* update year in bottombar

* update .ignore to add DS_STORE

* update values for kubeadm

* add ingress controller tracing

* update service name of ingress

* adding clusterconf

* update bind addr for contollermgr and scheduler

* update ing contr servicename for tracing

* add pv for mongo ss

* updating mongo init for envvar ref

* udpating notes

* update log format for trace id injection

* update rum init config in vue app

* add testing values.yaml for helm deployment

* update img of vue deploy to latest

* delete unused conf file

* fix merge conflict

* update flask image to latest

* add env to nginx ingress tracing config

* add modsecurity configmap

* update nginx ingr controller to enable modsecurity

* add to-do in notes

* fixing vuenotes

* upgrading datadog-ci

* update ns for modsec configmap

* add dd-env to nginx-elb

* enable logging for debugging modsec

* update flask

* enable rule processing

* update notes

* clean up logging in flask

* update dockerpush to accept alt dockerfile path

* update aboutme description

* fix docker push

* update ps-vue version

.gitignore

@@ -1 +1,2 @@
-*.icloud
+*.icloud
+*.DS_store

docker-compose-fromfile.yml

@@ -0,0 +1,53 @@
+version: '3'
+services:
+    ddagent:
+      image: datadog/agent:latest
+      container_name: datadog-agent-ps
+      env_file:
+        - ~/sandbox.docker.env
+      environment:
+        - DD_APM_ENABLED=true
+        - DD_LOGS_ENABLED=true
+        - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
+        - DD_APM_NON_LOCAL_TRAFFIC=true
+        # - DD_AC_EXCLUDE=name:datadog-agent-ft
+      volumes: 
+        - /var/run/docker.sock:/var/run/docker.sock:ro
+        - /proc/:/host/proc/:ro
+        - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
+      ports:
+        - "8126:8126"
+      restart: on-failure
+    mongodb:
+      image: cahillsf/ps-mongo:test
+      # image: mongo
+      # environment:
+      #   MONGO_INITDB_ROOT_USERNAME: root
+      #   MONGO_INITDB_ROOT_PASSWORD: example
+      # volumes:
+      #   - ./mongo-db/mongo-db-init-files:/docker-entrypoint-initdb.d
+    #   labels:
+    #     com.datadoghq.ad.check_names: '["mysql"]'
+    #     com.datadoghq.ad.init_configs: '[{}]'
+    #     com.datadoghq.ad.instances: '[{"server": "%%host%%", "user": "datadog","pass": "tacosandburritos"}]'
+      ports:
+        - '27017:27017'
+    flask_server:
+      # image: cahillsf/ps-flask:test
+      build:
+        dockerfile: Dockerfile
+        context: ./flask-server
+      env_file:
+        - ~/sandbox.docker.env
+      ports:
+        - '8000:8000'
+    vue_web:
+      # image: cahillsf/ps-vue:test
+      build:
+      #   dockerfile: Dockerfile
+        dockerfile: Dockerfiledev
+        context: ./funvue
+      env_file:
+        - ~/sandbox.docker.env
+      ports:
+        - '8080:8080'

docker-compose.yaml

@@ -2,55 +2,54 @@ version: '3'
 services:
     # ddagent:
     #   image: datadog/agent:latest
-    #   container_name: datadog-agent-ft
+    #   container_name: datadog-agent-ps
     #   env_file:
     #     - ~/sandbox.docker.env
     #   environment:
     #     - DD_APM_ENABLED=true
     #     - DD_LOGS_ENABLED=true
     #     - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
     #     - DD_APM_NON_LOCAL_TRAFFIC=true
-    #     - DD_AC_EXCLUDE=name:datadog-agent-ft
+    #     # - DD_AC_EXCLUDE=name:datadog-agent-ft
     #   volumes: 
     #     - /var/run/docker.sock:/var/run/docker.sock:ro
     #     - /proc/:/host/proc/:ro
     #     - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
     #   ports:
     #     - "8126:8126"
     #   restart: on-failure
-    #   depends_on: 
-    #     - mysql_db
     mongodb:
-      image: mongo
-      environment:
-        MONGO_INITDB_ROOT_USERNAME: root
-        MONGO_INITDB_ROOT_PASSWORD: example
-      volumes:
-        - ./mongo-db/mongo-db-init-files:/docker-entrypoint-initdb.d
+      image: cahillsf/ps-mongo:test
+      # image: mongo
+      # environment:
+      #   MONGO_INITDB_ROOT_USERNAME: root
+      #   MONGO_INITDB_ROOT_PASSWORD: example
+      # volumes:
+      #   - ./mongo-db/mongo-db-init-files:/docker-entrypoint-initdb.d
     #   labels:
     #     com.datadoghq.ad.check_names: '["mysql"]'
     #     com.datadoghq.ad.init_configs: '[{}]'
     #     com.datadoghq.ad.instances: '[{"server": "%%host%%", "user": "datadog","pass": "tacosandburritos"}]'
       ports:
         - '27017:27017'
-    flask_server:
-      build:
-        dockerfile: Dockerfile
-        context: ./flask-server
-      # depends_on: 
-      #   - mysql_db
-      # environment: 
-      #   - DD_AGENT_HOST=datadog-agent-ft
-      ports:
-        - '8000:8000'
-    vue_web:
-      build:
-        dockerfile: Dockerfile
-        context: ./funvue
-      environment:
-        VUE_APP_DD_APP_ID: root
-        VUE_APP_DD_CLIENT_TOKEN: example
-      # env_file:
-      #   - ~/sandbox.docker.env
-      ports:
-        - '8080:80'
+    # flask_server:
+    #   image: cahillsf/ps-flask:test
+    #   # build:
+    #   #   dockerfile: Dockerfile
+    #   #   context: ./flask-server
+    #   # depends_on: 
+    #   #   - mysql_db
+    #   env_file:
+    #     - ~/sandbox.docker.env
+    #   ports:
+    #     - '8000:8000'
+    # vue_web:
+    #   image: cahillsf/ps-vue:test
+    #   # build:
+    #   #   dockerfile: Dockerfile
+    #     # dockerfile: Dockerfiledev
+    #     # context: ./funvue
+    #   env_file:
+    #     - ~/sandbox.docker.env
+    #   ports:
+    #     - '8080:8080'

docker_push.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+echo "Hello world"
+
+while getopts r:t:i:p:d: flag
+do
+    case "${flag}" in
+        r) repo=${OPTARG};;
+        t) tag=${OPTARG};;
+        i) image_name=${OPTARG};;
+        p) path=${OPTARG};;
+        d) dockerfilepath=${OPTARG};;
+    esac
+done
+echo "Repo: $repo";
+echo "Tag: $tag";
+echo "Image Name: $image_name";
+echo "Path: $path";
+
+docker build -f $dockerfilepath -t $image_name:$tag $path
+docker image tag $image_name:$tag $repo/$image_name:$tag
+docker image push $repo/$image_name:$tag

flask-server/Dockerfile

@@ -3,5 +3,6 @@ RUN mkdir /app
 WORKDIR /app
 ADD requirements.txt /app
 ADD app.py /app
+ADD config.py /app
 RUN pip3 install -r requirements.txt
-CMD ["gunicorn", "-w 4", "-b", "0.0.0.0:8000", "app:app"]
+CMD ["gunicorn", "-w 4", "-b", "0.0.0.0:8000", "--log-level=debug", "app:app"]

flask-server/app.py

@@ -1,41 +1,85 @@
 from flask import Flask, jsonify, request, Response, make_response
 from flask_cors import CORS
-from ddtrace import Pin, patch
+import logging
 import pymongo
 from pprint import pprint
 import sys
 import datetime
 import os
-#import ddtrace.profiling.auto
-# from ddtrace.profiling.profiler import Profiler
-
+import requests
+import json
+import werkzeug
+## THIS IS THE TRACING BLOCK IN USE IN IMAGE FOR K8S#####
+# import ddtrace.profiling.auto
+from ddtrace import config, patch_all, Pin, patch, tracer
+config.env = "dev"      # the environment the application is in
+config.service = "flask-server"  # name of your application
+config.version = "0.0.1"  # version of your application
+patch(logging=True)
+patch_all()
+#################
 
 # configuration
-DEBUG = True
+
+# DEBUG = True
 # prof = Profiler(
 #     env="dev",  # if not specified, falls back to environment variable DD_ENV
 #     service="flask-server",  # if not specified, falls back to environment variable DD_SERVICE
 #     version="0.0.1",   # if not specified, falls back to environment variable DD_VERSION
 # )
 # prof.start()
 
+FORMAT = ('%(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d] '
+          '[dd.service=%(dd.service)s dd.env=%(dd.env)s dd.version=%(dd.version)s dd.trace_id=%(dd.trace_id)s dd.span_id=%(dd.span_id)s] '
+          '- %(message)s')
+logging.basicConfig(format=FORMAT)
+
 # instantiate the app
 app = Flask(__name__)
-app.config.from_object(__name__)
+deployment = os.environ.get('DEPLOYMENT')
+config_string = 'config.' + deployment + 'Config'
+my_obj_instance = werkzeug.utils.import_string(config_string)()
+app.config.from_object(my_obj_instance)
+
+db_uri = str(app.config['DATABASE_URI'])
+print(app.config['DATABASE_URI'], file=sys.stderr)
+client = pymongo.MongoClient(db_uri)
+
 # enable CORS
 # CORS(app, resources={r'/*': {'origins': '*'}})
 CORS(app, origins=["http://localhost:8080"], headers=['Content-Type'], expose_headers=['Access-Control-Allow-Origin'], supports_credentials=True)
-# client = pymongo.MongoClient('mongodb://flask-role:toor@localhost:27017/sitecontent?authSource=sitecontent')
-client = pymongo.MongoClient('mongodb://flask-role:toor@mongodb:27017/sitecontent?authSource=sitecontent')
+
+
+def deploy_to_mongo_host(argument):
+    switcher = {
+        "local": "localhost",
+        "docker": "mongodb",
+        "kubernetes": "ps-mongo-service",
+    }
+    return switcher.get(argument, "nothing")
 
 def create_user_obj(input):
     clean_obj = {'email': str(input['email']), 'name': str(input['submitterName']), 'message':str(input['message'])}
     print(clean_obj, file=sys.stderr)
+    app.logger.info(clean_obj)
     return clean_obj
 
+def validate_response(resp): 
+    app.logger.info("validating response" + str(resp))
+    print(resp['success'], file=sys.stderr)
+    success = True if resp['success'] == True else False
+    print(success, file=sys.stderr)
+    human = True if resp['score'] > 0.3 else False
+    correct_action = True if resp['action'] == 'formSubmit' else False
+    if (human and correct_action and resp['success']):
+        return "valid"
+    return "invalid"
 
-@app.route('/cards', methods=['GET'])
+
+@app.route('/api/cards', methods=['GET'])
+# @app.route(base_url + '/cards', methods=['GET'])
 def all_cards():
+    app.logger.info("getting cards")
     db = client['sitecontent']
     cards = db.cards
     cards_cursor = cards.find({})
@@ -44,17 +88,47 @@ def all_cards():
     for index, document in enumerate(cards_cursor):
         print(document, file=sys.stderr)
         cards_dict[index] = document
-    # print(type(cards_dict), file=sys.stderr)
     return (cards_dict)
 
-@app.route('/createMessage', methods=['POST'])
+@app.route('/api/createMessage', methods=['POST'])
 def createMessage():
     print(request.json, file=sys.stderr)
     db = client['sitecontent']
     clean_user_obj = create_user_obj(request.json)
-    x = db.messages.insert_one(clean_user_obj)
+    x = db.messages.insert_one(clean_user_obj)  
     print(x, file=sys.stderr)
     return "OKAY", 200
 
+@app.route('/api/testRoute', methods=['GET'])
+def testRoute():
+    print(os.environ.get('DD_AGENT_HOST'), file=sys.stderr)
+    app.logger.info(os.environ.get('DD_AGENT_HOST'))
+    print("test route", file=sys.stderr)
+    app.logger.info("test route")
+    return "OKAY", 200
+
+@app.route('/api/recaptcha', methods=['POST'])
+def recaptcha():
+    print(request.json, file=sys.stderr)
+    recaptcha_secret= str(os.environ.get('RECAPTCHA_SECRET'))
+    token = request.json['token']
+    app.logger.info("validating with recaptcha")
+    url = 'https://www.google.com/recaptcha/api/siteverify?secret='+ recaptcha_secret + '&response=' + token
+    response = requests.post(url)
+    response_dict = json.loads(response.text)
+    response_assement = validate_response(response_dict)
+    return response_assement, 200
+
+#local deployment
 if __name__ == '__main__':
-    app.run(host="localhost", port=8000, debug=True)
+    app.run(host="localhost", port=8000, debug=True)
+    log = logging.getLogger(__name__)
+    log.level = logging.DEBUG
+
+#k8s deployment using gunicorn
+if __name__ != '__main__':
+    gunicorn_logger = logging.getLogger('gunicorn.error')
+    app.logger.handlers = gunicorn_logger.handlers
+    app.logger.setLevel(gunicorn_logger.level)
+    base_url = "/api"
+    mongo_client = "ps-mongo-service"

flask-server/config.py

@@ -0,0 +1,25 @@
+class Config(object):
+    """Base config, uses staging database server."""
+    TESTING = False
+    DB_SERVER = 'localhost'
+
+    @property
+    def DATABASE_URI(self):  # Note: all caps
+        if self.DB_SERVER == 'mongod-0.mongodb-service.default.svc.cluster.local:27017':
+            return f"mongodb://flask-role:toor@{self.DB_SERVER}/sitecontent?authSource=sitecontent&replicaSet=MainRepSet"
+        return f"mongodb://flask-role:toor@{self.DB_SERVER}:27017/sitecontent?authSource=sitecontent"
+
+class localConfig(Config):
+    DB_SERVER = 'localhost'
+
+class dockerConfig(Config):
+    DB_SERVER = 'mongodb'
+
+class kubernetesConfig(Config):
+    DB_SERVER = 'ps-mongo-service'
+
+class headlessConfig(Config):
+    DB_SERVER = 'mongodb-service'
+
+class headlessConfigMultiConnectConfig(Config):
+    DB_SERVER = 'mongod-0.mongodb-service.default.svc.cluster.local:27017'

flask-server/requirements.txt

@@ -1,10 +1,13 @@
 attrs==21.2.0
+certifi==2021.10.8
+charset-normalizer==2.0.12
 click==8.0.1
 ddtrace==0.57.0
 Flask==1.1.2
 Flask-Cors==3.0.10
 Flask-PyMongo==2.3.0
 gunicorn==20.1.0
+idna==3.3
 importlib-metadata==4.7.1
 itsdangerous==2.0.1
 Jinja2==3.0.1
@@ -13,8 +16,10 @@ packaging==21.0
 protobuf==3.19.0
 pymongo==3.12.0
 pyparsing==2.4.7
+requests==2.27.1
 six==1.16.0
 tenacity==8.0.1
 typing-extensions==3.10.0.0
+urllib3==1.26.8
 Werkzeug==2.0.1
 zipp==3.5.0

funvue/Dockerfile

@@ -10,6 +10,8 @@ RUN npm run build
 FROM nginx:stable-alpine as production-stage
 COPY --from=build-stage /app/dist /usr/share/nginx/html
 COPY ./docker_init ./
+RUN rm -rf /etc/nginx/conf.d/default.conf
+COPY ./nginx.conf /etc/nginx/conf.d
 RUN chmod +x ./docker_entrypoint.sh ./generate_env_config.sh
 EXPOSE 80
 # CMD ["nginx", "-g", "daemon off;"]