[TT-561] move chainlink-env as a package (commit-splitted)

.github/ISSUE_TEMPLATE/bug_report.md

@@ -12,9 +12,9 @@ A clear and concise description of what the bug is.
 
 **Versions**
 
-* OS: 
-* Integrations Framework Version: 
-* chainlink-env Version: 
+* OS:
+* Integrations Framework Version:
+* chainlink-testing-framework Version:
 
 **To Reproduce**
 Steps to reproduce the behavior:

.github/workflows/k8s-e2e.yaml

@@ -0,0 +1,123 @@
+name: (k8s package) E2E tests
+on:
+  pull_request:
+    paths:
+      - "k8s/**"
+
+concurrency:
+  group: e2e-tests-k8s-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  INTERNAL_DOCKER_REPO: ${{ secrets.QA_AWS_ACCOUNT_NUMBER }}.dkr.ecr.${{ secrets.QA_AWS_REGION }}.amazonaws.com
+  ENV_JOB_IMAGE: ${{ secrets.QA_AWS_ACCOUNT_NUMBER }}.dkr.ecr.${{ secrets.QA_AWS_REGION }}.amazonaws.com/chainlink-testing-framework-tests:ci.${{ github.sha }}
+  BASE_IMAGE_NAME: ${{ secrets.QA_AWS_ACCOUNT_NUMBER }}.dkr.ecr.${{ secrets.QA_AWS_REGION }}.amazonaws.com/test-base-image:ci.${{ github.sha }}
+  CHAINLINK_IMAGE: ${{ secrets.QA_AWS_ACCOUNT_NUMBER }}.dkr.ecr.${{ secrets.QA_AWS_REGION }}.amazonaws.com/chainlink
+  CHAINLINK_VERSION: develop
+  SELECTED_NETWORKS: SIMULATED
+  CHAINLINK_COMMIT_SHA: ${{ github.sha }}
+  CHAINLINK_ENV_USER: ${{ github.actor }}
+  TEST_LOG_LEVEL: debug
+
+jobs:
+  build_tests:
+    runs-on: ubuntu-latest
+    environment: integration
+    permissions:
+      id-token: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v3
+      - name: Build Base Image
+        uses: smartcontractkit/chainlink-github-actions/docker/build-push@ce87f8986ca18336cc5015df75916c2ec0a7c4b3 # v2.1.2
+        with:
+          tags: ${{ env.BASE_IMAGE_NAME }}
+          file: k8s/Dockerfile.base
+          AWS_REGION: ${{ secrets.QA_AWS_REGION }}
+          AWS_ROLE_TO_ASSUME: ${{ secrets.QA_AWS_ROLE_TO_ASSUME }}
+      - name: Base Image Built
+        run: |
+          # shellcheck disable=SC2086
+          cat <<EOT >>$GITHUB_STEP_SUMMARY
+          ### chainlink image tag used for this test run :link: => \`${{ env.CHAINLINK_VERSION }}\`
+          ### test-base-image image tag for this test run :ship: => \`ci.${{ github.sha }}\`
+          EOT
+      - name: Build Test Runner
+        uses: smartcontractkit/chainlink-github-actions/docker/build-push@ce87f8986ca18336cc5015df75916c2ec0a7c4b3 # v2.1.2
+        with:
+          tags: ${{ env.ENV_JOB_IMAGE }}
+          file: k8s/Dockerfile
+          build-args: |
+            BASE_IMAGE=${{ secrets.QA_AWS_ACCOUNT_NUMBER }}.dkr.ecr.${{ secrets.QA_AWS_REGION }}.amazonaws.com/test-base-image
+            IMAGE_VERSION=ci.${{ github.sha }}
+          AWS_REGION: ${{ secrets.QA_AWS_REGION }}
+          AWS_ROLE_TO_ASSUME: ${{ secrets.QA_AWS_ROLE_TO_ASSUME }}
+      - name: Test Image Built
+        run: |
+         # shellcheck disable=SC2086
+         cat <<EOT >>$GITHUB_STEP_SUMMARY
+         ### chainlink-testing-framework-tests image tag for this test run :ship: -> \`ci.${{ github.sha }}\`
+         EOT
+  e2e_tests:
+    runs-on: ubuntu-latest
+    environment: integration
+    permissions:
+      id-token: write
+      contents: read
+    env:
+      TEST_SUITE: local-runner
+    steps:
+      - uses: actions/checkout@v3
+      - name: Run Tests
+        uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/run-tests@ce87f8986ca18336cc5015df75916c2ec0a7c4b3 # v2.1.2
+        with:
+          cl_repo: ${{ env.CHAINLINK_IMAGE }}
+          cl_image_tag: ${{ env.CHAINLINK_VERSION }}
+          test_command_to_run: unset ENV_JOB_IMAGE && make k8s_test_e2e_ci
+          test_download_vendor_packages_command: go mod download
+          artifacts_location: ./e2e/logs
+          publish_check_name: E2E Test Results
+          token: ${{ secrets.GITHUB_TOKEN }}
+          go_mod_path: go.mod
+          QA_AWS_REGION: ${{ secrets.QA_AWS_REGION }}
+          QA_AWS_ROLE_TO_ASSUME: ${{ secrets.QA_AWS_ROLE_TO_ASSUME }}
+          QA_KUBECONFIG: ${{ secrets.QA_KUBECONFIG }}
+      - name: Upload test log
+        uses: actions/upload-artifact@v3
+        if: failure()
+        with:
+          name: test-log
+          path: /tmp/gotest.log
+
+  e2e_remote_runner_tests:
+    runs-on: ubuntu-latest
+    environment: integration
+    needs: [build_tests]
+    permissions:
+      id-token: write
+      contents: read
+    env:
+      TEST_SUITE: remote-runner
+      TEST_TRIGGERED_BY: chainlink-testing-framework-remote-runner-ci
+    steps:
+      - uses: actions/checkout@v3
+      - name: Run Remote Runner Tests
+        uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/run-tests@ce87f8986ca18336cc5015df75916c2ec0a7c4b3 # v2.1.2
+        with:
+          cl_repo: ${{ env.CHAINLINK_IMAGE }}
+          cl_image_tag: ${{ env.CHAINLINK_VERSION }}
+          test_command_to_run: make k8s_test_e2e_ci_remote_runner
+          test_download_vendor_packages_command: go mod download
+          artifacts_location: ./k8s/logs
+          publish_check_name: E2E Remote Runner Test Results
+          token: ${{ secrets.GITHUB_TOKEN }}
+          go_mod_path: go.mod
+          QA_AWS_REGION: ${{ secrets.QA_AWS_REGION }}
+          QA_AWS_ROLE_TO_ASSUME: ${{ secrets.QA_AWS_ROLE_TO_ASSUME }}
+          QA_KUBECONFIG: ${{ secrets.QA_KUBECONFIG }}
+      - name: Upload test log
+        uses: actions/upload-artifact@v2
+        if: failure()
+        with:
+          name: remote-runner-test-log
+          path: /tmp/remoterunnergotest.log

.github/workflows/k8s-publish-test-base-image.yaml

@@ -0,0 +1,24 @@
+name: (k8s package) Publish Test Base Image
+on:
+  push:
+    tags:
+      - 'v*'
+
+jobs:
+  publish_test_base_image:
+    runs-on: ubuntu-latest
+    environment: integration
+    permissions:
+      id-token: write
+      contents: read
+    env:
+      BASE_IMAGE_TAG: ${{ secrets.QA_AWS_ACCOUNT_NUMBER }}.dkr.ecr.${{ secrets.QA_AWS_REGION }}.amazonaws.com/test-base-image:${{ github.ref_name }}
+    steps:
+      - uses: actions/checkout@v3
+      - name: Build Base Image
+        uses: smartcontractkit/chainlink-github-actions/docker/build-push@cb4a8f51d77cbf77ea6a765bd1f437ffc7a18730 # v2.0.28
+        with:
+          tags: ${{ env.BASE_IMAGE_TAG }}
+          file: k8s/Dockerfile.base
+          AWS_REGION: ${{ secrets.QA_AWS_REGION }}
+          AWS_ROLE_TO_ASSUME: ${{ secrets.QA_AWS_ROLE_TO_ASSUME }}

.github/workflows/lint.yaml

@@ -1,4 +1,4 @@
-name: Linting and SonarQube Analysis
+name: (all packages) Lints
 on:
   push:
     branches:

.github/workflows/static-analysis.yaml

@@ -1,4 +1,4 @@
-name: Static Analysis
+name: (all packages) Static Analysis
 on:
   push:
     branches: [main]

.github/workflows/test.yaml

@@ -1,4 +1,4 @@
-name: Tests
+name: (all packages) Unit tests
 on:
   push:
     tags:
@@ -29,7 +29,9 @@ jobs:
           PATH=$PATH:$(go env GOPATH)/bin
           export PATH
           set -euo pipefail
-          go test -json -cover -covermode=count -coverprofile=unit-test-coverage.out ./... 2>&1 | tee /tmp/gotest.log | gotestfmt
+          # disabled, because we want to use a multiline output of go list command
+          # shellcheck disable=SC2046
+          go test -json -cover -covermode=count -coverprofile=unit-test-coverage.out $(go list ./... | grep -v /k8s/e2e/ | grep -v /k8s/examples/) 2>&1 | tee /tmp/gotest.log | gotestfmt
       - name: Code Coverage
         uses: codecov/codecov-action@v3
         with:

.gitignore

@@ -54,4 +54,14 @@ docs/Gemfile.lock
 dist/
 
 **/remote_runner_config.yaml
-logs/
+logs/
+
+env/cmd/chaos
+env/bin/
+# temp manifest for deployment and validation
+tmp-manifest-*.yaml
+# remote runner binary
+remote.test
+e2e.test
+
+k3dvolume/

.golangci.yaml

@@ -3,6 +3,8 @@ run:
   timeout: 5m
   skip-dirs:
     - contracts/ethereum
+    - examples
+    - imports
 linters:
   enable:
     # defaults

.tool-versions

@@ -1,7 +1,10 @@
 golang 1.21.3
 nodejs 14.20.0
-k3d 5.4.6
+k3d 5.5.1
 act 0.2.30
 golangci-lint 1.54.2
 actionlint 1.6.17
 shellcheck 0.9.0
+helm 3.10.3
+kubectl 1.25.5
+yarn 1.22.19

Makefile

@@ -1,6 +1,7 @@
 BIN_DIR = bin
 export GOPATH ?= $(shell go env GOPATH)
 export GO111MODULE ?= on
+CDK8S_CLI_VERSION=2.1.48
 
 LINUX=LINUX
 OSX=OSX
@@ -44,15 +45,97 @@ ifeq ($(OSFLAG),$(OSX))
 	asdf plugin add golangci-lint https://github.com/hypnoglow/asdf-golangci-lint.git || true
 	asdf plugin add actionlint || true
 	asdf plugin add shellcheck || true
+	asdf plugin-add yarn || true
+	asdf plugin-add k3d || true
+	asdf plugin-add helm || true
+	asdf plugin-add kubectl || true
 	asdf install
+	mkdir /tmp/k3dvolume/ || true
+	yarn global add cdk8s-cli@$(CDK8S_CLI_VERSION)
+	curl https://github.com/raw/helm/helm/main/scripts/get-helm-3 | bash
+	helm repo add chainlink-qa https://github.com/raw/smartcontractkit/qa-charts/gh-pages/
+	helm repo add grafana https://grafana.github.io/helm-charts
+	helm repo update
 endif
 
 install: go_mod install_tools
 
 install_ci: go_mod install_tools
 
+docker_prune:
+	docker system prune -a -f
+	docker volume prune -f
+
 compile_contracts:
 	python3 ./utils/compile_contracts.py
 
 test_unit: install_gotestfmt
-	go test -json -cover -covermode=count -coverprofile=unit-test-coverage.out ./client ./gauntlet ./testreporters ./docker/test_env 2>&1 | tee /tmp/gotest.log | gotestfmt
+	go test -json -cover -covermode=count -coverprofile=unit-test-coverage.out ./client ./gauntlet ./testreporters ./docker/test_env ./k8s/config 2>&1 | tee /tmp/gotest.log | gotestfmt
+
+
+#######################
+# K8s Helpers
+#######################
+.PHONY: create_cluster
+create_cluster:
+	k3d cluster create local --config ./k8s/k3d.yaml
+
+.PHONY: start_cluster
+start_cluster:
+	k3d cluster start local
+
+.PHONY: stop_cluster
+stop_cluster:
+	k3d cluster stop local
+
+.PHONY: stop_cluster
+delete_cluster:
+	k3d cluster delete local
+
+.PHONY: install_monitoring
+install_monitoring:
+	helm repo add grafana https://grafana.github.io/helm-charts
+	helm repo update
+	kubectl create namespace monitoring || true
+	helm upgrade --wait --namespace monitoring --install loki grafana/loki-stack  --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false,loki.persistence.enabled=false --values k8s/grafana/values.yml
+	kubectl port-forward --namespace monitoring service/loki-grafana 3000:80
+
+.PHONY: uninstall_monitoring
+uninstall_monitoring:
+	helm uninstall --namespace monitoring loki
+
+.PHONY: build_test_base_image
+build_k8s_test_base_image:
+	./k8s/scripts/buildBaseImage "$(tag)"
+
+.PHONY: build_test_image
+build_k8s_test_image:
+	./k8s/scripts/buildTestImage "$(tag)" "$(base_tag)"
+
+k8s_test:
+	go test -race ./k8s/config -count 1 -v
+
+k8s_test_e2e:
+	go test ./k8s/e2e/local-runner -count 1 -test.parallel=12 -v $(args)
+
+k8s_test_e2e_ci:
+	go test ./k8s/e2e/local-runner -count 1 -v -test.parallel=14 -test.timeout=1h -json 2>&1 | tee /tmp/gotest.log | gotestfmt
+
+k8s_test_e2e_ci_remote_runner:
+	go test ./k8s/e2e/remote-runner -count 1 -v -test.parallel=16 -test.timeout=1h -json 2>&1 | tee /tmp/remoterunnergotest.log | gotestfmt
+
+.PHONY: examples
+examples:
+	go run k8s/cmd/test.go
+
+.PHONY: chaosmesh
+chaosmesh: ## there is currently a bug on JS side to import all CRDs from one yaml file, also a bug with stdin, so using cluster directly trough file
+	kubectl get crd networkchaos.chaos-mesh.org -o json > tmp.json && cdk8s import -o k8s/imports/k8s/networkchaos tmp.json
+	kubectl get crd stresschaos.chaos-mesh.org -o json > tmp.json && cdk8s import -o k8s/imports/k8s/stresschaos tmp.json
+	kubectl get crd timechaos.chaos-mesh.org -o json > tmp.json && cdk8s import -o k8s/imports/k8s/timechaos tmp.json
+	kubectl get crd podchaos.chaos-mesh.org -o json > tmp.json && cdk8s import -o k8s/imports/k8s/podchaos tmp.json
+	kubectl get crd podiochaos.chaos-mesh.org -o json > tmp.json && cdk8s import -o k8s/imports/k8s/podiochaos tmp.json
+	kubectl get crd httpchaos.chaos-mesh.org -o json > tmp.json && cdk8s import -o k8s/imports/k8s/httpchaos tmp.json
+	kubectl get crd iochaos.chaos-mesh.org -o json > tmp.json && cdk8s import -o k8s/imports/k8s/iochaos tmp.json
+	kubectl get crd podnetworkchaos.chaos-mesh.org -o json > tmp.json && cdk8s import -o k8s/imports/k8s/podnetworkchaos tmp.json
+	rm -rf tmp.json

README.md

@@ -12,6 +12,59 @@
 
 </div>
 
-The Chainlink Testing Framework is a blockchain development framework written in Go. Its primary purpose is to help chainlink developers create extensive integration, e2e, performance, and chaos tests to ensure the stability of the chainlink project. It can also be helpful to those who just want to use chainlink oracles in their projects to help test their contracts, or even for those that aren't using chainlink. This project makes ample use of the [chainlink-env](https://github.com/smartcontractkit/chainlink-env) package to launch resources and testing apparatus.
+The Chainlink Testing Framework is a blockchain development framework written in Go. Its primary purpose is to help chainlink developers create extensive integration, e2e, performance, and chaos tests to ensure the stability of the chainlink project. It can also be helpful to those who just want to use chainlink oracles in their projects to help test their contracts, or even for those that aren't using chainlink.
 
 If you're looking to implement a new chain integration for the testing framework, head over to the [blockchain](./blockchain/) directory for more info.
+
+## k8s package
+We have a k8s package we are using in tests, it provides:
+- [cdk8s](https://cdk8s.io/) based wrappers
+- High-level k8s API
+- Automatic port forwarding
+
+You can also use this package to spin up standalone environments.
+
+### Local k8s cluster
+Read [here](./k8s/KUBERNETES.md) about how to spin up a local cluster
+
+#### Install
+Set up deps, you need to have `node 14.x.x`, [helm](https://helm.sh/docs/intro/install/) and [yarn](https://classic.yarnpkg.com/lang/en/docs/install/#mac-stable)
+
+Then use
+```shell
+make install_deps
+```
+
+### Running tests in k8s
+To read how to run a test in k8s, read [here](./k8s/REMOTE_RUN.md)
+
+### Usage
+Create an env in a separate file and run it
+```
+export CHAINLINK_IMAGE="public.ecr.aws/chainlink/chainlink"
+export CHAINLINK_TAG="1.4.0-root"
+export CHAINLINK_ENV_USER="Satoshi"
+go run k8s/examples/simple/env.go
+```
+For more features follow [tutorial](./k8s/TUTORIAL.md)
+
+### Development
+#### Running standalone example environment
+```shell
+go run k8s/examples/simple/env.go
+```
+If you have another env of that type, you can connect by overriding environment name
+```
+ENV_NAMESPACE="..."  go run k8s/examples/chainlink/env.go
+```
+
+Add more presets [here](./k8s/presets)
+
+Add more programmatic examples [here](./k8s/examples/)
+
+If you have [chaosmesh]() installed in your cluster you can pull and generated CRD in go like that
+```
+make chaosmesh
+```
+
+If you need to check your system tests coverage, use [that](./k8s/TUTORIAL.md#coverage)