444 add a gh action for deploying and verifying dashboards alerts load ok with quickstart #708

ehearneRedHat · 2024-06-18T10:45:12Z

What:

GitHub Action Verify Dashboards and Alerts OK created to test whether dashboards and alerts are deployed successfully.

How:

Within the Verify Dashboard and Alerts OK workflow, it spins up the infrastructure needs for self-hosted runner via Terraform, then it runs /hack/quickstart-setup.sh first to run kuadrant-operator and in particular loads in the Observability Stack which contains our dashboards via Grafana, and our alerts via Prometheus:

It exposes the grafana service endpoint in the kind cluster and performs HTTP GET requests via Grafana's local API to ensure it can detect the dashboards in examples/dashboards/ . Once the requests are carried out, it kills the process ID associated with grafana service endpoint exposure.
It performs a very similar process with alerts in examples/alerts/ via Prometheus.

Verify:

Create a branch referencing the commits from 444-add-a-gh-action-for-deploying-and-verifying-dashboards-alerts-load-ok-with-quickstart using the origin https://www.github.com/Kuadrant/kuadrant-operator.
Comment out the triggers in .github/workflows/verify-dashboard-alerts.yaml so that it gets triggered everytime there is a push. Then, commit and push the changes to observe the workflow. If the workflow runs quickstart, performs HTTP requests, it did its job correctly.
Revert the changes made and delete the branch.

codecov · 2024-06-18T10:48:51Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.48%. Comparing base (ece13e8) to head (fc8472f).
Report is 131 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (ece13e8) and HEAD (fc8472f). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (ece13e8) HEAD (fc8472f)

integration 1 0

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #708       +/-   ##
===========================================
- Coverage   80.20%   69.48%   -10.73%     
===========================================
  Files          64       76       +12     
  Lines        4492     5896     +1404     
===========================================
+ Hits         3603     4097      +494     
- Misses        600     1483      +883     
- Partials      289      316       +27

Flag	Coverage Δ
bare-k8s-integration	`4.56% <ø> (?)`
gatewayapi-integration	`10.90% <ø> (?)`
integration	`?`
istio-integration	`57.03% <ø> (?)`
unit	`33.03% <ø> (+3.00%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
api/v1beta1 (u)	`71.42% <ø> (ø)`
api/v1beta2 (u)	`90.85% <100.00%> (-0.58%)`	⬇️
pkg/common (u)	`79.66% <ø> (-9.17%)`	⬇️
pkg/istio (u)	`73.88% <ø> (-0.03%)`	⬇️
pkg/log (u)	`94.73% <ø> (ø)`
pkg/reconcilers (u)	`∅ <ø> (∅)`
pkg/rlptools (u)	`83.33% <ø> (+3.87%)`	⬆️
controllers (i)	`64.99% <66.00%> (-11.81%)`	⬇️

see 44 files with indirect coverage changes

changed deployment to prometheus adapter changed deployment to blackbox-operator changed wait condition to prometheus k8s pod

fix for loop syntax fixed string declaration errors and array read issues fixed pid declaration

better implementation of tf script for use with github actions and secrets

syntax error another one fixed variable errors passed secrets as variables fixed variable declaration so many issues... added additional changes fixed syntax errors more syntax errors hopefully fixed syntax added public ip to terraform and action fixed env var declare

fix for ssh issue moved step created ssh dir and touched new file a b changed perms to 400 changed user to ubuntu added additional package to install for gh runner install libicu changed url check for what the url is another one echo repo changed repo name to static a changed auth type B added permissions for github token e ee e changed from github token to secret

added nohup hopefully fix :D source bashrc changed shell prefs fixed typo install expect and change perms for user in action adding chown because i am a clown :D install yq because y not remove classical flag a

attempt to use statefile new changes *E* a

o . o a a

david-martin · 2024-07-02T15:41:22Z

the quickstart script is unable to set up networking related components of kuadrant-operator

What components cannot be set up using a built in runner?

self-hosted-runner.tf

.github/workflows/verify-dashboards-alerts.yaml

david-martin · 2024-07-03T08:02:08Z

Thanks for the awesome work on this @ehearneRedHat
It definitely adds a lot of value and I'm inclined to see it merged soon.
I'd like to understand a few things a little better first though (questions inline) so we know how best to maintain the workflows being added.

ehearneRedHat · 2024-07-04T07:52:46Z

What components cannot be set up using a built in runner?

When running the quickstart in a GitHub hosted runner I am met with plenty of errors pertaining to missing network libraries, and even when these libraries are installed, because the quickstart requests to run a container in privileged mode, it cannot, as the runner itself is not privileged.

Therefore in order to run the network components, the runner had to be self hosted, and to avoid costs surrounding a self hosted runner powerful enough to run all the components within the quickstart, Terraform was used as keeping such a runner up for a prolonged period of time would be quite costly.

If you have any further questions please let me know. :D

Thanks for the awesome work on this @ehearneRedHat .

No problem, @david-martin ! It was great to have had the opportunity to learn so much about this side of runners. :)

ehearneRedHat · 2024-07-08T09:22:30Z

@pehala @jsmolar @david-martin

New build of ubuntu-latest now seems to run quickstart script just fine in GitHub hosted runner. Therefore, there is no longer a need to run workflow in self-hosted runner.

Therefore, verification steps have been updated, and redundant files have now been removed.

pehala

It looks much better now! Thanks for updating it

.github/workflows/verify-dashboards-alerts.yaml

fixed typo E e e e a a a meep meep

e a beep bop boop boop

e e e e create ami e e final touches :) e

test test

test test 1

a

a a a

ehearneRedHat · 2024-07-08T14:45:33Z

🎉 😄

david-martin

Thanks for taking the time to address all the comments and feedback @ehearneRedHat
This solves the original problem in a standard way that should be relatively easy to maintain.

david-martin · 2024-07-08T16:14:40Z

Merging with known test flake unrelated to the changes in this PR (see https://kubernetes.slack.com/archives/C05J0D0V525/p1720453954262009)

ehearneRedHat added 5 commits June 14, 2024 11:52

added script

101b306

added faulty code to github action

7256b63

switched to podman (limited success)

71ec541

changed to self-hosted and removed install for packages

6c8678e

test push

6f4d502

ehearneRedHat requested a review from a team as a code owner June 18, 2024 10:45

ehearneRedHat linked an issue Jun 18, 2024 that may be closed by this pull request

Add a GH action for deploying and verifying dashboards & alerts load OK with quickstart #444

Closed

changed trigger to pull request + fix typos

a41016e

ehearneRedHat force-pushed the 444-add-a-gh-action-for-deploying-and-verifying-dashboards-alerts-load-ok-with-quickstart branch 2 times, most recently from 4a08f4a to f7a5cc4 Compare June 20, 2024 14:45

ehearneRedHat added 4 commits June 20, 2024 15:46

fixed typos + added wait for prometheus

fa55047

changed deployment to prometheus adapter changed deployment to blackbox-operator changed wait condition to prometheus k8s pod

added alias for sudo podman

83bb3f3

fix for loop syntax fixed string declaration errors and array read issues fixed pid declaration

added files to .gitignore

c3b9691

added terraform file for self-hosted-runner

9796180

better implementation of tf script for use with github actions and secrets

ehearneRedHat force-pushed the 444-add-a-gh-action-for-deploying-and-verifying-dashboards-alerts-load-ok-with-quickstart branch from f7a5cc4 to 9796180 Compare June 20, 2024 14:47

ehearneRedHat added 3 commits June 24, 2024 14:35

ran command in background

fb7b5ac

added nohup hopefully fix :D source bashrc changed shell prefs fixed typo install expect and change perms for user in action adding chown because i am a clown :D install yq because y not remove classical flag a

ehearneRedHat force-pushed the 444-add-a-gh-action-for-deploying-and-verifying-dashboards-alerts-load-ok-with-quickstart branch from 39036eb to fb7b5ac Compare June 24, 2024 13:35

ehearneRedHat marked this pull request as draft June 26, 2024 09:07

ehearneRedHat changed the title ~~[TEST - do not review] 444 add a gh action for deploying and verifying dashboards alerts load ok with quickstart~~ [DRAFT] 444 add a gh action for deploying and verifying dashboards alerts load ok with quickstart Jun 26, 2024

ehearneRedHat added 4 commits June 26, 2024 13:00

have deregister run if previous job fails

37b517b

attempt to use statefile new changes *E* a

terraform.tfstate

6a08c43

o . o a a

changed over to github app for token gen

ebc3d8c

save changes (undone)

63c2a78

ehearneRedHat force-pushed the 444-add-a-gh-action-for-deploying-and-verifying-dashboards-alerts-load-ok-with-quickstart branch 2 times, most recently from e86b8cc to 4bdee2b Compare June 26, 2024 13:54

ehearneRedHat changed the title ~~[DRAFT] 444 add a gh action for deploying and verifying dashboards alerts load ok with quickstart~~ 444 add a gh action for deploying and verifying dashboards alerts load ok with quickstart Jun 27, 2024

ehearneRedHat marked this pull request as ready for review June 27, 2024 10:16

david-martin reviewed Jul 2, 2024

View reviewed changes

self-hosted-runner.tf Outdated Show resolved Hide resolved

david-martin reviewed Jul 2, 2024

View reviewed changes

self-hosted-runner.tf Outdated Show resolved Hide resolved

david-martin reviewed Jul 2, 2024

View reviewed changes

.github/workflows/verify-dashboards-alerts.yaml Outdated Show resolved Hide resolved

pehala reviewed Jul 8, 2024

View reviewed changes

ehearneRedHat added 9 commits July 8, 2024 11:35

swapped out github action for manual api interactions instead

67f8bd4

fixed typo E e e e a a a meep meep

added code to remove runner through ec2 instance

003d5f7

e a beep bop boop boop

add terraform script and workflow for create ami

aa3d658

e e e e create ami e e final touches :) e

null resource block comment

d5c7dba

check for ssh-ability

cd52dff

cleanup + changed runner to github for verify workflow .

b65101c

removed chown

b4fbe1e

test test

separated set up and tear down from tests

653bfb3

test test 1

additional clean up

be80445

a

ehearneRedHat force-pushed the 444-add-a-gh-action-for-deploying-and-verifying-dashboards-alerts-load-ok-with-quickstart branch 3 times, most recently from f2ab911 to 7c69e7b Compare July 8, 2024 11:25

removed quickstart dependency

fc8472f

a a a

ehearneRedHat force-pushed the 444-add-a-gh-action-for-deploying-and-verifying-dashboards-alerts-load-ok-with-quickstart branch from c4e2c16 to fc8472f Compare July 8, 2024 14:38

ehearneRedHat requested a review from pehala July 8, 2024 14:39

pehala approved these changes Jul 8, 2024

View reviewed changes

ehearneRedHat requested a review from david-martin July 8, 2024 14:45

david-martin approved these changes Jul 8, 2024

View reviewed changes

david-martin merged commit e627fb3 into main Jul 8, 2024
24 of 26 checks passed

ehearneRedHat deleted the 444-add-a-gh-action-for-deploying-and-verifying-dashboards-alerts-load-ok-with-quickstart branch July 26, 2024 13:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

444 add a gh action for deploying and verifying dashboards alerts load ok with quickstart #708

444 add a gh action for deploying and verifying dashboards alerts load ok with quickstart #708

ehearneRedHat commented Jun 18, 2024 •

edited

Loading

codecov bot commented Jun 18, 2024 •

edited

Loading

david-martin commented Jul 2, 2024

david-martin commented Jul 3, 2024

ehearneRedHat commented Jul 4, 2024

ehearneRedHat commented Jul 8, 2024

pehala left a comment

ehearneRedHat commented Jul 8, 2024

david-martin left a comment

david-martin commented Jul 8, 2024

444 add a gh action for deploying and verifying dashboards alerts load ok with quickstart #708

444 add a gh action for deploying and verifying dashboards alerts load ok with quickstart #708

Conversation

ehearneRedHat commented Jun 18, 2024 • edited Loading

What:

How:

Verify:

codecov bot commented Jun 18, 2024 • edited Loading

Codecov Report

david-martin commented Jul 2, 2024

david-martin commented Jul 3, 2024

ehearneRedHat commented Jul 4, 2024

ehearneRedHat commented Jul 8, 2024

pehala left a comment

Choose a reason for hiding this comment

ehearneRedHat commented Jul 8, 2024

david-martin left a comment

Choose a reason for hiding this comment

david-martin commented Jul 8, 2024

ehearneRedHat commented Jun 18, 2024 •

edited

Loading

codecov bot commented Jun 18, 2024 •

edited

Loading