chore: add nightly gke cluster cleanup job

[amandavialva01]

Description

Create nightly cleanup job that removes all namespaces (and therefore all k8s objects within them) from the CircleCI GKE cluster and deletes all buckets created by CI jobs used for testing with that cluster.

Successful run with resources to delete: https://app.circleci.com/pipelines/github/determined-ai/determined/52752/workflows/ef46cdef-ca8d-4345-a1e7-49e15d7473aa/jobs/2370042

Successful run with no resources to delete: https://app.circleci.com/pipelines/github/determined-ai/determined/52764/workflows/0a6c0b3f-8e4b-427e-ba92-d6f7b5e930a6/jobs/2370726

Test Plan

CI passes.

Checklist

• Changes have been manually QA'd
• User-facing API changes need the "User-facing API Change" label.
• Release notes should be added as a separate file under docs/release-notes/.
  See Release Note for details.
• Licenses should be included for new code which was copied and/or modified from any external code.

Ticket

DET-10117

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.63%. Comparing base (5cb7927) to head (80bcf54).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9031      +/-   ##
==========================================
- Coverage   46.64%   46.63%   -0.01%     
==========================================
  Files        1172     1172              
  Lines      143619   143619              
  Branches     2410     2410              
==========================================
- Hits        66986    66983       -3     
- Misses      76428    76431       +3     
  Partials      205      205              

Flag	Coverage Δ
backend	43.38% <ø> (ø)
harness	63.99% <ø> (-0.01%)	⬇️
web	37.77% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 3 files with indirect coverage changes

[carolinaecalderon]

LGTM

[corban-beaird]

Also LGTM! 😄

[cla-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @djanicekpach, @jesse-amano-hpe, @determined-ci on file. In order for us to review and merge your code, please start the CLA process at https://determined.ai/cla.

After we approve your CLA, we will update the contributors list (private) and comment @cla-bot[bot] check to rerun the check.

[determined-ci]

Docsite preview being generated for this PR.
You can (eventually) find the generated docsite here.

[dannysauer]

Approving for the sake of moving forward, but there are come changes I'd like to see longer-term. :)

[dannysauer]

The right-er way to mass delete from kubernetes is to apply labels, and then delete items which have a given label value. If the namespaces were created with a label which was, for example, like "determiend.ai/ci" and an arbitrary value (perhaps the ID of the test), then this could be as simple as:

Suggested change

	kubectl get namespace | grep -Eo "^test-cpu-[a-z0-9]+-[a-z0-9]+-[0-9]" | xargs -L1 kubectl delete namespace || true
	kubectl delete namespace -l determiend.ai/ci

[amandavialva01]

That's a great idea. I will look into adding a unique label to each namespace to simplify this cleanup job in the future

[dannysauer]

The gcloud delete commands mostly take multiple targets for delete, so this could be done (faster) with a single command along the lines of this:

Suggested change

	gcloud compute networks get-effective-firewalls $GCP_NETWORK_NAME --project "$GCP_PROJECT_ID" \
	--format="table(name)" | tail -n +2 | \
	while read fw; do
	if [[ $fw =~ "k8s" ]]; then
	gcloud compute firewall-rules delete "$fw" --quiet --project "$GCP_PROJECT_ID"
	fi
	done
	gcloud compute firewall-rules delete "$fw" --quiet --project "$GCP_PROJECT_ID" $( \
	gcloud compute networks get-effective-firewalls $GCP_NETWORK_NAME --project "$GCP_PROJECT_ID" --regexp=".*k8s.*" \
	--format="table(name)" | tail -n +2
	)

Piping into a while loop like that means any command inside the loop which reads from stdin will consume output before the read gets another chance, so it's generally ideal to avoid that structure for maintenance (works now, but what if someone adds a second command in the loop later?).

[dannysauer]

This would actually be even easier with a cloud-custodian yaml file which could just select the stuff to remove. :D

[amandavialva01]

It seems like the kubectl delete namespace command removes firewall rules, target pools, and forwarding rules (since those resources are namespace). So, I'll remove these three steps from the cleaup job since they get inherently deleted from the deletion of the namespace!

[dannysauer]

Awesome, even better!

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	80bcf54
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/660f5f82a99ac000087e898a