feat: [DENA-603] Example of init database job #50

MarcinGinszt · 2024-07-11T08:45:31Z

create new resource which ensures that database and user exists and creates cert for the user
create example of using it
fix the OpsLevel annotations
Fix two bugs:

wrong hostname for init job- with current hostname (cockroachdb-proxy) cluster was not initialized correctly and nodes were unable to communicate
ensure init job won't crash for fresh databases

…annotations

linear · 2024-07-11T08:45:33Z

DENA-603 Create example of Cockroachdb initialization (user, db)

matthewhughes-uw · 2024-07-11T09:38:40Z

cockroachdb/example/cert-manager/db-init-manifests/example-database/kustomization.yaml

+      kind: Job
+      name: cockroach-init
+      version: v1
+  # This patch creates new client cert (to be mounted within client)


confused by this comment: below it looks to modify an existing one, not create a new one (since the patch is replaceing)?

We create new client cert. When you were writing this comment, we were doing that by replacing the name. Now, we are doing that by adding name prefix.
We are using base to create new resources with overlays 🙂
Adding new directories like example-database with similar content will cause creation of new jobs and ceriticates. So that you can initialize multiple databases

matthewhughes-uw · 2024-07-11T09:39:55Z

cockroachdb/example/cert-manager/db-init-manifests/example-database/kustomization.yaml

+        path: /metadata/name
+        value: example-database-cockroachdb-init
+      - op: replace
+        path: /spec/template/spec/containers/0/env/3/value


is a strategic merge patch possible here, so we can set values based on env var names, and not their position, which have the potential to change over time with updates?

matthewhughes-uw · 2024-07-11T10:14:13Z

cockroachdb/manifests-cert-manager/scripts.yaml

+
+    # If database already exists and contains tables, grant access to them for user. 
+    # This command will fail for freshly initialized databases.
+    $SQL_CMD --execute "GRANT ALL ON TABLE $DB_NAME.* TO $DB_USER;" || true


I don't suppose there's an easy way to make this conditional in the SQL like TABLES_COUNT > 0 && GRANT ... so we avoid any confusing/worrying looking logs in the case no tables exist

Cockroachdb doesn't have conditional statements, it seems.
That's why I came up with this solution. Currently, logs are looking like

Cluster is not ready yet. Retry 1/20. Waiting for 3 seconds... Cluster is not ready yet. Retry 2/20. Waiting for 3 seconds... Cluster is not ready yet. Retry 3/20. Waiting for 3 seconds... Cluster is not ready yet. Retry 4/20. Waiting for 3 seconds... Cluster is not ready yet. Retry 5/20. Waiting for 3 seconds... Cluster is not ready yet. Retry 6/20. Waiting for 3 seconds... CREATE ROLE CREATE DATABASE GRANT ERROR: no object matched SQLSTATE: 42704 Failed running "sql"

Which is not awful, I think.
But why not, I made them nicer here

matthewhughes-uw · 2024-07-11T10:32:02Z

cockroachdb/manifests-cert-manager/db-init/cockroach-db-init.yaml

+            - name: COCKROACH_PORT
+              valueFrom:
+                configMapKeyRef:
+                  name: cockroach
+                  key: cockroach.port
+            - name: COCKROACH_CERTS_DIR
+              value: "/cockroach-certs/"
+            # This variables should be replaced by patches
+            - name: DB_NAME
+              value: "default"


why do we mix some values being set from configMap and some being patched in, what are the different use cases for each approach?

It's db init job- it's used as a base to create new jobs for each new db/user.
Some of the env will be the same for each of jobs (set from CM), while some needs to be customized (I almost spelled it through k, lol), and those are patched.

Would you suggest documenting this better?

Speaking of documentation, do we reference these examples in READMEs?

I plan to rewrite most of the documentation, so that it does fit the current formula from shared-kustomize-bases. Hence, I'll do it it another ticket

github-actions · 2024-07-17T13:24:26Z

Post kustomize build diff:

diff --git a/built-manifests/cockroachdb/example/cert-manager/db-init-manifests/example-database/manifests.yaml b/built-manifests/cockroachdb/example/cert-manager/db-init-manifests/example-database/manifests.yaml
new file mode 100644
index 0000000..414e1bc
--- /dev/null
+++ b/built-manifests/cockroachdb/example/cert-manager/db-init-manifests/example-database/manifests.yaml
@@ -0,0 +1,78 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  labels:
+    app: cockroach-db-init
+    version: v1
+  name: example-database-cockroach-db-init
+spec:
+  template:
+    spec:
+      containers:
+      - command:
+        - /bin/bash
+        - /opt/scripts/user-schema-bootstrap.sh
+        env:
+        - name: DB_NAME
+          value: example_database
+        - name: DB_USER
+          value: example_user
+        - name: COCKROACH_HOST
+          valueFrom:
+            configMapKeyRef:
+              key: cockroach.host
+              name: cockroach
+        - name: COCKROACH_PORT
+          valueFrom:
+            configMapKeyRef:
+              key: cockroach.port
+              name: cockroach
+        - name: COCKROACH_CERTS_DIR
+          value: /cockroach-certs/
+        image: cockroachdb/cockroach:v23.2.2
+        imagePullPolicy: IfNotPresent
+        name: db-init
+        resources:
+          limits:
+            cpu: 1
+            memory: 512Mi
+          requests:
+            cpu: 0
+            memory: 128Mi
+        volumeMounts:
+        - mountPath: /cockroach-certs
+          name: client-certs
+        - mountPath: /opt/scripts
+          name: cockroachdb-scripts
+          readOnly: true
+      restartPolicy: OnFailure
+      volumes:
+      - name: client-certs
+        projected:
+          defaultMode: 256
+          sources:
+          - secret:
+              items:
+              - key: ca.crt
+                path: ca.crt
+              - key: tls.crt
+                path: client.root.crt
+              - key: tls.key
+                path: client.root.key
+              name: cockroachdb.client.root
+      - configMap:
+          name: cockroachdb-scripts
+        name: cockroachdb-scripts
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: example-database-cockroach-client
+spec:
+  commonName: example_user
+  issuerRef:
+    kind: Issuer
+    name: cockroachdb-ca-issuer
+  secretName: cockroachdb.client.example-user
+  usages:
+  - client auth
diff --git a/root-manifests/cockroachdb/example/cert-manager/manifests.yaml b/built-manifests/cockroachdb/example/cert-manager/manifests.yaml
index b7e4552..1e675e0 100644
--- a/root-manifests/cockroachdb/example/cert-manager/manifests.yaml
+++ b/built-manifests/cockroachdb/example/cert-manager/manifests.yaml
@@ -2,6 +2,8 @@ apiVersion: v1
 kind: ServiceAccount
 metadata:
   annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
     vault.uw.systems/aws-role: <your aws role here>
   name: cockroachdb
   namespace: <your namespace here>
@@ -12,6 +14,9 @@ data:
   cockroach.port: "26257"
 kind: ConfigMap
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   name: cockroach-g92fdb9h27
 ---
 apiVersion: v1
@@ -20,6 +25,9 @@ data:
   schedule: 0 0,12 * * *
 kind: ConfigMap
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   name: cockroach.backup.config-mch5d9m542
 ---
 apiVersion: v1
@@ -75,12 +83,17 @@ data:
     EOF
 kind: ConfigMap
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   name: cockroachdb-scripts
 ---
 apiVersion: v1
 kind: Service
 metadata:
   annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
     prometheus.io/path: _status/vars
     prometheus.io/port: "8001"
     prometheus.io/scrape: "true"
@@ -107,6 +120,9 @@ spec:
 apiVersion: v1
 kind: Service
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   labels:
     app: cockroachdb
   name: cockroachdb-proxy
@@ -128,10 +144,10 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   annotations:
-    app.uw.systems/description: Used to connect and query the cockroachdb databases.
+    app.uw.systems/description: Cockroachdb instance used to store ...
     app.uw.systems/repos: https://github.com/utilitywarehouse/shared-kustomize-bases/cockroachdb
     app.uw.systems/tags.oss: "true"
-    app.uw.systems/tier: tier_4
+    app.uw.systems/tier: tier_1
     secret.reloader.stakater.com/reload: cockroachdb.client.root
   name: cockroachdb-client
 spec:
@@ -141,6 +157,9 @@ spec:
       app: cockroachdb-client
   template:
     metadata:
+      annotations:
+        app.uw.systems/description: Cockroachdb instance used to store ...
+        app.uw.systems/tier: tier_1
       labels:
         app: cockroachdb-client
       name: cockroachdb-client
@@ -199,10 +218,10 @@ apiVersion: apps/v1
 kind: StatefulSet
 metadata:
   annotations:
-    app.uw.systems/description: Used to store data projections of events in multiple
-      services.
+    app.uw.systems/description: Cockroachdb instance used to store ...
     app.uw.systems/repos: https://github.com/utilitywarehouse/shared-kustomize-bases/cockroachdb
     app.uw.systems/tags.oss: "true"
+    app.uw.systems/tier: tier_1
     secret.reloader.stakater.com/reload: cockroachdb.node
   labels:
     app: cockroachdb
@@ -217,6 +236,7 @@ spec:
   template:
     metadata:
       annotations:
+        app.uw.systems/description: Cockroachdb instance used to store ...
         app.uw.systems/tier: tier_1
         uw.systems/kyverno-inject-sidecar-request: vault-sidecar-aws
       labels:
@@ -324,6 +344,9 @@ spec:
 apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   labels:
     app: cockroachdb
   name: cockroachdb-budget
@@ -336,10 +359,17 @@ spec:
 apiVersion: batch/v1
 kind: Job
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   name: cockroach-backup-init
 spec:
   backoffLimit: 10
   template:
+    metadata:
+      annotations:
+        app.uw.systems/description: Cockroachdb instance used to store ...
+        app.uw.systems/tier: tier_1
     spec:
       containers:
       - command:
@@ -407,11 +437,18 @@ spec:
 apiVersion: batch/v1
 kind: Job
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   labels:
     app: cockroachdb
   name: cockroach-init
 spec:
   template:
+    metadata:
+      annotations:
+        app.uw.systems/description: Cockroachdb instance used to store ...
+        app.uw.systems/tier: tier_1
     spec:
       containers:
       - command:
@@ -455,9 +492,85 @@ spec:
               name: cockroachdb.client.root
   ttlSecondsAfterFinished: 600
 ---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
+  labels:
+    app: cockroach-db-init
+    version: v1
+  name: example-database-cockroach-db-init
+spec:
+  template:
+    metadata:
+      annotations:
+        app.uw.systems/description: Cockroachdb instance used to store ...
+        app.uw.systems/tier: tier_1
+    spec:
+      containers:
+      - command:
+        - /bin/bash
+        - /opt/scripts/user-schema-bootstrap.sh
+        env:
+        - name: DB_NAME
+          value: example_database
+        - name: DB_USER
+          value: example_user
+        - name: COCKROACH_HOST
+          valueFrom:
+            configMapKeyRef:
+              key: cockroach.host
+              name: cockroach-g92fdb9h27
+        - name: COCKROACH_PORT
+          valueFrom:
+            configMapKeyRef:
+              key: cockroach.port
+              name: cockroach-g92fdb9h27
+        - name: COCKROACH_CERTS_DIR
+          value: /cockroach-certs/
+        image: cockroachdb/cockroach:v23.2.2
+        imagePullPolicy: IfNotPresent
+        name: db-init
+        resources:
+          limits:
+            cpu: 1
+            memory: 512Mi
+          requests:
+            cpu: 0
+            memory: 128Mi
+        volumeMounts:
+        - mountPath: /cockroach-certs
+          name: client-certs
+        - mountPath: /opt/scripts
+          name: cockroachdb-scripts
+          readOnly: true
+      restartPolicy: OnFailure
+      volumes:
+      - name: client-certs
+        projected:
+          defaultMode: 256
+          sources:
+          - secret:
+              items:
+              - key: ca.crt
+                path: ca.crt
+              - key: tls.crt
+                path: client.root.crt
+              - key: tls.key
+                path: client.root.key
+              name: cockroachdb.client.root
+      - configMap:
+          name: cockroachdb-scripts
+        name: cockroachdb-scripts
+---
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   name: client
 spec:
   commonName: root
@@ -471,6 +584,9 @@ spec:
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   name: cockroachdb-ca
 spec:
   commonName: cockroachdb-ca
@@ -486,6 +602,25 @@ spec:
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
+  name: example-database-cockroach-client
+spec:
+  commonName: example_user
+  issuerRef:
+    kind: Issuer
+    name: cockroachdb-ca-issuer
+  secretName: cockroachdb.client.example-user
+  usages:
+  - client auth
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   name: node
 spec:
   commonName: node
@@ -496,6 +631,7 @@ spec:
   - cockroachdb-0.cockroachdb.<your namespace here>.svc.cluster.local
   - cockroachdb-1.cockroachdb.<your namespace here>.svc.cluster.local
   - cockroachdb-2.cockroachdb.<your namespace here>.svc.cluster.local
+  - cockroachdb-proxy
   issuerRef:
     kind: Issuer
     name: cockroachdb-ca-issuer
@@ -507,6 +643,9 @@ spec:
 apiVersion: cert-manager.io/v1
 kind: Issuer
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   name: cockroachdb-ca-issuer
 spec:
   ca:
@@ -515,6 +654,9 @@ spec:
 apiVersion: cert-manager.io/v1
 kind: Issuer
 metadata:
+  annotations:
+    app.uw.systems/description: Cockroachdb instance used to store ...
+    app.uw.systems/tier: tier_1
   name: cockroachdb-selfsigned-issuer
 spec:
   selfSigned: {}
diff --git a/built-manifests/cockroachdb/manifests-cert-manager/db-init/manifests.yaml b/built-manifests/cockroachdb/manifests-cert-manager/db-init/manifests.yaml
new file mode 100644
index 0000000..21999bc
--- /dev/null
+++ b/built-manifests/cockroachdb/manifests-cert-manager/db-init/manifests.yaml
@@ -0,0 +1,78 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  labels:
+    app: cockroach-db-init
+    version: v1
+  name: cockroach-db-init
+spec:
+  template:
+    spec:
+      containers:
+      - command:
+        - /bin/bash
+        - /opt/scripts/user-schema-bootstrap.sh
+        env:
+        - name: COCKROACH_HOST
+          valueFrom:
+            configMapKeyRef:
+              key: cockroach.host
+              name: cockroach
+        - name: COCKROACH_PORT
+          valueFrom:
+            configMapKeyRef:
+              key: cockroach.port
+              name: cockroach
+        - name: COCKROACH_CERTS_DIR
+          value: /cockroach-certs/
+        - name: DB_NAME
+          value: default
+        - name: DB_USER
+          value: default
+        image: cockroachdb/cockroach:v23.2.2
+        imagePullPolicy: IfNotPresent
+        name: db-init
+        resources:
+          limits:
+            cpu: 1
+            memory: 512Mi
+          requests:
+            cpu: 0
+            memory: 128Mi
+        volumeMounts:
+        - mountPath: /cockroach-certs
+          name: client-certs
+        - mountPath: /opt/scripts
+          name: cockroachdb-scripts
+          readOnly: true
+      restartPolicy: OnFailure
+      volumes:
+      - name: client-certs
+        projected:
+          defaultMode: 256
+          sources:
+          - secret:
+              items:
+              - key: ca.crt
+                path: ca.crt
+              - key: tls.crt
+                path: client.root.crt
+              - key: tls.key
+                path: client.root.key
+              name: cockroachdb.client.root
+      - configMap:
+          name: cockroachdb-scripts
+        name: cockroachdb-scripts
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: cockroach-client
+spec:
+  commonName: userName
+  issuerRef:
+    kind: Issuer
+    name: cockroachdb-ca-issuer
+  secretName: cockroachdb.client.userName
+  usages:
+  - client auth
diff --git a/root-manifests/cockroachdb/manifests-cert-manager/manifests.yaml b/built-manifests/cockroachdb/manifests-cert-manager/manifests.yaml
index 4d1ba3e..d0e7ea6 100644
--- a/root-manifests/cockroachdb/manifests-cert-manager/manifests.yaml
+++ b/built-manifests/cockroachdb/manifests-cert-manager/manifests.yaml
@@ -27,28 +27,16 @@ data:
           WITH SCHEDULE OPTIONS ignore_existing_backups;
       COMMIT;
     EOF
-  user-schema-bootstrap.sh: |
-    #!/bin/bash
-    set -e
-
-    if [ -z "$DB_USER" ]; then
-        echo "DB_USER envvar is not set"
-        exit 1
-    fi
-
-    if [ -z "$DB_NAME" ]; then
-        echo "DB_NAME envvar not set"
-        exit 1
-    fi
-
-    SQL_CMD="/cockroach/cockroach -d system sql" > /dev/null
-
-    $SQL_CMD << EOF
-      CREATE USER IF NOT EXISTS $DB_USER;
-      CREATE DATABASE IF NOT EXISTS $DB_NAME;
-      GRANT ALL ON DATABASE $DB_NAME TO $DB_USER;
-      GRANT ALL ON TABLE $DB_NAME.* TO $DB_USER;
-    EOF
+  user-schema-bootstrap.sh: "#!/bin/bash\nset -e\n\nif [ -z \"$DB_USER\" ]; then\n
+    \   echo \"DB_USER envvar is not set\"\n    exit 1\nfi\n\nif [ -z \"$DB_NAME\"
+    ]; then\n    echo \"DB_NAME envvar not set\"\n    exit 1\nfi\n\nSQL_CMD=\"/cockroach/cockroach
+    -d system sql\" > /dev/null\n\n# try creating user, database, grant database permissions
+    to user\n$SQL_CMD << EOF\n  CREATE USER IF NOT EXISTS $DB_USER;\n  CREATE DATABASE
+    IF NOT EXISTS $DB_NAME;\n  GRANT ALL ON DATABASE $DB_NAME TO $DB_USER;\nEOF\n\n#
+    If database already exists and contains tables, grant access to them for user.
+    \n# This command will fail for freshly initialized databases.\n$SQL_CMD --execute
+    \"GRANT ALL ON TABLE $DB_NAME.* TO $DB_USER;\" 2>1 || echo \"no tables in database
+    yet\"\n"
 kind: ConfigMap
 metadata:
   name: cockroachdb-scripts
@@ -390,14 +378,11 @@ spec:
       - command:
         - /bin/bash
         - -c
-        - /cockroach/cockroach init --certs-dir=/cockroach/cockroach-certs --port=26357
-          2>&1 | grep 'initialized'
+        - /cockroach/cockroach init --certs-dir=/cockroach/cockroach-certs --host=$(COCKROACH_INIT_HOST)
+          --port=26357 2>&1 | grep 'initialized'
         env:
-        - name: COCKROACH_HOST
-          valueFrom:
-            configMapKeyRef:
-              key: cockroach.host
-              name: cockroach
+        - name: COCKROACH_INIT_HOST
+          value: cockroachdb-0.cockroachdb
         image: cockroachdb/cockroach:v23.2.2
         imagePullPolicy: IfNotPresent
         name: cluster-init

=============================================
k8s objects: 6 to add, 0 to destroy
22 changed hunks

sbuliarca · 2024-07-18T14:09:46Z

cockroachdb/example/cert-manager/db-init-manifests/example-database/kustomization.yaml

+# similar kustomize file should exist for every database/ user pair
+
+resources:
+  - github.com/utilitywarehouse/shared-kustomize-bases//cockroachdb/manifests-cert-manager/db-init?ref=DENA-603


We'll need to update this ref once merged and released, right?

MarcinGinszt added 6 commits July 4, 2024 16:05

add init job with example how to use it, enhance script and opslevel …

511fd97

…annotations

restoring init job host parameter to find reason for faling db init

f8e5d83

deploy db-init script with cockroachdb

bd24b8e

move db init to separate directory

00c5b49

replace image for db init container

6b08163

add resource to db init, add mock version to example

989ebb9

MarcinGinszt requested a review from a team as a code owner July 11, 2024 08:45

remove tag pinning from examples, as it breaks the cicd

0a13af6

MarcinGinszt changed the title ~~[DENA-603] Example of init database job~~ feat: [DENA-603] Example of init database job Jul 11, 2024

reference current branch to unlock CICD

42ab67b

MarcinGinszt force-pushed the DENA-603 branch from 9dbee7c to 42ab67b Compare July 11, 2024 08:52

matthewhughes-uw reviewed Jul 11, 2024

View reviewed changes

MarcinGinszt added 5 commits July 17, 2024 13:49

remove common name, add proxy url to dns

1aa4046

separate name for db-init job

c3a3164

change client cert name

967f5f1

use strategic merge to create new database init job

c419375

nicer logs in the script

0902cde

MarcinGinszt requested a review from matthewhughes-uw July 17, 2024 13:31

sbuliarca reviewed Jul 18, 2024

View reviewed changes

MarcinGinszt requested a review from sbuliarca July 22, 2024 06:37

sbuliarca approved these changes Jul 22, 2024

View reviewed changes

MarcinGinszt merged commit 8bac405 into main Jul 22, 2024
2 checks passed

MarcinGinszt deleted the DENA-603 branch July 22, 2024 07:08

github-actions bot mentioned this pull request Jul 22, 2024

chore(main): release cockroachdb 1.1.0 #54

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: [DENA-603] Example of init database job #50

feat: [DENA-603] Example of init database job #50

MarcinGinszt commented Jul 11, 2024

linear bot commented Jul 11, 2024

matthewhughes-uw Jul 11, 2024

MarcinGinszt Jul 17, 2024

matthewhughes-uw Jul 11, 2024

MarcinGinszt Jul 17, 2024

matthewhughes-uw Jul 11, 2024

MarcinGinszt Jul 17, 2024

matthewhughes-uw Jul 11, 2024

MarcinGinszt Jul 17, 2024

sbuliarca Jul 18, 2024

MarcinGinszt Jul 22, 2024

github-actions bot commented Jul 17, 2024

sbuliarca Jul 18, 2024

feat: [DENA-603] Example of init database job #50

feat: [DENA-603] Example of init database job #50

Conversation

MarcinGinszt commented Jul 11, 2024

linear bot commented Jul 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 17, 2024

Choose a reason for hiding this comment