Add Database abstraction and README update #39

ytsarev · 2022-10-25T17:15:28Z

Description of your changes

Implement Database abstraction.
Use https://cloud.google.com/sql/docs/postgres/configure-private-services-access and https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine#private-ip for configuring private VPC-based access without cloud SQL proxy sidecar
Make XRD and Composition naming consistent with the rest of the platform refs
Update README with a consistent layout.

Fixes #26

I have:

Read and followed Upbound's contribution process.
Run make reviewable to ensure this PR is ready for review.
Added backport release-x.y labels to auto-backport this PR, as appropriate.

How has this code been tested

k apply -f examples/cluster-claim.yaml
k apply -f examples/postgres-claim.yaml

 k get claim
NAME                                                                READY   CONNECTION-SECRET          AGE
postgresqlinstance.gcp.platformref.upbound.io/platform-ref-gcp-db   True    platform-ref-gcp-db-conn   129m

NAME                                                          READY   CONNECTION-SECRET   AGE
cluster.gcp.platformref.upbound.io/platform-ref-gcp-cluster   True    cluster-conn        130m

 k view-secret platform-ref-gcp-db-conn
Multiple sub keys found. Specify another argument, one of:
-> privateIP
-> serverCACertificateCert

To test the cluster-database private connectivity end-to-end.

#On the target gke cluster

cat no-sidecar-psql.yaml
apiVersion: v1
kind: Pod
metadata:
  name: see-db
  namespace: default
spec:
  containers:
  - name: see-db
    image: postgres:12
    command: ['psql']
    args: ['-c', 'SELECT current_database();']
    env:
    - name: PGDATABASE
      value: postgres
    - name: PGHOST
      value: 10.100.0.3
    - name: PGUSER
      value: platform-ref-gcp-db-r26cb-5qjrf
    - name: PGPASSWORD
      value: <password

k apply -f no-sidecar-psql.yaml

k logs -f see-db
 current_database
------------------
 postgres
(1 row)

Signed-off-by: Yury Tsarev <yury@upbound.io>

…uster * Avoid the need for cloud sql proxy sidecar for workload deployments Signed-off-by: Yury Tsarev <yury@upbound.io>

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev · 2022-10-25T17:16:41Z

@stevendborrelli please use upbound/platform-ref-gcp-staging:v0.3.0 for the testing.

ytsarev · 2022-10-25T18:25:53Z

/test-e2e

ytsarev · 2022-10-25T20:06:15Z

/test-e2e

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev · 2022-10-25T20:10:34Z

/test-e2e

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev · 2022-10-25T21:22:22Z

/test-e2e

ytsarev · 2022-10-26T07:34:56Z

/test-e2e

Signed-off-by: Yury Tsarev <yury@upbound.io>

* Use new `pre-delete` hooks * Explicitly specify namespace for the database claim Signed-off-by: Yury Tsarev <yury@upbound.io>

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev · 2022-10-26T19:57:22Z

/test-e2e

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev · 2022-10-26T23:37:29Z

/test-e2e

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev · 2022-10-27T07:55:10Z

/test-e2e

ytsarev · 2022-10-27T09:23:13Z

/test-e2e

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev · 2022-10-27T10:07:42Z

/test-e2e

* This way DatabaseInstance deletion is not dependend on Cluster and we need no complex ordering * Drawback: stable network name in Cluster composition * GKE tweaks to spare some resources * Isolated simple test-hooks Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev · 2022-10-27T15:18:24Z

/test-e2e

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev · 2022-10-27T20:07:22Z

/test-e2e

ytsarev · 2022-10-27T21:44:06Z

/test-e2e

stevendborrelli

LGTM.

All claims became ready:

kubectl get claim,composite        
NAME                                                          READY   CONNECTION-SECRET   AGE
cluster.gcp.platformref.upbound.io/platform-ref-gcp-cluster   True    cluster-conn        82m

NAME                                                                READY   CONNECTION-SECRET          AGE
postgresqlinstance.gcp.platformref.upbound.io/platform-ref-gcp-db   True    platform-ref-gcp-db-conn   80m

NAME                                                                   READY   COMPOSITION                       AGE
xgke.gcp.platformref.upbound.io/platform-ref-gcp-cluster-xftvn-cp6lf   True    xgke.gcp.platformref.upbound.io   81m

NAME                                                                       READY   COMPOSITION                            AGE
xnetwork.gcp.platformref.upbound.io/platform-ref-gcp-cluster-xftvn-tng6l   True    xnetworks.gcp.platformref.upbound.io   81m

NAME                                                                 READY   COMPOSITION                            AGE
xcluster.gcp.platformref.upbound.io/platform-ref-gcp-cluster-xftvn   True    xclusters.gcp.platformref.upbound.io   81m

NAME                                                                       READY   COMPOSITION                                       AGE
xpostgresqlinstance.gcp.platformref.upbound.io/platform-ref-gcp-db-mf49v   True    xpostgresqlinstances.gcp.platformref.upbound.io   80m

NAME                                                                        READY   COMPOSITION                            AGE
xservices.gcp.platformref.upbound.io/platform-ref-gcp-cluster-xftvn-8qg2l   True    xservices.gcp.platformref.upbound.io   81m

examples/postgres-claim.yaml

examples/configuration.yaml

stevendborrelli · 2022-10-28T13:28:16Z

README.md


 ```console
-kubectl -n upbound-system get cluster
+kubectl delete -f examples/cluster-claim.yaml,examples/postgres-claim.yaml


I got caught with hanging resources that required manual deletion (the helm release and SQL user). Perhaps we should document the scripts.

Thanks. I've added note on ordered delete 69bb0f7

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev added 4 commits October 25, 2022 19:02

Database abstraction layout

f45b5cb

Signed-off-by: Yury Tsarev <yury@upbound.io>

Configure Private SQL Instance directly accessible from within GKE cl…

8748f68

…uster * Avoid the need for cloud sql proxy sidecar for workload deployments Signed-off-by: Yury Tsarev <yury@upbound.io>

Revamp XRDs/Compositions naming for consistency

c621003

Signed-off-by: Yury Tsarev <yury@upbound.io>

Doc update

e048649

Signed-off-by: Yury Tsarev <yury@upbound.io>

pull-request-size bot added the size/XL label Oct 25, 2022

ytsarev requested a review from stevendborrelli October 25, 2022 17:15

Include database into e2e tests

c959180

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev force-pushed the database branch from bd53306 to c959180 Compare October 25, 2022 20:10

Delete user.sql in test hook to avoid the orphan state

ef3ddf8

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev mentioned this pull request Oct 26, 2022

user.sql.gcp.upbound.io should populate connection secret crossplane-contrib/provider-upjet-gcp#37

Closed

Add note on servicenetworking.googleapis.com global binding

1fd42b6

Signed-off-by: Yury Tsarev <yury@upbound.io>

turkenh mentioned this pull request Oct 26, 2022

Handle secrets properly upbound/official-providers-ci#37

Merged

3 tasks

ytsarev added 2 commits October 26, 2022 17:19

Update examples for uptest

aa4592f

* Use new `pre-delete` hooks * Explicitly specify namespace for the database claim Signed-off-by: Yury Tsarev <yury@upbound.io>

Deleter ordering, new uptest with secret handling

939493c

Signed-off-by: Yury Tsarev <yury@upbound.io>

Additional testhook arrangement

29abef5

Signed-off-by: Yury Tsarev <yury@upbound.io>

Increase default timeout as the deployment takes time

d2d414e

Signed-off-by: Yury Tsarev <yury@upbound.io>

Add doc covering required roles for SA

3b86244

Signed-off-by: Yury Tsarev <yury@upbound.io>

More precise creds gen link

0da3d64

Signed-off-by: Yury Tsarev <yury@upbound.io>

This was referenced Oct 28, 2022

uptest e2e fails when namespaced is not specified for the namespaced Claim upbound/official-providers-ci#38

Open

New GKE cluster ends up in repairing state #40

Closed

stevendborrelli approved these changes Oct 28, 2022

View reviewed changes

examples/postgres-claim.yaml Show resolved Hide resolved

examples/configuration.yaml Show resolved Hide resolved

stevendborrelli reviewed Oct 28, 2022

View reviewed changes

Add note on ordered deletion

69bb0f7

Signed-off-by: Yury Tsarev <yury@upbound.io>

ytsarev force-pushed the database branch from f8ecba9 to 69bb0f7 Compare October 31, 2022 08:16

ytsarev merged commit bba9e81 into upbound:main Oct 31, 2022

ytsarev deleted the database branch October 31, 2022 08:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Database abstraction and README update #39

Add Database abstraction and README update #39

ytsarev commented Oct 25, 2022

ytsarev commented Oct 25, 2022

ytsarev commented Oct 25, 2022

ytsarev commented Oct 25, 2022

ytsarev commented Oct 25, 2022

ytsarev commented Oct 25, 2022

ytsarev commented Oct 26, 2022

ytsarev commented Oct 26, 2022

ytsarev commented Oct 26, 2022

ytsarev commented Oct 27, 2022

ytsarev commented Oct 27, 2022

ytsarev commented Oct 27, 2022

ytsarev commented Oct 27, 2022

ytsarev commented Oct 27, 2022

ytsarev commented Oct 27, 2022

stevendborrelli left a comment

stevendborrelli Oct 28, 2022

ytsarev Oct 31, 2022

Add Database abstraction and README update #39

Add Database abstraction and README update #39

Conversation

ytsarev commented Oct 25, 2022

Description of your changes

How has this code been tested

ytsarev commented Oct 25, 2022

ytsarev commented Oct 25, 2022

ytsarev commented Oct 25, 2022

ytsarev commented Oct 25, 2022

ytsarev commented Oct 25, 2022

ytsarev commented Oct 26, 2022

ytsarev commented Oct 26, 2022

ytsarev commented Oct 26, 2022

ytsarev commented Oct 27, 2022

ytsarev commented Oct 27, 2022

ytsarev commented Oct 27, 2022

ytsarev commented Oct 27, 2022

ytsarev commented Oct 27, 2022

ytsarev commented Oct 27, 2022

stevendborrelli left a comment

Choose a reason for hiding this comment

stevendborrelli Oct 28, 2022

Choose a reason for hiding this comment

ytsarev Oct 31, 2022

Choose a reason for hiding this comment