Docs for MinIO support in wal-g

docs/examples/postgres/initialization/recovered-postgres.yaml

@@ -4,7 +4,7 @@ metadata:
   name: recovered-postgres
   namespace: demo
 spec:
-  version: "9.6-v2"
+  version: "9.6-v4"
   databaseSecret:
     secretName: script-postgres-auth
   storage:

docs/examples/postgres/initialization/replay-postgres-minio.yaml

@@ -0,0 +1,24 @@
+apiVersion: kubedb.com/v1alpha1
+kind: Postgres
+metadata:
+  name: replay-postgres
+  namespace: demo
+spec:
+  version: "11.1-v2"
+  replicas: 2
+  databaseSecret:
+    secretName: wal-postgres-minio-auth
+  storage:
+    storageClassName: "standard"
+    accessModes:
+      - ReadWriteOnce
+    resources:
+      requests:
+        storage: 1Gi
+  init:
+    postgresWAL:
+      storageSecretName: s3-secret
+      s3:
+        bucket: kubedb
+        prefix: 'kubedb/demo/wal-postgres-minio/archive'
+        endpoint: https://minio-service.storage.svc:443/

docs/examples/postgres/snapshot/wal-postgres-minio.yaml

@@ -0,0 +1,23 @@
+apiVersion: kubedb.com/v1alpha1
+kind: Postgres
+metadata:
+  name: wal-postgres-minio
+  namespace: demo
+spec:
+  version: "9.6.7-v3"
+  replicas: 2
+  updateStrategy:
+    type: RollingUpdate
+  storage:
+    storageClassName: "standard"
+    accessModes:
+      - ReadWriteOnce
+    resources:
+      requests:
+        storage: 1Gi
+  archiver:
+    storage:
+      storageSecretName: s3-secret
+      s3:
+        bucket: kubedb
+        endpoint: https://minio-service.demo.svc:443/

docs/guides/postgres/initialization/point-in-time-recovery.md

@@ -77,7 +77,7 @@ metadata:
   name: pg-original
   namespace: demo
 spec:
-  version: "10.2-v2"
+  version: "10.2-v4"
   replicas: 1
   terminationPolicy: Delete
   storage:
@@ -197,7 +197,7 @@ metadata:
   name: pitr-1
   namespace: demo
 spec:
-  version: "10.2-v2"
+  version: "10.2-v4"
   replicas: 1
   terminationPolicy: Delete
   storage:
@@ -264,7 +264,7 @@ metadata:
   name: pitr-2
   namespace: demo
 spec:
-  version: "10.2-v2"
+  version: "10.2-v4"
   replicas: 1
   terminationPolicy: Delete
   storage:

docs/guides/postgres/initialization/replay_from_minio.md

@@ -0,0 +1,221 @@
+---
+title: Initialize Postgres from MinIO
+menu:
+  docs_0.11.0:
+    identifier: pg-wal-source-initialization-minio
+    name: From WAL(MiniIO)
+    parent: pg-initialization-postgres
+    weight: 55
+menu_name: docs_0.11.0
+section_menu_id: guides
+---
+
+> New to KubeDB? Please start [here](/docs/concepts/README.md).
+> Don't know how to take continuous backup? Check this [tutorial](/docs/guides/postgres/snapshot/continuous_archiving.md) on Continuous Archiving.
+
+# PostgreSQL Initialization from MinIO
+
+**WAL-G** is used to handle replay, and restoration mechanism. Please refer to [Initialization from WAL files in KubeDB](/docs/guides/postgres/initialization/wal_source.md) to know more.
+
+## Before You Begin
+
+At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster.
+If you do not already have a cluster, you can create one by using [minikube](https://github.com/kubernetes/minikube).
+
+Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/install.md).
+
+To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial.
+
+```console
+$ kubectl create ns demo
+namespace/demo created
+```
+
+## Prepare WAL Archive
+
+We need a WAL archive to perform initialization. If you don't have a WAL archive ready, get started by following the tutorial [here](/docs/guides/postgres/snapshot/continuous_archiving.md).
+MinIO specific archiving guides can be found [here](/docs/guides/postgres/snapshot/archiving_to_minio.md).
+
+Let's populate the database so that we can verify that the initialized database has the same data. We will `exec` into the database pod and use `psql` command-line tool to create a table.
+
+At first, find out the primary replica using the following command,
+
+```console
+$ kubectl get pods -n demo --selector="kubedb.com/name=wal-postgres-minio","kubedb.com/role=primary"
+NAME             READY     STATUS    RESTARTS   AGE
+wal-postgres-minio-0   1/1       Running   0          8m
+```
+
+Now, let's `exec` into the pod and create a table,
+
+```console
+$ kubectl exec -it -n demo wal-postgres-minio-0 sh
+# login as "postgres" superuser.
+/ # psql -U postgres
+psql (11.1)
+Type "help" for help.
+
+# list available databases
+postgres=# \l
+                                 List of databases
+   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
+-----------+----------+----------+------------+------------+-----------------------
+ postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
+ template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
+           |          |          |            |            | postgres=CTc/postgres
+ template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
+           |          |          |            |            | postgres=CTc/postgres
+(3 rows)
+
+# connect to "postgres" database
+postgres=# \c postgres
+You are now connected to database "postgres" as user "postgres".
+
+# create a table
+postgres=# CREATE TABLE COMPANY( NAME TEXT NOT NULL, EMPLOYEE INT NOT NULL);
+CREATE TABLE
+
+# list tables
+postgres=# \d
+          List of relations
+ Schema |  Name   | Type  |  Owner
+--------+---------+-------+----------
+ public | company | table | postgres
+
+# quit from the database
+postgres=# \q
+
+# exit from the pod
+/ # exit
+```
+
+Now, we are ready to proceed to the rest of the tutorial.
+
+> Note: YAML files used in this tutorial are stored in [docs/examples/postgres](https://github.com/kubedb/cli/tree/master/docs/examples/postgres) folder in GitHub repository [kubedb/cli](https://github.com/kubedb/cli).
+
+## Create Postgres with WAL source
+
+User can initialize a new database from this archived WAL files. We have to specify the archive backend in the `spec.init.postgresWAL` field of Postgres object.
+
+The YAML file in this tutorial creates a Postgres object using WAL files from Amazon S3.
+
+```yaml
+apiVersion: kubedb.com/v1alpha1
+kind: Postgres
+metadata:
+  name: replay-postgres
+  namespace: demo
+spec:
+  version: "11.1-v2"
+  replicas: 2
+  databaseSecret:
+    secretName: wal-postgres-minio-auth
+  storage:
+    storageClassName: "standard"
+    accessModes:
+      - ReadWriteOnce
+    resources:
+      requests:
+        storage: 1Gi
+  init:
+    postgresWAL:
+      storageSecretName: s3-secret
+      s3:
+        bucket: kubedb
+        prefix: "kubedb/demo/wal-postgres-minio/archive"
+        endpoint: https://minio-service.storage.svc:443/
+```
+
+Here,
+
+- `spec.init.postgresWAL` specifies storage information that will be used by `WAL-G`
+  - `storageSecretName` points to the Secret containing the credentials for cloud storage destination.
+  - `s3` points to S3 storage configuration.
+  - `s3.bucket` points to the bucket name where archived WAL data is stored.
+  - `s3.prefix` points to the path of archived WAL data.
+  - `storage.s3.endpoint` points to the storage location where the bucket can be found.
+
+**Archiver Storage Secret**
+
+Storage Secret should contain credentials that were used to create the archive.
+
+**wal-g** receives archived WAL data from a directory inside the bucket called `/kubedb/{namespace}/{postgres-name}/archive/`.
+
+Here, `{namespace}` & `{postgres-name}` indicates Postgres object whose WAL archived data will be replayed.
+
+> Note: Postgres `replay-postgres` must have same superuser credentials as archived Postgres. In our case, it is `wal-postgres-minio`.
+
+Now, let's create the Postgres object that's YAML has shown above,
+
+```console
+$ kubectl create -f https://github.com/raw/kubedb/cli/0.11.0/docs/examples/postgres/initialization/replay-postgres-minio.yaml
+postgres.kubedb.com/replay-postgres created
+```
+
+This will create a new database and will initialize the database from the archived WAL files.
+
+## Verify Initialization
+
+Let's verify that the new database has been initialized successfully from the WAL archive. It must contain the table we have created for `wal-postgres` database.
+
+We will `exec` into new database pod and use `psql` command-line tool to list tables of `postgres` database.
+
+```console
+$ kubectl exec -it -n demo replay-postgres-0 sh
+# login as "postgres" superuser
+/ # psql -U postgres
+psql (11.1)
+Type "help" for help.
+
+# list available databases
+postgres=# \l
+                                 List of databases
+   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
+-----------+----------+----------+------------+------------+-----------------------
+ postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
+ template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
+           |          |          |            |            | postgres=CTc/postgres
+ template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
+           |          |          |            |            | postgres=CTc/postgres
+(3 rows)
+
+# connect to "postgres" database
+postgres=# \c postgres
+You are now connected to database "postgres" as user "postgres".
+
+# list tables
+postgres=# \d
+          List of relations
+ Schema |  Name   | Type  |  Owner
+--------+---------+-------+----------
+ public | company | table | postgres
+(1 row)
+
+# quit from the database
+postgres=# \q
+
+# exit from pod
+/ # exit
+```
+
+So, we can see that our new database `replay-postgres` has been initialized successfully and contains the data we had inserted into `wal-postgres-minio`.
+
+## Cleaning up
+
+To cleanup the Kubernetes resources created by this tutorial, run:
+
+```console
+kubectl patch -n demo pg/replay-postgres -p '{"spec":{"terminationPolicy":"WipeOut"}}' --type="merge"
+kubectl delete -n demo pg/replay-postgres
+
+kubectl delete ns demo
+```
+
+Also cleanup the resources created for `wal-postgres-minio` following the guide [here](/docs/guides/postgres/snapshot/archiving_to_minio.md#cleaning-up).
+
+## Next Steps
+
+- Learn about initializing [PostgreSQL with Script](/docs/guides/postgres/initialization/script_source.md).
+- Monitor your PostgreSQL database with KubeDB using [built-in Prometheus](/docs/guides/postgres/monitoring/using-builtin-prometheus.md).
+- Monitor your PostgreSQL database with KubeDB using [CoreOS Prometheus Operator](/docs/guides/postgres/monitoring/using-coreos-prometheus-operator.md).
+- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).