Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs for wal-g support for local storage #429

Merged
merged 3 commits into from
May 4, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions docs/examples/postgres/initialization/replay-postgres-local.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: replay-postgres
namespace: demo
spec:
version: "10.2-v3"
iamrz1 marked this conversation as resolved.
Show resolved Hide resolved
replicas: 2
databaseSecret:
secretName: wal-postgres-auth
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
init:
postgresWAL:
local:
mountPath: /tmp/replay-postgres
subPath: wal-postgres-0
persistentVolumeClaim:
claimName: pgbackup
38 changes: 38 additions & 0 deletions docs/examples/postgres/snapshot/wal-postgres-local.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
namespace: demo
name: pgbackup
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: standard
---

apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: postgres-db
namespace: demo
spec:
version: "10.2-v3"
iamrz1 marked this conversation as resolved.
Show resolved Hide resolved
storageType: Durable
replicas: 2
updateStrategy:
type: RollingUpdate
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
archiver:
storage:
local:
mountPath: /data
persistentVolumeClaim:
claimName: pgbackup
215 changes: 215 additions & 0 deletions docs/guides/postgres/initialization/replay_from_local.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
---
title: Initialize Postgres from WAL (Local Storage)
menu:
docs_0.11.0:
identifier: pg-wal-source-initialization-local
name: From WAL(Local Storage)
parent: pg-initialization-postgres
weight: 50
menu_name: docs_0.11.0
section_menu_id: guides
---

> New to KubeDB? Please start [here](/docs/concepts/README.md).
> Don't know how to take continuous backup? Check this [tutorial](/docs/guides/postgres/snapshot/continuous_archiving.md) on Continuous Archiving.

# PostgreSQL Initialization from Local Storage

**WAL-G** is used to handle replay, and restoration mechanism. Please refer to [Initialization from WAL files in KubeDB](/docs/guides/postgres/initialization/wal_source.md) to know more about it.

## Before You Begin

At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster.
If you do not already have a cluster, you can create one by using [minikube](https://github.com/kubernetes/minikube).

Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/install.md).

To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial.

```console
$ kubectl create ns demo
namespace/demo created
```

## Prepare WAL Archive

We need a WAL archive to perform initialization. If you don't have a WAL archive ready, create one by following the tutorial [here](/docs/guides/postgres/snapshot/continuous_archiving.md).

Let's populate the database so that we can verify that the initialized database has the same data. We will `exec` into the database pod and use `psql` command-line tool to create a table.

At first, find out the primary replica using the following command,

```console
$ kubectl get pods -n demo --selector="kubedb.com/name=wal-postgres","kubedb.com/role=primary"
NAME READY STATUS RESTARTS AGE
wal-postgres-0 1/1 Running 0 8m
```

Now, let's `exec` into the pod and create a table,

```console
$ kubectl exec -it -n demo wal-postgres-0 sh
# login as "postgres" superuser.
/ # psql -U postgres
psql (11.1)
Type "help" for help.

# list available databases
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)

# connect to "postgres" database
postgres=# \c postgres
You are now connected to database "postgres" as user "postgres".

# create a table
postgres=# CREATE TABLE COMPANY( NAME TEXT NOT NULL, EMPLOYEE INT NOT NULL);
CREATE TABLE

# list tables
postgres=# \d
List of relations
Schema | Name | Type | Owner
--------+---------+-------+----------
public | company | table | postgres

# quit from the database
postgres=# \q

# exit from the pod
/ # exit
```

Now, we are ready to proceed for rest of the tutorial.

> Note: YAML files used in this tutorial are stored in [docs/examples/postgres](https://github.com/kubedb/cli/tree/master/docs/examples/postgres) folder in GitHub repository [kubedb/cli](https://github.com/kubedb/cli).

## Create Postgres with WAL source

User can initialize a new database from this archived WAL files. We have to specify the archive backend in the `spec.init.postgresWAL` field of Postgres object.

The YAML file in this tutorial creates a Postgres object using WAL files from Local Storage.

```yaml
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: replay-postgres
namespace: demo
spec:
version: "10.2-v3"
replicas: 2
databaseSecret:
secretName: wal-postgres-auth
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
init:
postgresWAL:
local:
mountPath: /tmp/replay-postgres
subPath: wal-postgres-0
persistentVolumeClaim:
claimName: pgbackup

```

Here,

- `spec.init.postgresWAL` specifies storage information that will be used by `WAL-G`
- `spec.init.postgresWAL.local` points to Local Storage configuration.
- `spec.init.postgresWAL.mountPath` points to a path where Local Storage containing archived WAL data will be mounted.
- `spec.init.postgresWAL.subPath` points to the path inside the mountPath where archived WAL data can be found. It is usually the name of the archiver's primary pod.
- `spec.init.postgresWAL.persistentVolumeClaim.claimName` points to the same local volume that was used to store WAL files.

**wal-g** receives archived WAL data from a directory `{mountPath}/{subPath}` inside the pod.

> Note: Postgres `replay-postgres` must have same superuser credentials as archived Postgres. In our case, it is `wal-postgres`.

Now, let's create the Postgres object that's YAML has shown above,

```console
$ kubectl create -f https://github.com/raw/kubedb/cli/0.11.0/docs/examples/postgres/initialization/replay-postgres-local.yaml
postgres.kubedb.com/replay-postgres created
```

This will create a new database and will initialize the database from the archived WAL files.

## Verify Initialization

Let's verify that the new database has been initialized successfully from the WAL archive. It must contain the table we have created for `wal-postgres` database.

We will `exec` into new database pod and use `psql` command-line tool to list tables of `postgres` database.

```console
$ kubectl exec -it -n demo replay-postgres-0 sh
# login as "postgres" superuser
/ # psql -U postgres
psql (11.1)
Type "help" for help.

# list available databases
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)

# connect to "postgres" database
postgres=# \c postgres
You are now connected to database "postgres" as user "postgres".

# list tables
postgres=# \d
List of relations
Schema | Name | Type | Owner
--------+---------+-------+----------
public | company | table | postgres
(1 row)

# quit from the database
postgres=# \q

# exit from pod
/ # exit
```

So, we can see that our new database `replay-postgres` has been initialized successfully and contains the data we had inserted into `wal-postgres`.

## Cleaning up

To cleanup the Kubernetes resources created by this tutorial, run:

```console
kubectl patch -n demo pg/replay-postgres -p '{"spec":{"terminationPolicy":"WipeOut"}}' --type="merge"
kubectl delete -n demo pg/replay-postgres

kubectl delete ns demo
```

Also cleanup the resources created for `wal-postgres` following the guide [here](/docs/guides/postgres/snapshot/archiving_to_local.md#cleaning-up).

## Next Steps

- Learn about initializing [PostgreSQL with Script](/docs/guides/postgres/initialization/script_source.md).
- Monitor your PostgreSQL database with KubeDB using [built-in Prometheus](/docs/guides/postgres/monitoring/using-builtin-prometheus.md).
- Monitor your PostgreSQL database with KubeDB using [CoreOS Prometheus Operator](/docs/guides/postgres/monitoring/using-coreos-prometheus-operator.md).
- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).

Loading