Transactional, distributed backups

[mbautin]

Jira Link: DB-2323
This is a master issue tracking transactional, distributed, point-in-time backups. This feature should enable the following:

• Take a transactional backup of an entire database, or multiple tables
• It should include all transactions as of some timestamp
• A transaction should either be fully included or excluded (depending on the timestamp)
• Each node would keep a backup of the subset of the total data
• A snapshot from one cluster can be restored to another cluster possibly with a different number of nodes

NOTE: A backup refers to an in-cluster snapshot which can subsequently get copied out to an external location.

Prerequisites

✅ Design Doc: distributed backups and restore

Phase 1 - non-transactional YCQL tables

✅ Enable instantaneous checkpointing of DocDB data files on each tablet
✅ Ability to initiate backup of one table from YB-Master
✅ Ability to restore one table from a backup (into the same or remote cluster)
✅ Include schemas for YCQL backups

Yugabyte Platform enhancements

✅ [Platform] Backup an entire keyspace in YCQL
✅ [Platform] Nightly backups in a schedule

Phase 2 - Transactional backups - YSQL and YCQL

✅ Checkpoint an entire table at a consistent point in time #1032
✅ Consistent backup of YSQL database (target v2.2) #1139
⬜️ Use backup / restore roles for doing backup / restore #4798
✅ Backup individual tables along with indexes for YCQL
✅ Backup entire database/schema for YSQL

Phase 3 - PITR, incremental backups, cluster backup

⬜️ Ability to perform point in time restores #1820
⬜️ Incremental backups
⬜️ Backup entire YugabyteDB cluster: This will backup all databases as well as roles, permissions and other cluster metadata

Usability improvements

⬜️ Document backups and restore for DB and platform #2349 #2757 #2926 #3241
⬜️ Enable YSQL backups through ysqlsh shell
⬜️ Ability to perform a cluster-wide YCQL backup using ybadmin or ycqlsh #2350

Yugabyte Platform enhancements

⬜️ [Platform] Purge older backups automatically from backup targets