Skip to content

Commit

Permalink
add backup api that can be run incrementally (#1116)
Browse files Browse the repository at this point in the history
This exposes the sqlite3 backup api as described at https://sqlite.org/backup.html.

This implementation draws on #883,
extending it to create a backup object that can be used in the background,
without leaving the database locked for an extended period of time.  This is
crucial for making backups of large live databases in a non-disruptive manner.
Example usage:

```
var db = new sqlite3.Database('live.db');
var backup = db.backup('backup.db');
...
// in event loop, move backup forward when we have time.
if (backup.idle) { backup.step(NPAGES); }
if (backup.completed) { /* success! backup made */  }
if (backup.failed)    { /* sadness! backup broke */ }
// do other work in event loop - fine to modify live.db
...
```

Here is how sqlite's backup api is exposed:

 * `sqlite3_backup_init`: This is implemented as `db.backup(filename, [callback])`
   or `db.backup(filename, destDbName, sourceDbName, filenameIsDest, [callback])`.
 * `sqlite3_backup_step`: This is implemented as `backup.step(pages, [callback])`.
 * `sqlite3_backup_finish`: This is implemented as `backup.finish([callback])`.
 * `sqlite3_backup_remaining`: This is implemented as a `backup.remaining` getter.
 * `sqlite3_backup_pagecount`: This is implemented as a `backup.pageCount` getter.

Some conveniences are added in the node api.

There are the following read-only properties:
 * `backup.completed` is set to `true` when the backup succeeeds.
 * `backup.failed` is set to `true` when the backup has a fatal error.
 * `backup.idle` is set to `true` when no operation is currently in progress or
   queued for the backup.
 * `backup.remaining` is an integer with the remaining number of pages after the
   last call to `backup.step` (-1 if `step` not yet called).
 * `backup.pageCount` is an integer with the total number of pages measured during
   the last call to `backup.step` (-1 if `step` not yet called).

There is the following writable property:
 * `backup.retryErrors`: an array of sqlite3 error codes that are treated as
   non-fatal - meaning, if they occur, backup.failed is not set, and the backup
   may continue.  By default, this is `[sqlite3.BUSY, sqlite3.LOCKED]`.

The `db.backup(filename, [callback])` shorthand is sufficient for making a
backup of a database opened by node-sqlite3.  If using attached or temporary
databases, or moving data in the opposite direction, the more complete
(but daunting) `db.backup(filename, destDbName, sourceDbName, filenameIsDest, [callback])`
signature is provided.

A backup will finish automatically when it succeeds or a fatal error
occurs, meaning it is not necessary to call `db.finish()`.
By default, SQLITE_LOCKED and SQLITE_BUSY errors are not treated as
failures, and the backup will continue if they occur.  The set of errors
that are tolerated can be controlled by setting `backup.retryErrors`.
To disable automatic finishing and stick strictly to sqlite's raw api,
set `backup.retryErrors` to `[]`.  In that case, it is necessary to call
`backup.finish()`.

In the same way as node-sqlite3 databases and statements, backup methods
can be called safely without callbacks, due to an internal call queue.  So
for example this naive code will correctly back up a db, if there are
no errors:
```
var backup = db.backup('backup.db');
backup.step(-1);
backup.finish();
```
  • Loading branch information
paulfitz authored and kewde committed Feb 21, 2019
1 parent 4a8b4bd commit 2bd051d
Show file tree
Hide file tree
Showing 8 changed files with 972 additions and 0 deletions.
1 change: 1 addition & 0 deletions binding.gyp
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
]
],
"sources": [
"src/backup.cc",
"src/database.cc",
"src/node_sqlite3.cc",
"src/statement.cc"
Expand Down
19 changes: 19 additions & 0 deletions lib/sqlite3.js
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,11 @@ sqlite3.cached = {

var Database = sqlite3.Database;
var Statement = sqlite3.Statement;
var Backup = sqlite3.Backup;

inherits(Database, EventEmitter);
inherits(Statement, EventEmitter);
inherits(Backup, EventEmitter);

// Database#prepare(sql, [bind1, bind2, ...], [callback])
Database.prototype.prepare = normalizeMethod(function(statement, params) {
Expand Down Expand Up @@ -99,6 +101,23 @@ Database.prototype.map = normalizeMethod(function(statement, params) {
return this;
});

// Database#backup(filename, [callback])
// Database#backup(filename, destName, sourceName, filenameIsDest, [callback])
Database.prototype.backup = function() {
var backup;
if (arguments.length <= 2) {
// By default, we write the main database out to the main database of the named file.
// This is the most likely use of the backup api.
backup = new Backup(this, arguments[0], 'main', 'main', true, arguments[1]);
} else {
// Otherwise, give the user full control over the sqlite3_backup_init arguments.
backup = new Backup(this, arguments[0], arguments[1], arguments[2], arguments[3], arguments[4]);
}
// Per the sqlite docs, exclude the following errors as non-fatal by default.
backup.retryErrors = [sqlite3.BUSY, sqlite3.LOCKED];
return backup;
};

Statement.prototype.map = function() {
var params = Array.prototype.slice.call(arguments);
var callback = params.pop();
Expand Down
Loading

0 comments on commit 2bd051d

Please sign in to comment.