From cb239cbb27943ad58c304d85ee9b61ba917af79c Mon Sep 17 00:00:00 2001 From: Dan Connolly Date: Fri, 25 Jun 2021 19:11:55 -0500 Subject: [PATCH] feat: demand-paged vats are reloaded from heap snapshots (#2848) This enhances SwingSet to have a "Vat Warehouse" which limits the number of "paged-in" vats to some maximum (currently 50). The idea is to conserve system RAM by allowing idle vats to remain "paged-out", which consumes only space on disk, until someone sends a message to them. The vat is then paged in, by creating a new xsnap process and reloading the necessary vat state. This reload process is greatly accelerated by loading a heap snapshot, if one is available. We only need to replay the suffix of the transcript that was recorded after the snapshot was taken, rather than the full (huge) transcript. Heap snapshots are stored in a new swingstore component named the "stream store". For each vat, the warehouse saves a heap snapshot after a configurable number of deliveries (default 200). In addition, it saves an initial snapshot after just a few deliveries (default 2), because all contracts vats start out with a large delivery that provides the contract bundle to evaluate. By taking a snapshot quickly, we can avoid the time needed to re-evaluate that large bundle on almost all process restarts. This algorithm is a best guess: we'll refine it as we gather more data about the tradeoff between work now (the time it takes to create and write a snapshot), the storage space consumed by those snapshots, and work later (replaying more transcript). We're estimating that a typical contract snapshot consumes about 300kB (compressed). closes #2273 closes #2277 refs #2422 refs #2138 (might close it) * refactor(replay): hoist handle declaration * chore(xsnap): clarify names of snapStore temp files for debugging * feat(swingset): initializeSwingset snapshots XS supervisor - solo: add xsnap, tmp dependencies - cosmic-swingset: declare dependencies on xsnap, tmp - snapshotSupervisor() - vk.saveSnapshot(), vk.getLastSnapshot() - test: mock vatKeeper needs getLastSnapshot() - test(snapstore): update snapshot hash - makeSnapstore in solo, cosmic-swingset - chore(solo): create xs-snapshots directory - more getVatKeeper -> provideVatKeeper - startPos arg for replayTransript() - typecheck shows vatAdminRootKref could be missing - test pre-SES snapshot size - hoist snapSize to test title - clarify SES vs. pre-SES XS workers - factor bootWorker out of bootSESWorker - hoist Kb, relativeSize for sharing between tests misc: - WIP: restore from snapshot - hard-code remote style fix(swingset): don't leak xs-worker in initializeSwingset When taking a snapshot of the supervisor in initializeSwingset, we neglected to `.close()` it. Lack of a name hindered diagnosis, so let's fix that while we're at it. * feat(swingset): save snapshot periodically after deliveries - vk.saveSnapShot() handles snapshotInterval - annotate type of kvStore in makeVatKeeper - move getLastSnapshot up for earlier use - refactor: rename snapshotDetail to lastSnapshot - factor out getTranscriptEnd - vatWarehouse.maybeSaveSnapshot() - saveSnapshot: - don't require snapStore - fix startPos type - provide snapstore to vatKeeper via kernelKeeper - buildKernel: get snapstore out of hostStorage - chore: don't try to snapshot a terminated vat * feat(swingset): load vats from snapshots - don't `setBundle` when loading from snapshot - provide startPos to replayTranscript() - test reloading a vat * refactor(vatWarehouse): factor out, test LRU logic * fix(vat-warehouse): remove vatID from LRU when evicting * chore(vatKeeper): prune debug logging in saveSnapshot (FIXUP) * feat(swingset): log bringing vats online (esp from snapshot) - manager.replayTranscript returns number of entries replayed * chore: resove "skip crank buffering?" issue after discussion with CM: maybeSaveSnapshot() happens before commitCrank() so nothing special needed here * chore: prune makeSnapshot arg from evict() Not only is this option not implemented now, but CM's analysis shows that adding it would likely be harmful. * test(swingset): teardown snap-store * chore(swingset): initial sketch of snapshot reload test * refactor: let itemCount be not-optional in StreamPosition * feat: snapshot early then infrequently - refactor: move snapshot decision up from vk.saveSnapshot() up to vw.maybeSaveSnapshot * test: provide getLastSnapshot to mock vatKeeper * chore: vattp: turn off managerType local work-around * chore: vat-warehouse: initial snapshot after 2 deliveries integration testing shows this is closer to ideal * chore: prune deterministic snapshot assertion oops. rebase problem. * chore: fix test-snapstore ld.asset rebase / merge problem?! * chore: never mind supervisorHash optimization With snapshotInitial at 2, there is little reason to snapshot after loading the supervisor bundles. The code doesn't carry its own weight. Plus, it seems to introduce a strange bug with marshal or something... ``` test/test-home.js:37 36: const { board } = E.get(home); 37: await t.throwsAsync( 38: () => E(board).getValue('148'), getting a value for a fake id throws Returned promise rejected with unexpected exception: Error { message: 'Remotable (a string) is already frozen', } ``` * docs(swingset): document lastSnapshot kernel DB key * refactor: capitalize makeSnapStore consistently * refactor: replayTranscript caller is responsible to getLastSnapshot * test(swingset): consistent vat-warehouse test naming * refactor(swingset): compute transcriptSnapshotStats in vatKeeper In an attempt to avoid reading the lastSnapshot DB key if the t.endPosition key was enough information to decide to take a snapshot, the vatWarehouse was peeking into the vatKeeper's business. Let's go with code clarity over (un-measured) performance. * chore: use harden, not freeze; clarify lru * chore: use distinct fixture directories to avoid collision The "temporary" snapstore directories used by two different tests began to overlap when the tests were moved into the same parent dir, and one test was deleting the directory while the other was still using it (as well as mingling files at runtime), causing an xsnap process to die with an IO error if the test were run in parallel. This changes the the two tests to use distinct directories. In the long run, we should either have them use `mktmp` to build a randomly-named known-unique directory, or establish a convention where tempdir names match the name of the test file and case using them, to avoid collisions as we add more tests. Co-authored-by: Brian Warner --- packages/SwingSet/src/controller.js | 54 +++--- packages/SwingSet/src/initializeSwingset.js | 5 - packages/SwingSet/src/kernel/kernel.js | 9 +- .../SwingSet/src/kernel/state/kernelKeeper.js | 10 +- .../SwingSet/src/kernel/state/vatKeeper.js | 58 ++++++- .../src/kernel/vatManager/manager-helper.js | 27 ++- .../vatManager/manager-subprocess-xsnap.js | 56 ++++-- .../src/kernel/vatManager/vat-warehouse.js | 159 +++++++++++++++--- packages/SwingSet/src/types.js | 6 +- packages/SwingSet/test/test-controller.js | 2 +- packages/SwingSet/test/test-gc-transcript.js | 1 + packages/SwingSet/test/test-xsnap-errors.js | 4 +- .../{warehouse => vat-warehouse}/bootstrap.js | 0 .../vat-warehouse/test-reload-snapshot.js | 83 +++++++++ .../test/vat-warehouse/test-warehouse.js | 137 +++++++++++++++ .../vat-target.js | 0 .../vat-warehouse/vat-warehouse-reload.js | 12 ++ .../SwingSet/test/warehouse/test-warehouse.js | 86 ---------- packages/cosmic-swingset/package.json | 4 +- packages/cosmic-swingset/src/launch-chain.js | 16 ++ packages/solo/package.json | 2 + packages/solo/src/start.js | 14 ++ .../src/simpleSwingStore.js | 2 +- packages/xsnap/src/index.js | 2 +- packages/xsnap/src/replay.js | 4 +- packages/xsnap/src/snapStore.js | 21 ++- packages/xsnap/test/test-snapstore.js | 79 ++++++--- 27 files changed, 636 insertions(+), 217 deletions(-) rename packages/SwingSet/test/{warehouse => vat-warehouse}/bootstrap.js (100%) create mode 100644 packages/SwingSet/test/vat-warehouse/test-reload-snapshot.js create mode 100644 packages/SwingSet/test/vat-warehouse/test-warehouse.js rename packages/SwingSet/test/{warehouse => vat-warehouse}/vat-target.js (100%) create mode 100644 packages/SwingSet/test/vat-warehouse/vat-warehouse-reload.js delete mode 100644 packages/SwingSet/test/warehouse/test-warehouse.js diff --git a/packages/SwingSet/src/controller.js b/packages/SwingSet/src/controller.js index a2e3f1af1d3..3489407c7b2 100644 --- a/packages/SwingSet/src/controller.js +++ b/packages/SwingSet/src/controller.js @@ -1,7 +1,6 @@ /* global require */ // @ts-check import fs from 'fs'; -import path from 'path'; import process from 'process'; import re2 from 're2'; import { performance } from 'perf_hooks'; @@ -9,13 +8,12 @@ import { spawn as ambientSpawn } from 'child_process'; import { type as osType } from 'os'; import { Worker } from 'worker_threads'; import anylogger from 'anylogger'; -import { tmpName } from 'tmp'; import { assert, details as X } from '@agoric/assert'; import { isTamed, tameMetering } from '@agoric/tame-metering'; import { importBundle } from '@agoric/import-bundle'; import { makeMeteringTransformer } from '@agoric/transform-metering'; -import { xsnap, makeSnapstore, recordXSnap } from '@agoric/xsnap'; +import { xsnap, recordXSnap } from '@agoric/xsnap'; import engineGC from './engine-gc.js'; import { WeakRef, FinalizationRegistry } from './weakref.js'; @@ -49,12 +47,12 @@ function unhandledRejectionHandler(e) { /** * @param {{ moduleFormat: string, source: string }[]} bundles * @param {{ - * snapstorePath?: string, + * snapStore?: SnapStore, * spawn: typeof import('child_process').spawn * env: Record, * }} opts */ -export function makeStartXSnap(bundles, { snapstorePath, env, spawn }) { +export function makeStartXSnap(bundles, { snapStore, env, spawn }) { /** @type { import('@agoric/xsnap/src/xsnap').XSnapOptions } */ const xsnapOpts = { os: osType(), @@ -79,37 +77,27 @@ export function makeStartXSnap(bundles, { snapstorePath, env, spawn }) { }; } - /** @type { ReturnType } */ - let snapStore; - - if (snapstorePath) { - fs.mkdirSync(snapstorePath, { recursive: true }); - - snapStore = makeSnapstore(snapstorePath, { - tmpName, - existsSync: fs.existsSync, - createReadStream: fs.createReadStream, - createWriteStream: fs.createWriteStream, - rename: fs.promises.rename, - unlink: fs.promises.unlink, - resolve: path.resolve, - }); - } - - let supervisorHash = ''; /** * @param {string} name * @param {(request: Uint8Array) => Promise} handleCommand * @param { boolean } [metered] + * @param { string } [snapshotHash] */ - async function startXSnap(name, handleCommand, metered) { - if (supervisorHash) { - return snapStore.load(supervisorHash, async snapshot => { + async function startXSnap( + name, + handleCommand, + metered, + snapshotHash = undefined, + ) { + if (snapStore && snapshotHash) { + // console.log('startXSnap from', { snapshotHash }); + return snapStore.load(snapshotHash, async snapshot => { const xs = doXSnap({ snapshot, name, handleCommand, ...xsnapOpts }); await xs.evaluate('null'); // ensure that spawn is done return xs; }); } + // console.log('fresh xsnap', { snapStore: snapStore }); const meterOpts = metered ? {} : { meteringLimit: 0 }; const worker = doXSnap({ handleCommand, name, ...meterOpts, ...xsnapOpts }); @@ -121,9 +109,6 @@ export function makeStartXSnap(bundles, { snapstorePath, env, spawn }) { // eslint-disable-next-line no-await-in-loop await worker.evaluate(`(${bundle.source}\n)()`.trim()); } - if (snapStore) { - supervisorHash = await snapStore.save(async fn => worker.snapshot(fn)); - } return worker; } return startXSnap; @@ -140,7 +125,6 @@ export function makeStartXSnap(bundles, { snapstorePath, env, spawn }) { * slogFile?: string, * testTrackDecref?: unknown, * warehousePolicy?: { maxVatsOnline?: number }, - * snapstorePath?: string, * spawn?: typeof import('child_process').spawn, * env?: Record * }} runtimeOptions @@ -162,7 +146,6 @@ export async function makeSwingsetController( debugPrefix = '', slogCallbacks, slogFile, - snapstorePath, spawn = ambientSpawn, warehousePolicy = {}, } = runtimeOptions; @@ -300,7 +283,11 @@ export async function makeSwingsetController( // @ts-ignore assume supervisorBundle is set JSON.parse(kvStore.get('supervisorBundle')), ]; - const startXSnap = makeStartXSnap(bundles, { snapstorePath, env, spawn }); + const startXSnap = makeStartXSnap(bundles, { + snapStore: hostStorage.snapStore, + env, + spawn, + }); const kernelEndowments = { waitUntilQuiescent, @@ -430,7 +417,6 @@ export async function makeSwingsetController( * debugPrefix?: string, * slogCallbacks?: unknown, * testTrackDecref?: unknown, - * snapstorePath?: string, * warehousePolicy?: { maxVatsOnline?: number }, * slogFile?: string, * }} runtimeOptions @@ -447,7 +433,6 @@ export async function buildVatController( kernelBundles, debugPrefix, slogCallbacks, - snapstorePath, warehousePolicy, slogFile, } = runtimeOptions; @@ -455,7 +440,6 @@ export async function buildVatController( verbose, debugPrefix, slogCallbacks, - snapstorePath, warehousePolicy, slogFile, }; diff --git a/packages/SwingSet/src/initializeSwingset.js b/packages/SwingSet/src/initializeSwingset.js index 27986eca753..16b7ea9275c 100644 --- a/packages/SwingSet/src/initializeSwingset.js +++ b/packages/SwingSet/src/initializeSwingset.js @@ -310,11 +310,6 @@ export async function initializeSwingset( // it to comms config.vats.vattp = { bundle: kernelBundles.vattp, - creationOptions: { - // we saw evidence of vattp dropping messages, and out of caution, - // we're keeping it on an in-kernel worker for now. See #3039. - managerType: 'local', - }, }; // timer wrapper vat is added automatically, but TODO: bootstraps must diff --git a/packages/SwingSet/src/kernel/kernel.js b/packages/SwingSet/src/kernel/kernel.js index 6fb9fb3d13f..af9d7072e88 100644 --- a/packages/SwingSet/src/kernel/kernel.js +++ b/packages/SwingSet/src/kernel/kernel.js @@ -125,7 +125,11 @@ export default function buildKernel( } = kernelOptions; const logStartup = verbose ? console.debug : () => 0; - const { kvStore, streamStore } = /** @type { HostStore } */ (hostStorage); + const { + kvStore, + streamStore, + snapStore, + } = /** @type { HostStore } */ (hostStorage); insistStorageAPI(kvStore); const { enhancedCrankBuffer, abortCrank, commitCrank } = wrapStorage(kvStore); const vatAdminRootKref = kvStore.get('vatAdminRootKref'); @@ -138,6 +142,7 @@ export default function buildKernel( enhancedCrankBuffer, streamStore, kernelSlog, + snapStore, ); const meterManager = makeMeterManager(replaceGlobalMeter); @@ -673,6 +678,8 @@ export default function buildKernel( if (!didAbort) { kernelKeeper.processRefcounts(); kernelKeeper.saveStats(); + // eslint-disable-next-line no-use-before-define + await vatWarehouse.maybeSaveSnapshot(); } commitCrank(); kernelKeeper.incrementCrankNumber(); diff --git a/packages/SwingSet/src/kernel/state/kernelKeeper.js b/packages/SwingSet/src/kernel/state/kernelKeeper.js index 6b3c61b7797..7fdcd3a5f6d 100644 --- a/packages/SwingSet/src/kernel/state/kernelKeeper.js +++ b/packages/SwingSet/src/kernel/state/kernelKeeper.js @@ -52,6 +52,7 @@ const enableKernelGC = true; // v$NN.nextDeliveryNum = $NN // v$NN.t.endPosition = $NN // v$NN.vs.$key = string +// v$NN.lastSnapshot = JSON({ snapshotID, startPos }) // d$NN.o.nextID = $NN // d$NN.c.$kernelSlot = $deviceSlot = o-$NN/d+$NN/d-$NN @@ -109,8 +110,14 @@ const FIRST_CRANK_NUMBER = 0n; * @param {KVStorePlus} kvStore * @param {StreamStore} streamStore * @param {KernelSlog} kernelSlog + * @param {SnapStore=} snapStore */ -export default function makeKernelKeeper(kvStore, streamStore, kernelSlog) { +export default function makeKernelKeeper( + kvStore, + streamStore, + kernelSlog, + snapStore = undefined, +) { insistEnhancedStorageAPI(kvStore); /** @@ -939,6 +946,7 @@ export default function makeKernelKeeper(kvStore, streamStore, kernelSlog) { incStat, decStat, getCrankNumber, + snapStore, ); ephemeral.vatKeepers.set(vatID, vk); return vk; diff --git a/packages/SwingSet/src/kernel/state/vatKeeper.js b/packages/SwingSet/src/kernel/state/vatKeeper.js index 22724b6c4d5..f0fb348c206 100644 --- a/packages/SwingSet/src/kernel/state/vatKeeper.js +++ b/packages/SwingSet/src/kernel/state/vatKeeper.js @@ -42,7 +42,7 @@ export function initializeVatState(kvStore, streamStore, vatID) { /** * Produce a vat keeper for a vat. * - * @param {*} kvStore The keyValue store in which the persistent state will be kept + * @param {KVStorePlus} kvStore The keyValue store in which the persistent state will be kept * @param {StreamStore} streamStore Accompanying stream store, for the transcripts * @param {*} kernelSlog * @param {string} vatID The vat ID string of the vat in question @@ -60,6 +60,7 @@ export function initializeVatState(kvStore, streamStore, vatID) { * @param {*} incStat * @param {*} decStat * @param {*} getCrankNumber + * @param { SnapStore= } snapStore * returns an object to hold and access the kernel's state for the given vat */ export function makeVatKeeper( @@ -79,6 +80,7 @@ export function makeVatKeeper( incStat, decStat, getCrankNumber, + snapStore = undefined, ) { insistVatID(vatID); const transcriptStream = `transcript-${vatID}`; @@ -417,6 +419,57 @@ export function makeVatKeeper( kvStore.set(`${vatID}.t.endPosition`, `${JSON.stringify(newPos)}`); } + /** @returns { StreamPosition } */ + function getTranscriptEndPosition() { + return JSON.parse( + kvStore.get(`${vatID}.t.endPosition`) || + assert.fail('missing endPosition'), + ); + } + + /** + * @returns {{ snapshotID: string, startPos: StreamPosition } | undefined} + */ + function getLastSnapshot() { + const notation = kvStore.get(`${vatID}.lastSnapshot`); + if (!notation) { + return undefined; + } + const { snapshotID, startPos } = JSON.parse(notation); + assert.typeof(snapshotID, 'string'); + assert(startPos); + return { snapshotID, startPos }; + } + + function transcriptSnapshotStats() { + const totalEntries = getTranscriptEndPosition().itemCount; + const lastSnapshot = getLastSnapshot(); + const snapshottedEntries = lastSnapshot + ? lastSnapshot.startPos.itemCount + : 0; + return { totalEntries, snapshottedEntries }; + } + + /** + * Store a snapshot, if given a snapStore. + * + * @param { VatManager } manager + * @returns { Promise } + */ + async function saveSnapshot(manager) { + if (!snapStore || !manager.makeSnapshot) { + return false; + } + + const snapshotID = await manager.makeSnapshot(snapStore); + const endPosition = getTranscriptEndPosition(); + kvStore.set( + `${vatID}.lastSnapshot`, + JSON.stringify({ snapshotID, startPos: endPosition }), + ); + return true; + } + function vatStats() { function getCount(key, first) { const id = Nat(BigInt(kvStore.get(key))); @@ -477,8 +530,11 @@ export function makeVatKeeper( deleteCListEntry, deleteCListEntriesForKernelSlots, getTranscript, + transcriptSnapshotStats, addToTranscript, vatStats, dumpState, + saveSnapshot, + getLastSnapshot, }); } diff --git a/packages/SwingSet/src/kernel/vatManager/manager-helper.js b/packages/SwingSet/src/kernel/vatManager/manager-helper.js index d193e8dfb68..63ceabca341 100644 --- a/packages/SwingSet/src/kernel/vatManager/manager-helper.js +++ b/packages/SwingSet/src/kernel/vatManager/manager-helper.js @@ -47,7 +47,8 @@ import { makeTranscriptManager } from './transcript.js'; /** * - * @typedef { { getManager: (shutdown: () => Promise) => VatManager, + * @typedef { { getManager: (shutdown: () => Promise, + * makeSnapshot?: (ss: SnapStore) => Promise) => VatManager, * syscallFromWorker: (vso: VatSyscallObject) => VatSyscallResult, * setDeliverToWorker: (dtw: unknown) => void, * } } ManagerKit @@ -178,12 +179,18 @@ function makeManagerKit( kernelSlog.write({ type: 'finish-replay-delivery', vatID, deliveryNum }); } - async function replayTranscript() { + /** + * @param {StreamPosition | undefined} startPos + * @returns { Promise } number of deliveries, or null if !useTranscript + */ + async function replayTranscript(startPos) { + // console.log('replay from', { vatID, startPos }); + if (transcriptManager) { const total = vatKeeper.vatStats().transcriptCount; kernelSlog.write({ type: 'start-replay', vatID, deliveries: total }); let deliveryNum = 0; - for (const t of vatKeeper.getTranscript()) { + for (const t of vatKeeper.getTranscript(startPos)) { // if (deliveryNum % 100 === 0) { // console.debug(`replay vatID:${vatID} deliveryNum:${deliveryNum} / ${total}`); // } @@ -194,7 +201,10 @@ function makeManagerKit( } transcriptManager.checkReplayError(); kernelSlog.write({ type: 'finish-replay', vatID }); + return deliveryNum; } + + return null; } /** @@ -235,10 +245,17 @@ function makeManagerKit( /** * * @param { () => Promise} shutdown + * @param { (ss: SnapStore) => Promise } makeSnapshot * @returns { VatManager } */ - function getManager(shutdown) { - return harden({ replayTranscript, replayOneDelivery, deliver, shutdown }); + function getManager(shutdown, makeSnapshot) { + return harden({ + replayTranscript, + replayOneDelivery, + deliver, + shutdown, + makeSnapshot, + }); } return harden({ getManager, syscallFromWorker, setDeliverToWorker }); diff --git a/packages/SwingSet/src/kernel/vatManager/manager-subprocess-xsnap.js b/packages/SwingSet/src/kernel/vatManager/manager-subprocess-xsnap.js index 8099cd1413c..6c9d0ed306e 100644 --- a/packages/SwingSet/src/kernel/vatManager/manager-subprocess-xsnap.js +++ b/packages/SwingSet/src/kernel/vatManager/manager-subprocess-xsnap.js @@ -23,7 +23,7 @@ const decoder = new TextDecoder(); * allVatPowers: VatPowers, * kernelKeeper: KernelKeeper, * kernelSlog: KernelSlog, - * startXSnap: (name: string, handleCommand: AsyncHandler, metered?: boolean) => Promise, + * startXSnap: (name: string, handleCommand: AsyncHandler, metered?: boolean, snapshotHash?: string) => Promise, * testLog: (...args: unknown[]) => void, * }} tools * @returns { VatManagerFactory } @@ -109,8 +109,16 @@ export function makeXsSubprocessFactory({ return encoder.encode(JSON.stringify(tagged)); } + const vatKeeper = kernelKeeper.provideVatKeeper(vatID); + const lastSnapshot = vatKeeper.getLastSnapshot(); + // start the worker and establish a connection - const worker = await startXSnap(`${vatID}:${name}`, handleCommand, metered); + const worker = await startXSnap( + `${vatID}:${name}`, + handleCommand, + metered, + lastSnapshot ? lastSnapshot.snapshotID : undefined, + ); /** @type { (item: Tagged) => Promise } */ async function issueTagged(item) { @@ -122,22 +130,26 @@ export function makeXsSubprocessFactory({ return { ...result, reply: [tag, ...rest] }; } - parentLog(vatID, `instructing worker to load bundle..`); - const { reply: bundleReply } = await issueTagged([ - 'setBundle', - vatID, - bundle, - vatParameters, - virtualObjectCacheSize, - enableDisavow, - enableVatstore, - gcEveryCrank, - ]); - if (bundleReply[0] === 'dispatchReady') { - parentLog(vatID, `bundle loaded. dispatch ready.`); + if (lastSnapshot) { + parentLog(vatID, `snapshot loaded. dispatch ready.`); } else { - const [_tag, errName, message] = bundleReply; - assert.fail(X`setBundle failed: ${q(errName)}: ${q(message)}`); + parentLog(vatID, `instructing worker to load bundle..`); + const { reply: bundleReply } = await issueTagged([ + 'setBundle', + vatID, + bundle, + vatParameters, + virtualObjectCacheSize, + enableDisavow, + enableVatstore, + gcEveryCrank, + ]); + if (bundleReply[0] === 'dispatchReady') { + parentLog(vatID, `bundle loaded. dispatch ready.`); + } else { + const [_tag, errName, message] = bundleReply; + assert.fail(X`setBundle failed: ${q(errName)}: ${q(message)}`); + } } /** @@ -184,7 +196,15 @@ export function makeXsSubprocessFactory({ function shutdown() { return worker.close().then(_ => undefined); } - return mk.getManager(shutdown); + /** + * @param {SnapStore} snapStore + * @returns {Promise} + */ + function makeSnapshot(snapStore) { + return snapStore.save(fn => worker.snapshot(fn)); + } + + return mk.getManager(shutdown, makeSnapshot); } return harden({ createFromBundle }); diff --git a/packages/SwingSet/src/kernel/vatManager/vat-warehouse.js b/packages/SwingSet/src/kernel/vatManager/vat-warehouse.js index 2fad4e2e229..1c2e738f74d 100644 --- a/packages/SwingSet/src/kernel/vatManager/vat-warehouse.js +++ b/packages/SwingSet/src/kernel/vatManager/vat-warehouse.js @@ -1,11 +1,55 @@ // @ts-check -import { assert } from '@agoric/assert'; +import { assert, details as X, quote as q } from '@agoric/assert'; import { makeVatTranslators } from '../vatTranslator.js'; +/** @param { number } max */ +export const makeLRU = max => { + /** @type { string[] } */ + const items = []; + + return harden({ + /** @param { string } item */ + add: item => { + const pos = items.indexOf(item); + // already most recently used + if (pos + 1 === max) { + return null; + } + // remove from former position + if (pos >= 0) { + items.splice(pos, 1); + } + items.push(item); + // not yet full + if (items.length <= max) { + return null; + } + const [removed] = items.splice(0, 1); + return removed; + }, + + get size() { + return items.length; + }, + + /** @param { string } item */ + remove: item => { + const pos = items.indexOf(item); + if (pos >= 0) { + items.splice(pos, 1); + } + }, + }); +}; + /** * @param { KernelKeeper } kernelKeeper * @param { ReturnType } vatLoader - * @param {{ maxVatsOnline?: number }=} policyOptions + * @param {{ + * maxVatsOnline?: number, + * snapshotInitial?: number, + * snapshotInterval?: number, + * }=} policyOptions * * @typedef {(syscall: VatSyscallObject) => ['error', string] | ['ok', null] | ['ok', Capdata]} VatSyscallHandler * @typedef {{ body: string, slots: unknown[] }} Capdata @@ -13,14 +57,24 @@ import { makeVatTranslators } from '../vatTranslator.js'; * @typedef { { moduleFormat: string }} Bundle */ export function makeVatWarehouse(kernelKeeper, vatLoader, policyOptions) { - const { maxVatsOnline = 50 } = policyOptions || {}; + const { + maxVatsOnline = 50, + // Often a large contract evaluation is among the first few deliveries, + // so let's do a snapshot after just a few deliveries. + snapshotInitial = 2, + // Then we'll snapshot at invervals of some number of cranks. + // Note: some measurements show 10 deliveries per sec on XS + // as of this writing. + snapshotInterval = 200, + } = policyOptions || {}; + // Idea: snapshot based on delivery size: after deliveries >10Kb. // console.debug('makeVatWarehouse', { policyOptions }); /** * @typedef {{ * manager: VatManager, * enablePipelining: boolean, - * options: { name?: string, description?: string }, + * options: { name?: string, description?: string, managerType?: ManagerType }, * }} VatInfo * @typedef { ReturnType } VatTranslators */ @@ -52,6 +106,7 @@ export function makeVatWarehouse(kernelKeeper, vatLoader, policyOptions) { const info = ephemeral.vats.get(vatID); if (info) return info; + assert(kernelKeeper.vatIsAlive(vatID), X`${q(vatID)}: not alive`); const vatKeeper = kernelKeeper.provideVatKeeper(vatID); const { source, options } = vatKeeper.getSourceAndOptions(); @@ -69,13 +124,15 @@ export function makeVatWarehouse(kernelKeeper, vatLoader, policyOptions) { return vatLoader.createVatDynamically; } }; - // console.log('provide: creating from bundle', vatID); const manager = await chooseLoader()(vatID, source, translators, options); // TODO(3218): persist this option; avoid spinning up a vat that isn't pipelined const { enablePipelining = false } = options; - await manager.replayTranscript(); + const lastSnapshot = vatKeeper.getLastSnapshot(); + const entriesReplayed = await manager.replayTranscript( + lastSnapshot ? lastSnapshot.startPos : undefined, + ); const result = { manager, @@ -83,6 +140,14 @@ export function makeVatWarehouse(kernelKeeper, vatLoader, policyOptions) { enablePipelining, options, }; + console.log( + vatID, + 'online:', + options.managerType, + options.description || '', + 'transcript entries replayed:', + entriesReplayed, + ); ephemeral.vats.set(vatID, result); // eslint-disable-next-line no-use-before-define await applyAvailabilityPolicy(vatID); @@ -139,17 +204,25 @@ export function makeVatWarehouse(kernelKeeper, vatLoader, policyOptions) { return { enablePipelining }; } + const recent = makeLRU(maxVatsOnline); + /** + * + * does not modify the kernelDB * * @param {string} vatID - * @param {boolean=} makeSnapshot * @returns { Promise } */ - async function evict(vatID, makeSnapshot = false) { - assert(!makeSnapshot, 'not implemented'); + async function evict(vatID) { assert(lookup(vatID)); + + recent.remove(vatID); + const info = ephemeral.vats.get(vatID); - if (!info) return undefined; + if (!info) { + // console.debug('evict: not online:', vatID); + return undefined; + } ephemeral.vats.delete(vatID); xlate.delete(vatID); kernelKeeper.closeVatTranscript(vatID); @@ -159,9 +232,6 @@ export function makeVatWarehouse(kernelKeeper, vatLoader, policyOptions) { return info.manager.shutdown(); } - /** @type { string[] } */ - const recent = []; - /** * Simple fixed-size LRU cache policy * @@ -173,29 +243,65 @@ export function makeVatWarehouse(kernelKeeper, vatLoader, policyOptions) { * @param {string} currentVatID */ async function applyAvailabilityPolicy(currentVatID) { - // console.log('applyAvailabilityPolicy', currentVatID, recent); - const pos = recent.indexOf(currentVatID); - // console.debug('applyAvailabilityPolicy', { currentVatID, recent, pos }); - // already most recently used - if (pos + 1 === maxVatsOnline) return; - if (pos >= 0) recent.splice(pos, 1); - recent.push(currentVatID); - // not yet full - if (recent.length <= maxVatsOnline) return; - const [lru] = recent.splice(0, 1); - // console.debug('evicting', { lru }); + const lru = recent.add(currentVatID); + if (!lru) { + return; + } + // const { + // options: { description, managerType }, + // } = ephemeral.vats.get(lru) || assert.fail(); + // console.info('evict', lru, description, managerType, 'for', currentVatID); await evict(lru); } + /** @type { string | undefined } */ + let lastVatID; + /** @type {(vatID: string, d: VatDeliveryObject) => Promise } */ async function deliverToVat(vatID, delivery) { await applyAvailabilityPolicy(vatID); - const recreate = true; // PANIC in the failure case + lastVatID = vatID; + const recreate = true; // PANIC in the failure case const { manager } = await ensureVatOnline(vatID, recreate); return manager.deliver(delivery); } + /** + * Save a snapshot of most recently used vat, + * depending on snapshotInterval. + */ + async function maybeSaveSnapshot() { + if (!lastVatID || !lookup(lastVatID)) { + return false; + } + + const recreate = true; // PANIC in the failure case + const { manager } = await ensureVatOnline(lastVatID, recreate); + if (!manager.makeSnapshot) { + return false; + } + + const vatKeeper = kernelKeeper.provideVatKeeper(lastVatID); + let reason; + const { + totalEntries, + snapshottedEntries, + } = vatKeeper.transcriptSnapshotStats(); + if (snapshotInitial === totalEntries) { + reason = { snapshotInitial }; + } else if (totalEntries - snapshottedEntries >= snapshotInterval) { + reason = { snapshotInterval }; + } + // console.log('maybeSaveSnapshot: reason:', reason); + if (!reason) { + return false; + } + await vatKeeper.saveSnapshot(manager); + lastVatID = undefined; + return true; + } + /** * @param {string} vatID * @param {unknown[]} kd @@ -235,7 +341,7 @@ export function makeVatWarehouse(kernelKeeper, vatLoader, policyOptions) { */ async function vatWasTerminated(vatID) { try { - await evict(vatID, false); + await evict(vatID); } catch (err) { console.debug('vat termination was already reported; ignoring:', err); } @@ -256,6 +362,7 @@ export function makeVatWarehouse(kernelKeeper, vatLoader, policyOptions) { lookup, kernelDeliveryToVatDelivery, deliverToVat, + maybeSaveSnapshot, // mostly for testing? activeVatsInfo: () => diff --git a/packages/SwingSet/src/types.js b/packages/SwingSet/src/types.js index 1c7f7c67b1c..9e581962c4a 100644 --- a/packages/SwingSet/src/types.js +++ b/packages/SwingSet/src/types.js @@ -125,10 +125,11 @@ * vatSyscallHandler: unknown) => Promise, * } } VatManagerFactory * @typedef { { deliver: (delivery: VatDeliveryObject) => Promise, - * replayTranscript: () => Promise, + * replayTranscript: (startPos: StreamPosition | undefined) => Promise, + * makeSnapshot?: (ss: SnapStore) => Promise, * shutdown: () => Promise, * } } VatManager - * @typedef { ReturnType } SnapStore + * @typedef { ReturnType } SnapStore * @typedef { () => Promise } WaitUntilQuiescent */ @@ -179,6 +180,7 @@ * @typedef {{ * kvStore: KVStore, * streamStore: StreamStore, + * snapStore?: SnapStore, * }} HostStore * * @typedef { ReturnType } KVStorePlus diff --git a/packages/SwingSet/test/test-controller.js b/packages/SwingSet/test/test-controller.js index 43cb0e6a93c..115222fb5b7 100644 --- a/packages/SwingSet/test/test-controller.js +++ b/packages/SwingSet/test/test-controller.js @@ -145,7 +145,7 @@ test('static vats are unmetered on XS', async t => { }, ); t.deepEqual(c.dump().log, ['bootstrap called']); - t.deepEqual(limited, [false, false, false]); + t.deepEqual(limited, [false, false, false, false]); }); test('validate config.defaultManagerType', async t => { diff --git a/packages/SwingSet/test/test-gc-transcript.js b/packages/SwingSet/test/test-gc-transcript.js index be18f25f48f..07002aa1cac 100644 --- a/packages/SwingSet/test/test-gc-transcript.js +++ b/packages/SwingSet/test/test-gc-transcript.js @@ -21,6 +21,7 @@ function setup(storedTranscript = []) { return storedTranscript; }, closeTranscript() {}, + getLastSnapshot: () => undefined, }; const kernelKeeper = { provideVatKeeper() { diff --git a/packages/SwingSet/test/test-xsnap-errors.js b/packages/SwingSet/test/test-xsnap-errors.js index 046272ac75e..0103ba7275f 100644 --- a/packages/SwingSet/test/test-xsnap-errors.js +++ b/packages/SwingSet/test/test-xsnap-errors.js @@ -46,7 +46,9 @@ test('child termination during crank', async t => { // just enough methods to not crash /** @type { any } */ const kernelKeeper = { - provideVatKeeper: () => undefined, + provideVatKeeper: () => ({ + getLastSnapshot: () => undefined, + }), }; const xsWorkerFactory = makeXsSubprocessFactory({ diff --git a/packages/SwingSet/test/warehouse/bootstrap.js b/packages/SwingSet/test/vat-warehouse/bootstrap.js similarity index 100% rename from packages/SwingSet/test/warehouse/bootstrap.js rename to packages/SwingSet/test/vat-warehouse/bootstrap.js diff --git a/packages/SwingSet/test/vat-warehouse/test-reload-snapshot.js b/packages/SwingSet/test/vat-warehouse/test-reload-snapshot.js new file mode 100644 index 00000000000..4f48f14ce5e --- /dev/null +++ b/packages/SwingSet/test/vat-warehouse/test-reload-snapshot.js @@ -0,0 +1,83 @@ +/* global __dirname */ +// eslint-disable-next-line import/order +import { test } from '../../tools/prepare-test-env-ava.js'; + +import fs from 'fs'; +import path from 'path'; +import { tmpName } from 'tmp'; +import { makeSnapStore } from '@agoric/xsnap'; +import { provideHostStorage } from '../../src/hostStorage.js'; +import { initializeSwingset, makeSwingsetController } from '../../src/index.js'; +import { capargs } from '../util.js'; + +test('vat reload from snapshot', async t => { + const config = { + vats: { + target: { + sourceSpec: path.join(__dirname, 'vat-warehouse-reload.js'), + creationOptions: { managerType: 'xs-worker' }, + }, + }, + }; + + const snapstorePath = path.resolve( + __dirname, + './fixture-test-reload-snapshot/', + ); + fs.mkdirSync(snapstorePath, { recursive: true }); + t.teardown(() => fs.rmdirSync(snapstorePath, { recursive: true })); + + const snapStore = makeSnapStore(snapstorePath, { + tmpName, + existsSync: fs.existsSync, + createReadStream: fs.createReadStream, + createWriteStream: fs.createWriteStream, + rename: fs.promises.rename, + unlink: fs.promises.unlink, + resolve: path.resolve, + }); + const hostStorage = { snapStore, ...provideHostStorage() }; + + const argv = []; + await initializeSwingset(config, argv, hostStorage); + + const c1 = await makeSwingsetController(hostStorage, null, { + warehousePolicy: { initialSnapshot: 2, snapshotInterval: 5 }, + }); + const vatID = c1.vatNameToID('target'); + + function getPositions() { + const lastSnapshot = hostStorage.kvStore.get(`${vatID}.lastSnapshot`); + const start = lastSnapshot + ? JSON.parse(lastSnapshot).startPos.itemCount + : 0; + const endPosition = hostStorage.kvStore.get(`${vatID}.t.endPosition`); + const end = JSON.parse(endPosition).itemCount; + return [start, end]; + } + + const expected1 = []; + c1.queueToVatExport('target', 'o+0', 'count', capargs([])); + expected1.push(`count = 0`); + await c1.run(); + t.deepEqual(c1.dump().log, expected1); + t.deepEqual(getPositions(), [0, 1]); + + for (let i = 1; i < 11; i += 1) { + c1.queueToVatExport('target', 'o+0', 'count', capargs([])); + expected1.push(`count = ${i}`); + } + await c1.run(); + t.deepEqual(c1.dump().log, expected1); + t.deepEqual(getPositions(), [7, 11]); + await c1.shutdown(); + + const c2 = await makeSwingsetController(hostStorage); + const expected2 = [`count = 7`, `count = 8`, `count = 9`, `count = 10`]; + t.deepEqual(c2.dump().log, expected2); // replayed 4 deliveries + c2.queueToVatExport('target', 'o+0', 'count', capargs([])); + expected2.push(`count = 11`); + await c2.run(); + t.deepEqual(c2.dump().log, expected2); // note: *not* 0-11 + t.deepEqual(getPositions(), [7, 12]); +}); diff --git a/packages/SwingSet/test/vat-warehouse/test-warehouse.js b/packages/SwingSet/test/vat-warehouse/test-warehouse.js new file mode 100644 index 00000000000..9ddfd50d738 --- /dev/null +++ b/packages/SwingSet/test/vat-warehouse/test-warehouse.js @@ -0,0 +1,137 @@ +/* global __dirname */ +// @ts-check + +// eslint-disable-next-line import/order +import { test } from '../../tools/prepare-test-env-ava'; +import path from 'path'; +import fs from 'fs'; +import { tmpName } from 'tmp'; +import { makeSnapStore } from '@agoric/xsnap'; +import { loadBasedir, buildVatController } from '../../src/index.js'; +import { provideHostStorage } from '../../src/hostStorage.js'; +import { makeLRU } from '../../src/kernel/vatManager/vat-warehouse.js'; + +async function makeController(managerType, runtimeOptions) { + const config = await loadBasedir(__dirname); + config.vats.target.creationOptions = { managerType, enableDisavow: true }; + config.vats.target2 = config.vats.target; + config.vats.target3 = config.vats.target; + config.vats.target4 = config.vats.target; + const c = await buildVatController(config, [], runtimeOptions); + return c; +} + +/** @type { (body: string, slots?: string[]) => SwingSetCapData } */ +function capdata(body, slots = []) { + return harden({ body, slots }); +} + +/** @type { (args: unknown[], slots?: string[]) => SwingSetCapData } */ +function capargs(args, slots = []) { + return capdata(JSON.stringify(args), slots); +} + +const maxVatsOnline = 2; +const steps = [ + { + // After we deliver to... + vat: 'target', + // ... we expect these vats online: + online: [ + { id: 'v3', name: 'bootstrap' }, + { id: 'v1', name: 'target' }, + ], + }, + { + vat: 'target2', + online: [ + { id: 'v1', name: 'target' }, + { id: 'v4', name: 'target2' }, + ], + }, + { + vat: 'target3', + online: [ + { id: 'v4', name: 'target2' }, + { id: 'v5', name: 'target3' }, + ], + }, + { + vat: 'target4', + online: [ + { id: 'v5', name: 'target3' }, + { id: 'v6', name: 'target4' }, + ], + }, + { + vat: 'target2', + online: [ + { id: 'v6', name: 'target4' }, + { id: 'v4', name: 'target2' }, + ], + }, +]; + +async function runSteps(c, t) { + t.teardown(c.shutdown); + + await c.run(); + for (const { vat, online } of steps) { + t.log('sending to vat', vat); + c.queueToVatExport(vat, 'o+0', 'append', capargs([1])); + // eslint-disable-next-line no-await-in-loop + await c.run(); + t.log( + 'max:', + maxVatsOnline, + 'expected online:', + online.map(({ id, name }) => [id, name]), + ); + t.deepEqual( + c + .getStatus() + .activeVats.map(({ id, options: { name } }) => ({ id, name })), + online, + ); + } +} + +test('4 vats in warehouse with 2 online', async t => { + const c = await makeController('xs-worker', { + warehousePolicy: { maxVatsOnline }, + }); + await runSteps(c, t); +}); + +test('snapshot after deliveries', async t => { + const snapstorePath = path.resolve(__dirname, './fixture-test-warehouse/'); + fs.mkdirSync(snapstorePath, { recursive: true }); + t.teardown(() => fs.rmdirSync(snapstorePath, { recursive: true })); + + const snapStore = makeSnapStore(snapstorePath, { + tmpName, + existsSync: fs.existsSync, + createReadStream: fs.createReadStream, + createWriteStream: fs.createWriteStream, + rename: fs.promises.rename, + unlink: fs.promises.unlink, + resolve: path.resolve, + }); + const hostStorage = { snapStore, ...provideHostStorage() }; + const c = await makeController('xs-worker', { + hostStorage, + warehousePolicy: { maxVatsOnline, snapshotInterval: 1 }, + }); + await runSteps(c, t); +}); + +test('LRU eviction', t => { + const recent = makeLRU(3); + const actual = []; + for (const current of ['v0', 'v1', 'v2', 'v3', 'v3', 'v2']) { + const evict = recent.add(current); + t.log({ size: recent.size, current, evict }); + actual.push(evict); + } + t.deepEqual(actual, [null, null, null, 'v0', null, null]); +}); diff --git a/packages/SwingSet/test/warehouse/vat-target.js b/packages/SwingSet/test/vat-warehouse/vat-target.js similarity index 100% rename from packages/SwingSet/test/warehouse/vat-target.js rename to packages/SwingSet/test/vat-warehouse/vat-target.js diff --git a/packages/SwingSet/test/vat-warehouse/vat-warehouse-reload.js b/packages/SwingSet/test/vat-warehouse/vat-warehouse-reload.js new file mode 100644 index 00000000000..56838d2d6a0 --- /dev/null +++ b/packages/SwingSet/test/vat-warehouse/vat-warehouse-reload.js @@ -0,0 +1,12 @@ +import { Far } from '@agoric/marshal'; + +export function buildRootObject(vatPowers) { + const { testLog: log } = vatPowers; + let count = 0; + return Far('root', { + count() { + log(`count = ${count}`); + count += 1; + }, + }); +} diff --git a/packages/SwingSet/test/warehouse/test-warehouse.js b/packages/SwingSet/test/warehouse/test-warehouse.js deleted file mode 100644 index fc7ee2eaeb1..00000000000 --- a/packages/SwingSet/test/warehouse/test-warehouse.js +++ /dev/null @@ -1,86 +0,0 @@ -/* global __dirname */ -// @ts-check - -import { test } from '../../tools/prepare-test-env-ava.js'; - -import { loadBasedir, buildVatController } from '../../src/index.js'; - -async function makeController(managerType, maxVatsOnline) { - const config = await loadBasedir(__dirname); - config.vats.target.creationOptions = { managerType, enableDisavow: true }; - config.vats.target2 = config.vats.target; - config.vats.target3 = config.vats.target; - config.vats.target4 = config.vats.target; - const warehousePolicy = { maxVatsOnline }; - const c = await buildVatController(config, [], { warehousePolicy }); - return c; -} - -/** @type { (body: string, slots?: string[]) => SwingSetCapData } */ -function capdata(body, slots = []) { - return harden({ body, slots }); -} - -/** @type { (args: unknown[], slots?: string[]) => SwingSetCapData } */ -function capargs(args, slots = []) { - return capdata(JSON.stringify(args), slots); -} - -const maxVatsOnline = 2; -const steps = [ - { - // After we deliver to... - vat: 'target', - // ... we expect these vats online: - online: [ - { id: 'v2', name: 'bootstrap' }, - { id: 'v1', name: 'target' }, - ], - }, - { - vat: 'target2', - online: [ - { id: 'v1', name: 'target' }, - { id: 'v3', name: 'target2' }, - ], - }, - { - vat: 'target3', - online: [ - { id: 'v3', name: 'target2' }, - { id: 'v4', name: 'target3' }, - ], - }, - { - vat: 'target4', - online: [ - { id: 'v4', name: 'target3' }, - { id: 'v5', name: 'target4' }, - ], - }, -]; - -test('4 vats in warehouse with 2 online', async t => { - const c = await makeController('xs-worker', maxVatsOnline); - t.teardown(c.shutdown); - - await c.run(); - for (const { vat, online } of steps) { - t.log('sending to vat', vat); - c.queueToVatExport(vat, 'o+0', 'append', capargs([1])); - // eslint-disable-next-line no-await-in-loop - await c.run(); - t.log( - 'max:', - maxVatsOnline, - 'expected online:', - online.map(({ id, name }) => [id, name]), - ); - t.deepEqual( - c - .getStatus() - .activeVats.map(({ id, options: { name } }) => ({ id, name })), - online, - ); - } -}); diff --git a/packages/cosmic-swingset/package.json b/packages/cosmic-swingset/package.json index 8936b8a7e1b..bbca9981ab2 100644 --- a/packages/cosmic-swingset/package.json +++ b/packages/cosmic-swingset/package.json @@ -36,6 +36,7 @@ "@agoric/swing-store-lmdb": "^0.5.4", "@agoric/swingset-vat": "^0.18.4", "@agoric/vats": "^0.2.8", + "@agoric/xsnap": "^0.6.3", "@iarna/toml": "^2.2.3", "@opentelemetry/exporter-prometheus": "^0.16.0", "@opentelemetry/metrics": "^0.16.0", @@ -43,7 +44,8 @@ "anylogger": "^0.21.0", "deterministic-json": "^1.0.5", "esm": "agoric-labs/esm#Agoric-built", - "node-lmdb": "^0.9.4" + "node-lmdb": "^0.9.4", + "tmp": "^0.2.1" }, "devDependencies": { "ava": "^3.12.1" diff --git a/packages/cosmic-swingset/src/launch-chain.js b/packages/cosmic-swingset/src/launch-chain.js index 5ffddfad4b0..d8f1ed6290a 100644 --- a/packages/cosmic-swingset/src/launch-chain.js +++ b/packages/cosmic-swingset/src/launch-chain.js @@ -1,4 +1,7 @@ +import fs from 'fs'; +import path from 'path'; import anylogger from 'anylogger'; +import { tmpName } from 'tmp'; // TODO: reconcile tmp vs. temp import { buildMailbox, @@ -13,6 +16,7 @@ import { } from '@agoric/swingset-vat'; import { assert, details as X } from '@agoric/assert'; import { openLMDBSwingStore } from '@agoric/swing-store-lmdb'; +import { makeSnapStore } from '@agoric/xsnap'; import { DEFAULT_METER_PROVIDER, exportKernelStats, @@ -98,9 +102,21 @@ export async function launch( console.info('Launching SwingSet kernel'); const { kvStore, streamStore, commit } = openLMDBSwingStore(kernelStateDBDir); + const snapshotDir = path.resolve(kernelStateDBDir, 'xs-snapshots'); + fs.mkdirSync(snapshotDir, { recursive: true }); + const snapStore = makeSnapStore(snapshotDir, { + tmpName, + existsSync: fs.existsSync, + createReadStream: fs.createReadStream, + createWriteStream: fs.createWriteStream, + rename: fs.promises.rename, + unlink: fs.promises.unlink, + resolve: path.resolve, + }); const hostStorage = { kvStore, streamStore, + snapStore, }; // Not to be confused with the gas model, this meter is for OpenTelemetry. diff --git a/packages/solo/package.json b/packages/solo/package.json index 56a720dd1f0..fe827e5c5f1 100644 --- a/packages/solo/package.json +++ b/packages/solo/package.json @@ -43,6 +43,7 @@ "@agoric/swing-store-lmdb": "^0.5.4", "@agoric/swingset-vat": "^0.18.4", "@agoric/vats": "^0.2.8", + "@agoric/xsnap": "^0.6.3", "anylogger": "^0.21.0", "deterministic-json": "^1.0.5", "esm": "agoric-labs/esm#Agoric-built", @@ -52,6 +53,7 @@ "node-fetch": "^2.6.0", "node-lmdb": "^0.9.4", "temp": "^0.9.1", + "tmp": "^0.2.1", "ws": "^7.2.0" }, "devDependencies": { diff --git a/packages/solo/src/start.js b/packages/solo/src/start.js index dd1c8a2df43..6529e6c15c5 100644 --- a/packages/solo/src/start.js +++ b/packages/solo/src/start.js @@ -2,6 +2,7 @@ import fs from 'fs'; import path from 'path'; import temp from 'temp'; +import { tmpName } from 'tmp'; // TODO: reconcile tmp vs. temp import { fork } from 'child_process'; import { promisify } from 'util'; // import { createHash } from 'crypto'; @@ -24,6 +25,7 @@ import { buildTimer, } from '@agoric/swingset-vat'; import { openLMDBSwingStore } from '@agoric/swing-store-lmdb'; +import { makeSnapStore } from '@agoric/xsnap'; import { connectToFakeChain } from '@agoric/cosmic-swingset/src/sim-chain'; import { makeWithQueue } from '@agoric/vats/src/queue'; @@ -135,9 +137,21 @@ async function buildSwingset( }; const { kvStore, streamStore, commit } = openLMDBSwingStore(kernelStateDBDir); + const snapshotDir = path.resolve(kernelStateDBDir, 'xs-snapshots'); + fs.mkdirSync(snapshotDir, { recursive: true }); + const snapStore = makeSnapStore(snapshotDir, { + tmpName, + existsSync: fs.existsSync, + createReadStream: fs.createReadStream, + createWriteStream: fs.createWriteStream, + rename: fs.promises.rename, + unlink: fs.promises.unlink, + resolve: path.resolve, + }); const hostStorage = { kvStore, streamStore, + snapStore, }; if (!swingsetIsInitialized(hostStorage)) { diff --git a/packages/swing-store-simple/src/simpleSwingStore.js b/packages/swing-store-simple/src/simpleSwingStore.js index 58d3c5e6c79..06188e57f2a 100644 --- a/packages/swing-store-simple/src/simpleSwingStore.js +++ b/packages/swing-store-simple/src/simpleSwingStore.js @@ -12,7 +12,7 @@ import { assert, details as X, q } from '@agoric/assert'; * * @typedef {{ * offset?: number, - * itemCount?: number, + * itemCount: number, * }} StreamPosition * * @typedef {{ diff --git a/packages/xsnap/src/index.js b/packages/xsnap/src/index.js index 31e2696da8b..5a5595c80e4 100644 --- a/packages/xsnap/src/index.js +++ b/packages/xsnap/src/index.js @@ -6,5 +6,5 @@ export { ErrorSignal, METER_TYPE, } from '../api.js'; -export { makeSnapstore } from './snapStore.js'; +export { makeSnapStore } from './snapStore.js'; export { recordXSnap, replayXSnap } from './replay.js'; diff --git a/packages/xsnap/src/replay.js b/packages/xsnap/src/replay.js index 5e6f6f7b8ca..716dd6a17ed 100644 --- a/packages/xsnap/src/replay.js +++ b/packages/xsnap/src/replay.js @@ -99,12 +99,10 @@ export function recordXSnap(options, folderPath, { writeFileSync }) { return folder.file(fn); }; - /** @param { Uint8Array } msg */ - const echo = msg => msg; + const { handleCommand: handle = msg => msg } = options; /** @param { Uint8Array} msg */ async function handleCommand(msg) { - const { handleCommand: handle = echo } = options; const result = await handle(msg); nextFile('reply').put(result); return result; diff --git a/packages/xsnap/src/snapStore.js b/packages/xsnap/src/snapStore.js index 2aa81b75d5c..274ed07bc5a 100644 --- a/packages/xsnap/src/snapStore.js +++ b/packages/xsnap/src/snapStore.js @@ -21,7 +21,7 @@ const { freeze } = Object; * unlink: typeof import('fs').promises.unlink, * }} io */ -export function makeSnapstore( +export function makeSnapStore( root, { tmpName, @@ -35,14 +35,17 @@ export function makeSnapstore( ) { /** @type {(opts: unknown) => Promise} */ const ptmpName = promisify(tmpName); - const tmpOpts = { tmpdir: root, template: 'tmp-XXXXXX.xss' }; /** * @param { (name: string) => Promise } thunk + * @param { string= } prefix * @returns { Promise } * @template T */ - async function withTempName(thunk) { - const name = await ptmpName(tmpOpts); + async function withTempName(thunk, prefix = 'tmp') { + const name = await ptmpName({ + tmpdir: root, + template: `${prefix}-XXXXXX.xss`, + }); let result; try { result = await thunk(name); @@ -63,7 +66,7 @@ export function makeSnapstore( * @template T */ async function atomicWrite(dest, thunk) { - const tmp = await ptmpName(tmpOpts); + const tmp = await ptmpName({ tmpdir: root, template: 'atomic-XXXXXX' }); let result; try { result = await thunk(tmp); @@ -101,12 +104,15 @@ export function makeSnapstore( return withTempName(async snapFile => { await saveRaw(snapFile); const h = await fileHash(snapFile); + // console.log('save', { snapFile, h }); if (existsSync(`${h}.gz`)) return h; await atomicWrite(`${h}.gz`, gztmp => filter(snapFile, createGzip(), gztmp), ); + const basename = snapFile.split('/').slice(-1)[0]; // @@WIP + await rename(snapFile, resolve(root, `${h}-${basename}`)); // @@WIP return h; - }); + }, 'save-raw'); } /** @@ -118,11 +124,12 @@ export function makeSnapstore( return withTempName(async raw => { await filter(resolve(root, `${hash}.gz`), createGunzip(), raw); const actual = await fileHash(raw); + // console.log('load', { raw, hash }); assert(actual === hash, d`actual hash ${actual} !== expected ${hash}`); // be sure to await loadRaw before exiting withTempName const result = await loadRaw(raw); return result; - }); + }, `${hash}-load`); } return freeze({ load, save }); diff --git a/packages/xsnap/test/test-snapstore.js b/packages/xsnap/test/test-snapstore.js index 1707cb33201..c2dd8dc0d23 100644 --- a/packages/xsnap/test/test-snapstore.js +++ b/packages/xsnap/test/test-snapstore.js @@ -13,7 +13,7 @@ import test from 'ava'; // eslint-disable-next-line import/no-extraneous-dependencies import tmp from 'tmp'; import { xsnap } from '../src/xsnap.js'; -import { makeSnapstore } from '../src/snapStore.js'; +import { makeSnapStore } from '../src/snapStore.js'; import { loader } from './message-tools.js'; const importMeta = { url: `file://${__filename}` }; @@ -22,8 +22,9 @@ const ld = loader(importMeta.url, fs.promises.readFile); // WARNING: ambient /** * @param {string} name * @param {(request:Uint8Array) => Promise} handleCommand + * @param {string} script to execute */ -async function bootWorker(name, handleCommand) { +async function bootWorker(name, handleCommand, script) { const worker = xsnap({ os: osType(), spawn, @@ -34,17 +35,35 @@ async function bootWorker(name, handleCommand) { // debug: !!env.XSNAP_DEBUG, }); - const bootScript = await ld.asset('../dist/bundle-ses-boot.umd.js'); - await worker.evaluate(bootScript); + await worker.evaluate(script); return worker; } +/** + * @param {string} name + * @param {(request:Uint8Array) => Promise} handleCommand + */ +async function bootSESWorker(name, handleCommand) { + const bootScript = await ld.asset('../dist/bundle-ses-boot.umd.js'); + return bootWorker(name, handleCommand, bootScript); +} + +/** @type {(fn: string, fullSize: number) => number} */ +const relativeSize = (fn, fullSize) => + Math.round((fs.statSync(fn).size / 1024 / fullSize) * 10) / 10; + +const snapSize = { + raw: 417, + SESboot: 858, + compression: 0.1, +}; + test('build temp file; compress to cache file', async t => { const pool = tmp.dirSync({ unsafeCleanup: true }); t.teardown(() => pool.removeCallback()); t.log({ pool: pool.name }); await fs.promises.mkdir(pool.name, { recursive: true }); - const store = makeSnapstore(pool.name, { + const store = makeSnapStore(pool.name, { ...tmp, ...path, ...fs, @@ -71,31 +90,48 @@ test('build temp file; compress to cache file', async t => { t.is(contents.toString(), 'abc', 'gunzip(contents) matches original'); }); -test('bootstrap, save, compress', async t => { - const vat = await bootWorker('ses-boot1', async m => m); +test(`create XS Machine, snapshot (${snapSize.raw} Kb), compress to ${snapSize.compression}x`, async t => { + const vat = await bootWorker('xs1', async m => m, '1 + 1'); t.teardown(() => vat.close()); const pool = tmp.dirSync({ unsafeCleanup: true }); t.teardown(() => pool.removeCallback()); await fs.promises.mkdir(pool.name, { recursive: true }); - const store = makeSnapstore(pool.name, { + const store = makeSnapStore(pool.name, { ...tmp, ...path, ...fs, ...fs.promises, }); - await vat.evaluate('globalThis.x = harden({a: 1})'); + const h = await store.save(async snapFile => { + await vat.snapshot(snapFile); + }); - /** @type {(fn: string, fullSize: number) => number} */ - const relativeSize = (fn, fullSize) => - Math.round((fs.statSync(fn).size / 1024 / fullSize) * 10) / 10; + const zfile = path.resolve(pool.name, `${h}.gz`); + t.is( + relativeSize(zfile, snapSize.raw), + snapSize.compression, + 'compressed snapshots are smaller', + ); +}); - const snapSize = { - raw: 858, - compression: 0.1, - }; +test('SES bootstrap, save, compress', async t => { + const vat = await bootSESWorker('ses-boot1', async m => m); + t.teardown(() => vat.close()); + + const pool = tmp.dirSync({ unsafeCleanup: true }); + t.teardown(() => pool.removeCallback()); + + const store = makeSnapStore(pool.name, { + ...tmp, + ...path, + ...fs, + ...fs.promises, + }); + + await vat.evaluate('globalThis.x = harden({a: 1})'); const h = await store.save(async snapFile => { await vat.snapshot(snapFile); @@ -103,25 +139,24 @@ test('bootstrap, save, compress', async t => { const zfile = path.resolve(pool.name, `${h}.gz`); t.is( - relativeSize(zfile, snapSize.raw), + relativeSize(zfile, snapSize.SESboot), snapSize.compression, 'compressed snapshots are smaller', ); }); -test('create, save, restore, resume', async t => { +test('create SES worker, save, restore, resume', async t => { const pool = tmp.dirSync({ unsafeCleanup: true }); t.teardown(() => pool.removeCallback()); - await fs.promises.mkdir(pool.name, { recursive: true }); - const store = makeSnapstore(pool.name, { + const store = makeSnapStore(pool.name, { ...tmp, ...path, ...fs, ...fs.promises, }); - const vat0 = await bootWorker('ses-boot2', async m => m); + const vat0 = await bootSESWorker('ses-boot2', async m => m); t.teardown(() => vat0.close()); await vat0.evaluate('globalThis.x = harden({a: 1})'); const h = await store.save(vat0.snapshot); @@ -137,7 +172,7 @@ test('create, save, restore, resume', async t => { }); // see https://github.com/Agoric/agoric-sdk/issues/2776 -test.failing('xs snapshots should be deterministic', t => { +test.failing('XS + SES snapshots should be deterministic', t => { const h = 'abc'; t.is('66244b4bfe92ae9138d24a9b50b492d231f6a346db0cf63543d200860b423724', h); });