Use Endo archives between bundleSource and importBundle

[kriskowal]

What is the Problem Being Solved?

Agoric uses bundleSource and importBundle for contract deployment. Currently, the getExports (legacy) and nestedEvaluate bundle formats do not support least-authority linkage. The new endoZipBase64 bundle format provides a foundation for eventually supporting least-authority linkage. We are collaborating with Consensys to add LavaMoat policies and linkage.

Toward this goal, we will migrate all uses of bundleSource in Agoric-SDK to use endoZipBase64 as the new default archive format.

Description of the Design

To get there, endoZipBase64 needs to have feature parity with nestEvaluate, including the parser-based evasive transforms for SES censorship rules. SES uses a primitive regular expression to prevent client code from reaching for dynamic import and other unsafe features like HTML comments. Many false positives are avoided by transforming import() calls within comments into something else. (In the future, import as a method can be changed from object.import() to object'import' automatically for a good effect.)

It transpires that the existing transformer in bundleSource is asynchronous and cannot be driven by the synchronous transforms supported by SES Compartments. Endo’s Compartment Mapper will need a new utility for providing asynchronous transforms.

Security Considerations

We will also need to find a way to make the Endo Compartment Mapper runtime freeze globalThis after injecting endowments. Endo Compartment Mapper does not currently depend on running under SES because it also serves as a demonstration of Compartments as module loaders that might not necessarily entrain lockdown or harden.

Test Plan

The existing tests should be sufficient since they already exercise bundleSource extensively. If we can swap the default bundle format without breaking tests, we should be golden.

Integration

RESM, NESM, & SESM

Much rests on our ability to create a package that can be executed either with Node.js ESM (NESM) or emulated node -r esm (RESM) and in some cases also packaged for use on chain as a Zip file (SESM). The RESM/NESM migration is tracked at #527, and hinges on the ability of a RESM package to import from a NESM package with our fork of RESM https://github.com/agoric-labs/esm.

RESM and NESM are mutually exclusive within the same package for multiple reasons, one of which is that NESM requires a "type": "module" in package.json and RESM requires the opposite when running on versions of Node.js that also support NESM.

In order to break the log jam, SESM will respect "type": "module" if present but will read "parsers": {"js": "mjs"} instead if it is present. This allows us to Zip up packages that also need to support RESM.

No export aliases

For packages that have multiple entry points, like @endo/comaprtment-mapper/archive.js vs @endo/compartment-mapper/import-archive.js, it is necessary to make sure those entry points are usable both before and after Node.js added NESM support and the exports property. The exports property allows modules to be aliased in their public interface, but for a module to be usable both with and without support for exports, the public name must be the same as the internal name, and that means that it must include the extension and should be at the root of the package.

RESM will support the "module" property in a package.json if present, and fall back to "main". NESM will only support the "main" property. So, for a package to be usable for either purpose, the "main" property must be used for an ESM module. This eliminates the possibility of a package supporting the combination of RESM, NESM, and legacy Node.js that only supports CommonJS. Without RESM, the combination of NESM and Node.js CJS is possible to support using "main" to import a compiled CommonJS build product from the ESM sources, and use "exports" to alias the ESM sources.

Babel and StaticModuleRecord

Much also rests on using Babel only when Babel is necessary, and initializing Babel before SES lockdown. Our module transform depends on Babel-Standalone, which itself does not initialize properly after lockdown because of the property override hazard. Babel is also a 3MB codebase that overwhelms the on-chain execution environment.

We’ve decoupled StaticModuleRecord from SES and factored Compartment Mapper so you can depend upon a slice of that package that only takes on the Babel dependency if strictly necessary.

Consequently, a contract can safely use importBundle from @agoric/import-bundle without taking on a dependency on Babel, but any deploy script that uses @agoric/bundle-source must import @agoric/babel-standalone before running SES lockdown (importing @agoric/install-ses or any of its wrappers).

[kriskowal]

At this time, Endo archives appear to have correct behavior but inadequate performance. This change is now dependent on endojs/endo#655.