feat: allow upload/retrieve of assets of arbitrary size from asset ca…

…nister (#1482)

Per the [design doc](https://github.com/dfinity/sdk/blob/master/docs/design/asset-canister.adoc):
1. Adds asset canister methods:
   - create_batch()
   - create_chunk()
   - commit_batch()
   - get()
   - get_chunk()

2. Reworks the asset installer in `dfx` to use these methods.  It can therefore upload assets that exceed the message ingress size.  Separate work (which this PR enables) will have to update `agent-js` to download these large assets.

Other than allowing the storage and retrieval of large assets, this PR does not address:
- multiple content types and content encodings:  in this PR `dfx` always uploads with `content-type: application/octet-stream` and `content-encoding: identity`
- smart/correct synchronization: in this PR `dfx` always uploads all assets (even those that have not changed), and still does not delete assets that were previously uploaded but no longer exist.
- All assets and chunks are uploaded in series.  See #1491 and dfinity/agent-rs#125
- The `store()`, `retrieve()`, and `list()` method signatures are unchanged for the time being, for backward compatibility.  Later work will remove the `retrieve()` method, change the semantics and parameters of `store()`, and change the parameters and return type of `list()`.

```
Uploading assets to asset canister...
  large-asset.bin 1/7 (1900000 bytes)
  large-asset.bin 2/7 (1900000 bytes)
  large-asset.bin 3/7 (1900000 bytes)
  large-asset.bin 4/7 (1900000 bytes)
  large-asset.bin 5/7 (1900000 bytes)
  large-asset.bin 6/7 (1900000 bytes)
  large-asset.bin 7/7 (1100000 bytes)
  index.js 1/1 (1218 bytes)
  sample-asset.txt 1/1 (24 bytes)
  index.js.map 1/1 (5625 bytes)
```

CHANGELOG.adoc

@@ -19,7 +19,15 @@ This commit also upgrades tokio and reqwest in order to work correctly. There ar
 
 Renamed the `project_name` in our own generated assets to `canister_name`, for things that are generated during canister build (and not project generation).
 
----
+== Asset Canister
+
+=== feat: The asset canister can now store assets that exceed the message ingress limit (2 MB)
+
+* Please note that neither the JS agent nor the HTTP server have been updated yet to server such large assets.
+* The existing interface is left in place for backwards-compatibility, but deprecated:
+** retrieve(): use get() and get_chunk() instead
+** store(): use create_batch(), create_chunk(), and commit_batch() instead
+** list(): use keys() instead
 
 = 0.6.25
 

distributed-canisters.nix

@@ -12,19 +12,19 @@ pkgs.runCommandNoCCLocal "distributed-canisters" {
 } ''
   mkdir -p $out
 
-  for canister_mo in ${distributed}/*.mo; do
-    canister_name=$(basename -s .mo $canister_mo)
+  for canister_dir in $(find ${distributed} -mindepth 1 -maxdepth 1 -type d); do
+    canister_name=$(basename $canister_dir)
 
     build_dir=$out/$canister_name
     mkdir -p $build_dir
 
     $moc/bin/moc \
-       $canister_mo \
+       $canister_dir/Main.mo \
        -o $build_dir/$canister_name.did \
        --idl \
        --package base $base
     $moc/bin/moc \
-       $canister_mo \
+       $canister_dir/Main.mo \
        -o $build_dir/$canister_name.wasm \
        -c --release \
        --package base $base

e2e/tests-dfx/assetscanister.bash

@@ -62,3 +62,47 @@ teardown() {
 it's cherry season
 CHERRIES" "$stdout"
 }
+
+@test 'can store arbitrarily large files' {
+    [ "$USE_IC_REF" ] && skip "skip for ic-ref" # this takes too long for ic-ref's wasm interpreter
+
+    install_asset assetscanister
+
+    dfx_start
+    dfx canister create --all
+    dfx build
+    dfx canister install e2e_project_assets
+
+    dd if=/dev/urandom of=src/e2e_project_assets/assets/large-asset.bin bs=1000000 count=6
+
+    dfx deploy
+
+    assert_command dfx canister call --query e2e_project_assets get '(record{key="/large-asset.bin";accept_encodings=vec{"identity"}})'
+    assert_match 'total_length = 6_000_000'
+    assert_match 'content_type = "application/octet-stream"'
+    assert_match 'content_encoding = "identity"'
+
+    assert_command dfx canister call --query e2e_project_assets get_chunk '(record{key="/large-asset.bin";content_encoding="identity";index=2})'
+
+    assert_command dfx canister call --query e2e_project_assets get_chunk '(record{key="/large-asset.bin";content_encoding="identity";index=3})'
+    assert_command_fail dfx canister call --query e2e_project_assets get_chunk '(record{key="/large-asset.bin";content_encoding="identity";index=4})'
+}
+
+@test "list() and keys() return asset keys" {
+    install_asset assetscanister
+
+    dfx_start
+    dfx canister create --all
+    dfx build
+    dfx canister install e2e_project_assets
+
+    assert_command dfx canister call --query e2e_project_assets list
+    assert_match '"/binary/noise.txt"'
+    assert_match '"/text-with-newlines.txt"'
+    assert_match '"/sample-asset.txt"'
+
+    assert_command dfx canister call --query e2e_project_assets keys
+    assert_match '"/binary/noise.txt"'
+    assert_match '"/text-with-newlines.txt"'
+    assert_match '"/sample-asset.txt"'
+}