Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upload asset chunks concurrently #1491

Closed
ericswanson-dfinity opened this issue Mar 10, 2021 · 0 comments
Closed

Upload asset chunks concurrently #1491

ericswanson-dfinity opened this issue Mar 10, 2021 · 0 comments
Assignees
@ericswanson-dfinity

Comments

@ericswanson-dfinity
Copy link
Member

ericswanson-dfinity commented Mar 10, 2021 •
edited
Loading

This was not possible through the bootstrap, for two reasons:

  • the bootstrap only allowed one connection (though this still allowed multiple requests to execute within a single block)
  • idle connections leaving agent in a state where all requests failed with a "channel closed" error after a http/408 timeout response

It's uncertain whether this will be possible post-bootstrap, but somehow, someday, we need to be able to upload more than one chunk per block.

Splitting up calls to call_and_wait() into many call() before waiting for the requests to finish would bypass the idle connection issue.
@ericswanson-dfinity ericswanson-dfinity mentioned this issue Mar 10, 2021
Merged
mergify bot pushed a commit that referenced this issue Mar 12, 2021
@ericswanson-dfinity
feat: allow upload/retrieve of assets of arbitrary size from asset ca…
464fd35 
…nister (#1482)

Per the [design doc](https://github.com/dfinity/sdk/blob/master/docs/design/asset-canister.adoc):
1. Adds asset canister methods:
   - create_batch()
   - create_chunk()
   - commit_batch()
   - get()
   - get_chunk()

2. Reworks the asset installer in `dfx` to use these methods.  It can therefore upload assets that exceed the message ingress size.  Separate work (which this PR enables) will have to update `agent-js` to download these large assets.

Other than allowing the storage and retrieval of large assets, this PR does not address:
- multiple content types and content encodings:  in this PR `dfx` always uploads with `content-type: application/octet-stream` and `content-encoding: identity`
- smart/correct synchronization: in this PR `dfx` always uploads all assets (even those that have not changed), and still does not delete assets that were previously uploaded but no longer exist.
- All assets and chunks are uploaded in series.  See #1491 and dfinity/agent-rs#125
- The `store()`, `retrieve()`, and `list()` method signatures are unchanged for the time being, for backward compatibility.  Later work will remove the `retrieve()` method, change the semantics and parameters of `store()`, and change the parameters and return type of `list()`.

```
Uploading assets to asset canister...
  large-asset.bin 1/7 (1900000 bytes)
  large-asset.bin 2/7 (1900000 bytes)
  large-asset.bin 3/7 (1900000 bytes)
  large-asset.bin 4/7 (1900000 bytes)
  large-asset.bin 5/7 (1900000 bytes)
  large-asset.bin 6/7 (1900000 bytes)
  large-asset.bin 7/7 (1100000 bytes)
  index.js 1/1 (1218 bytes)
  sample-asset.txt 1/1 (24 bytes)
  index.js.map 1/1 (5625 bytes)
```
@ericswanson-dfinity ericswanson-dfinity self-assigned this Apr 19, 2021
@dfinity dfinity locked and limited conversation to collaborators Oct 4, 2022
@sesi200 sesi200 converted this issue into discussion #2644 Oct 4, 2022

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Assignees

@ericswanson-dfinity ericswanson-dfinity

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ericswanson-dfinity