Implement wit pull

[lann]

This implements a wit pull subcommand which can be used to update local e.g. wit/deps/*.wit from a registry.

$ wit pull wasi:io --create-dirs
   Resolving package wasi:io
  Downloaded release wasi:io@0.2.0
       Wrote package wasi:io@0.2.0 to 'wit/deps/wasi-io@0.2.0.wit'

$ head -n1 wit/deps/wasi-io@0.2.0.wit
package wasi:io@0.2.0;

$ wit pull wasi:http
   Resolving package wasi:http
  Downloaded release wasi:http@0.2.0
       Wrote package wasi:clocks@0.2.0 to 'wit/deps/wasi-clocks@0.2.0.wit'
       Wrote package wasi:random@0.2.0 to 'wit/deps/wasi-random@0.2.0.wit'
       Wrote package wasi:cli@0.2.0 to 'wit/deps/wasi-cli@0.2.0.wit'
       Wrote package wasi:http@0.2.0 to 'wit/deps/wasi-http@0.2.0.wit'

There are two modes of operation:

• When given specific name(s) it will pull those registry package(s) and unpack all (not existing) WIT packages into wit/deps/<namespace>-<name>@<version>.wit
• When given no name(s) it will check the packages under wit/ for missing dependencies and attempt to pull them like above

Both of these can operate whether or not wit_parser::Resolve::push_dir can currently parse the wit/ directory. If it can parse, the existing dependency checks will just be more accurate (capturing e.g. wit/deps/some-package.wasm). This ought to be more consistent but really needs some refactoring in wit-parser first.

[lann]

There is still much room for improvement but I think its useful enough as-is to iterate on.

[lann]

One open question is what to do with the warg-loader crate. At this point it could make sense to merge that into this repo.

[rylev]

Seems like there's an issue with fetching direct dependencies that we should fix before merging, but overall this looks good.

I am getting errors when trying this out myself, presumably because I don't have access to the bytecodealliance.org registry. I think we'll want to figure out the defaults around registries as well before merging.

[lann]

I am getting errors when trying this out myself, presumably because I don't have access to the bytecodealliance.org registry.

The BA registry should be public, e.g. https://github.com/orgs/bytecodealliance/packages/container/package/warg%2Fwasi%2Fio

[rylev]

This is the error I'm seeing when running the command against a wit world that uses a wasi:http dependency.

error: OCI error: Authentication failure: {"errors":[{"code":"DENIED","message":"denied"}]}


Caused by:
    Authentication failure: {"errors":[{"code":"DENIED","message":"denied"}]}

[rylev]

I'm curious, when would this happen? What is the user supposed to do with this information?

[lann]

Currently nothing enforces that the outer registry version (Warg release version or OCI tag) matches the inner WIT package version. This is a problem here because we only have the WIT version for the "already installed" set and can only use the registry version to resolve packages to pull.

I'm not sure how to resolve this yet, so the warning serves primarily to alert publishers that something is wrong.

[rylev]

Nit: I'm not sure if present is the best term here. Perhaps resolved? 🤷‍♂️

[lann]

"Resolved" is kind of overloaded in this context. This was referred to as "existing" before but that seems even worse. 🤷

[rylev]

When I remove ghcr.io credentials by running docker logout ghcr.io then I end up with a different error:

error: failed to get registry credentials: Credential helper returned non-zero response code:
stdout:
credentials not found in native keychain


stderr:

And if I log back in with docker login ghcr.io then everything works as expected.

So it seems we're not allowing anonymous access to the ghcr instance and we're not handling the error case of when the credentials are bad.

I see you've opened lann/warg-loader#5 as well which should hopefully address this.

[lann]

I made 2 changes in warg-loader which should fix the above auth issues:

• Tweaked auth helper error handling to treat a non-zero exit code as "no credential found"; this is apparently what is expected 🤷
• Retry anonymous auth after credential failure, to catch the case where an expired token is used with a public registry

[rylev]

Not sure how I'd test an expired token, but this seems to have fixed the issue when no token is present. I think this is ready to merge!

[lann]

Not sure how I'd test an expired token

I tested by corrupting a token in ~/.docker/config.json which seems to be the same error path as expiration.

[Zelzahn]

If I'm understanding it correctly, this will not serve as a replacement for wit-deps?

[lann]

If I'm understanding it correctly, this will not serve as a replacement for wit-deps?

It might eventually. Part of the purpose of this tool is to be a proving ground for some of the more advanced import definition types now available in the component model which should be able to replace external dependency config files.

See "options for naming imports" here: https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md#import-and-export-definitions

[rylev]

I believe this PR was closed by mistake by #270. I believe the correct issue to close was #268.

@peterhuene can you reopen this PR?

[peterhuene]

Whoops, sorry about that!

[calvinrp]

Closing this PR. The wit CLI functionality is in the process of moving over to https://github.com/bytecodealliance/wasm-pkg-tools