Add support for registries (e.g. inspired by crates.io) and semver-based dependencies.

[mitchmindtree]

This is mentioned in #27, but I've opened this as a dedicated place for discussion.

---

Motivation

Having a package "registry" or "index" is useful for package sharing, search-ability, discoverability and allows for standardising semver for remote dependencies (as a convenient alternative to git branches or tags).

Features

A registry would consist of something like the following:

• A dedicated, immutable store (db/directory/repository/ipfs) for all packages.
• A searchable index with categories, package descriptions, etc (+ a fancy website frontend)?
• Ability for users to publish to the registry via forc publish. Note: Requires authentication, size limits, etc.
• Ability to specify dependencies via <name> = <semver> with implicit registry URL. E.g.

  [dependencies]
  foo = "1.2.3"

• Ability for specifying custom registries.

where the right hand side of = is a valid semver version.

Possible Approaches

Using git

I imagine registry-based dependencies could be a simple wrapper around git, where forc (and the registry repository) automatically manage relevant branches and tags for semver versioning and perform all fetching and publishing via git.

E.g. if version 1.2.3 is specified, a v1.2 branch could be used where each patch release is a tagged commit on that branch (e.g. v1.2.0, v.1.2.1, etc), and in this case the v1.2.3 would represent the tag of the minimum commit that must be used on that branch. Calls to forc update however, would automatically allow for grabbing the head commit of the branch (as introduced by #825), or in other words the latest patch release (matching the cargo update behaviour).

Consider "decentralised" approach to package storage? IPFS?

I wonder if we could take advantage of IPFS for hosting the pinned source, rather than storing every package in some central location and running the risk of ecosystem-wide outages like we get with crates.io once every few months? Looks like IPFS have some docs on git+ipfs.

If we were to go down this route, perhaps our registry could act purely as an index for searchability/discoverability, and as a mapping from <name> = <semver> to some content address during the dependency pinning step. Registry entries in the Forc.lock could simply be the IPFS content address. In this case, the centralised registry would still be required for pinning packages (on the initial forc build or forc update), however all subsequent builds for projects that already have a Forc.lock could continue to work without the registry, as long as there are other IPFS nodes up that host the src for each package in use.

forc publish would create the registry mapping, upon which we could also host the src under our own ipfs node to make sure each registry package has at least one hosted instance at all times. Mirroring the registry would just be a matter of running another ipfs node somewhere else.

Alternatives

Maintaining a registry like this is arguably quite a bit of work and a large responsibility for fuellabs to take on, particularly around maintaining a centralised index, making sure it's secure / always available, maintaining a dedicated website/front-end, etc. We want to be sure that it's worth taking this on, particularly when git dependencies are already quite easy to use and are now well supported.

We may want to consider other solutions to searchability and discoverability. E.g. one alternative that comes to mind is to do something similar to openframeworks' addons ecosystem. They have a bot that periodically scrapes GitHub searches with particular keywords and directory structures, and automatically makes them available and searchable via their site. We could potentially do something similar for git repositories with Forc.tomls in them? This runs the risk of creating too much dependency on GitHub, but with the benefit that GitHub is handling authentication, src storage, file limits, etc so that we don't have to.

[adlerjohn]

Is there really a huge benefit to a registry over just using Git URLs? It would be a lot of work as you say to set up, for what I would argue isn't that big of a benefit, so should probably be left to later.

[mitchmindtree]

Benefits of a registry over git dependencies that come to mind are:

• More concise dependency declarations.
• Being able to meaningfully use semver in a manner that the Forc.lock file and forc update are aware of.
• Having a single "guaranteed" immutable store for package source, i.e. no-one can delete their repo like on github.
• Users able to publish packages in a manner that makes them discoverable, searchable and opens up the avenue for auto-documentation generation (like docs.rs).

I'm with you though in that, while all of these would be nice, none of these are particularly essential right now and we have a lot of other greater priorities. We can always revisit this as the ecosystem begins to grow and we begin to see bigger dependency graphs than just depending on std, core and some other local libs :)

[mitchmindtree]

A whole chapter on cargo's Registries here for inspiration: https://doc.rust-lang.org/cargo/reference/registries.html

[Br1ght0ne]

Not directly related to registries, but having Nix build tooling for Sway projects (like rustPlatform.buildRustPackage or naersk for Rust/Cargo) could allow hosting sway dependencies similarly to what fuel.nix does with Fuel software.

Upsides:

• reuse existing package management tooling (Nix the package manager) and hosting infrastructure (Nix caches)
• possibility for easier integration with decentralized technologies like IPFS (mentioned above) as Nix builds are content-addressable (or having a fixed output hash at the very least)
• possibility for user-managed and addressable hosting (using the likes of Cachix)

Downsides:

• forc install and the likes will depend on nix, and we can't bundle Nix, only install it out-of-band
• not sure on how to only use Forc.{toml,lock} files without any user-facing Nix scaffolding
• maintaining Nix scaffolding for Sway probably requires advanced Nix knowledge :(

I feel like rolling our own package registry is the better call, but I'll leave these notes here for posterity.

[mitchmindtree]

@Br1ght0ne I have plans for hosting something like buildForcPackage at the fuel.nix repo itself, as I agree it'd be nice to offer full reproducibility from the OS-level and up for those who are serious about it.

It would be nice if we could lean into Nix a bit more in general :) Unfortunately, the Nix UX is still in a state of flux, especially with flakes not yet being stable and documentation being all over the place. As a result, it's probably best to offer Nix support (e.g. from fuel.nix), but not require that users use it.

Very keen to play with Nix's upcoming IPFS and CA features and see what we can learn.

[Br1ght0ne]

@mitchmindtree I created a (very very WIP) nix flake for building sway projects at https://github.com/Br1ght0ne/sway.nix, roughly based on https://github.com/nix-community/naersk (and fuel.nix, of course 🫡).

For now I'm looking at a 3-stage process:

• fetching and building Forc [dependencies] in a dummy project (impure)
• building the Rust project that provides forc test support (using naersk)
• building the actual project Sway code (should be pure, but WIP)

Let me know what you think!

[mitchmindtree]

I've opened a new issue with a much more thorough design proposal at #3752 - feedback appreciated!

[mitchmindtree]

Closing in favour of #3752.