Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support to forc-pkg for ipfs-sourced dependencies #3926

Closed
mitchmindtree opened this issue Jan 30, 2023 · 0 comments · Fixed by #4299
Closed

Add support to forc-pkg for ipfs-sourced dependencies #3926

mitchmindtree opened this issue Jan 30, 2023 · 0 comments · Fixed by #4299
Assignees
@mitchmindtree @kayagokalp
Labels
enhancement New feature or request forc forc-pkg Everything related to the `forc-pkg` crate. forc-registry Everything to do with forc-registry; IPFS sourcing, package registery P: high Should be looked at if there are no critical issues left

Comments

@mitchmindtree
Copy link
Contributor

This is one of the first implementation steps in the new package registry design discussed in #3752.

IPFS dependencies will be the first remote dependency alternative to git supported by forc-pkg.

Its usage should look something like the following:

foo = { ipfs = "QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG" }

where QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG is a CID for a directory that contains the package foo.

forc-pkg's pkg.rs module is getting a bit cluttered - it's likely worth first splitting up the different Source kinds into their own modules before adding in ipfs support. I'll open a dedicated issue for this.
@mitchmindtree mitchmindtree added enhancement New feature or request forc P: high Should be looked at if there are no critical issues left forc-pkg Everything related to the `forc-pkg` crate. labels Jan 30, 2023
@mitchmindtree mitchmindtree self-assigned this Jan 30, 2023
@mitchmindtree mitchmindtree mentioned this issue Jan 30, 2023
Closed
@mitchmindtree mitchmindtree removed their assignment Jan 30, 2023
@mitchmindtree mitchmindtree mentioned this issue Jan 30, 2023
Open
7 tasks
@mitchmindtree mitchmindtree mentioned this issue Feb 23, 2023
Merged
7 tasks
mitchmindtree added a commit that referenced this issue Feb 23, 2023
@mitchmindtree @JoshuaBatty
refactor(forc-pkg): Abstract out source-related items from pkg (#…
9346bf4 
…4168)

## Description

In anticipation of the upcoming ipfs support (#3926), this PR refactors
`forc-pkg` to more clearly separate out the "source" abstraction from
the rest of the pkg-handling logic.

We do so by creating a common interface for sources in the form of
traits for pinning and fetching. This should allow for adding in the
upcoming ipfs implementation (or future alternative source kind) in a
manner that doesn't further complicate the rest of package handling.

There is still room for further refactor of `pkg.rs`, but this should do
enough to clear the way for ipfs support and closes #3927.

## Checklist

- [x] I have linked to any relevant issues.
- [x] I have commented my code, particularly in hard-to-understand
areas.
- [x] I have updated the documentation where relevant (API docs, the
reference, and the Sway book).
- [x] I have added tests that prove my fix is effective or that my
feature works.
- [x] I have added (or requested a maintainer to add) the necessary
`Breaking*` or `New Feature` labels where relevant.
- [x] I have done my best to ensure that my PR adheres to [the Fuel Labs
Code Review
Standards](https://github.com/FuelLabs/rfcs/blob/master/text/code-standards/external-contributors.md).
- [x] I have requested a review from the relevant team or maintainers.

Co-authored-by: Joshua Batty <joshpbatty@gmail.com>
@mitchmindtree mitchmindtree self-assigned this Feb 26, 2023
mitchmindtree added a commit that referenced this issue Mar 16, 2023
@mitchmindtree
feature(forc-pkg): Support IPFS source (WIP)
b88dc0d 
Currently very WIP.

Closes #3926.

TODO:

- [ ] Currently no good go-ipfs-compatible UnixFS support in Rust.
      For now, we can just compress the package directory and insert
      that.
- [ ] Finish the `pin`, `fetch` and `dep_path` implementations.
- [ ] Add tests.
@mitchmindtree mitchmindtree mentioned this issue Mar 16, 2023
Merged
10 tasks
@ra0x3 ra0x3 mentioned this issue Apr 10, 2023
Closed
@kayagokalp kayagokalp self-assigned this May 1, 2023
@kayagokalp kayagokalp added the forc-registry Everything to do with forc-registry; IPFS sourcing, package registery label Jun 19, 2023
kayagokalp added a commit that referenced this issue Jun 19, 2023
@mitchmindtree @kayagokalp
feature(forc-pkg): Support IPFS-sourced dependencies (#4299)
96c6103 
## Description
Closes #3926.

(edited by @kayagokalp to reflect latest status of the PR)

This PR adds IPFS support for forc-pkg. To do so, forc relies on
existing ipfs local node without embedding a node to forc itself. If
local system does not have an ipfs node running, a public gateway
(https://ipfs.io) is used as a fallback mechanism and packages are
fetched through that.

## Some Context and Future Work

For our package registry stuff to work properly we will eventually need
to host an IPFS node to make sure that our published packages are pinned
by at least one node in the network. That is the case as long as we
don't have a way to incentivize people pinning the packages they
published. What will happen is that a package will be published by a
developer and after we detect the published package our own IPFS node
will pin it to make sure it is always accessible.

Also some benchmarks are done for public gateway during development. It
looks like for initial fetch operations from public gateway spends some
time looking for the package pinned by the IPFS node simulating our
incoming fuel IPFS node which is explained above. It is manageable but
fetching from the fuel IPFS node is instant as it already got the
package pinned. We can consider falling back to our own node's gateway
api rather than a public one to smooth the process.

Finally maybe we can explore embedded ipfs node option as a follow-up,
due to the status of IPFS with Rust, it is not very obvious (maybe not
event possible atm) how to be able to be fully compatible with kubo, for
fetching/publishing our packages. But this is still an open question
worth exploring.

## Testing

### To test with a local node:

1. You can install kubo, you can follow the instructions in their
website while installing https://docs.ipfs.tech/install/command-line/
2. Run the `ipfs daemon` service with `ipfs daemon`
3. Add an ipfs source, I have a package already pinned by my IPFS node
that you can use:
```toml
[dependencies]
test_lib = { ipfs = "QmfZ3uH7dFEDkYN5RQfyu4m7L8uk8kGiLkNwzHqsrormSj" }
```
4. Run `forc build`

### To test public gateway fallback:

1. Either do this before starting `ipfs daemon` or stop your local
daemon with `ipfs shutdown`.
Add an ipfs source, I have a package already pinned by my IPFS node that
you can use:
```toml
[dependencies]
test_lib = { ipfs = "QmfZ3uH7dFEDkYN5RQfyu4m7L8uk8kGiLkNwzHqsrormSj" }
```
2. Run `forc build`


---------

Co-authored-by: kayagokalp <kaya.gokalp@fuel.sh>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mitchmindtree mitchmindtree

@kayagokalp kayagokalp

Labels
enhancement New feature or request forc forc-pkg Everything related to the `forc-pkg` crate. forc-registry Everything to do with forc-registry; IPFS sourcing, package registery P: high Should be looked at if there are no critical issues left
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feature(forc-pkg): Support IPFS-sourced dependencies FuelLabs/sway
2 participants
@mitchmindtree @kayagokalp