Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create helia-remote-pinning example #147

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

SgtPooki
Copy link
Contributor

@SgtPooki SgtPooki commented Sep 14, 2023

  • feat: start helia-remote-pinning-example
  • feat: add node & browser pinning example

Note: the version of @helia/remote-pinning being used here is currently in PR at ipfs/helia-remote-pinning#13

Browser example

NOTE: the browser example does not succeed with web3.storage. There are two problems:

  1. They do not seem to be dialing our origins, or we do not seem to be receiving their dials.
  2. The returned delegates are not reachable from the browser

I've tested all the main pinning services I know to try, and they all have errors of some sort:

  1. web3.storage
    • does not seem to dial our p2p-circuit multiaddrs to pull the data
    • Does not return browser-dialable delegates
  2. estuary
    • authentication is failing. our ipfs-webui CI is also failing to pin using estuary. The service may be down, or something changed and I need to update our tokens?
    • Estuary is no longer supported/maintained.
  3. Pinata
  4. crust.io
    • setup is a pain and authentication is failing.
  5. 4EVERLAND
    • returns single delegate: '/dnsaddr/node-1.ipfs.4everland.net/p2p/12D3KooWQ85aSCFwFkByr5e3pUCQeuheVhobVxGSSs1DrRQHGv1t' that throws an error on unsupported multiaddr
    • pinning moves from queued to pinning but doesn't seem to ever succeed.
  6. scaleway
    • pinning-service-client is CORS failure and also returning 405 for OPTIONS request that is sent as pre-flight request (pinged @vmscw on slack)
  7. filebase
    • TBD

Node example

The node example seems to work without issue (because tcp transports are available in nodejs)

Pinning Service Success? AgentVersion of delegates Delegates returned
web3.storage Yes kubo/0.18.1/675f8bd/docker '/ip4/54.68.186.137/tcp/4001/p2p/12D3KooWCw2pRmt1J8mLXRm7HQEVgqfWj3Njs7SkDimPLeyQyATK' '/ip4/54.68.186.137/udp/4001/quic-v1/p2p/12D3KooWCw2pRmt1J8mLXRm7HQEVgqfWj3Njs7SkDimPLeyQyATK' '/ip4/54.68.186.137/udp/4001/quic/p2p/12D3KooWCw2pRmt1J8mLXRm7HQEVgqfWj3Njs7SkDimPLeyQyATK'
pinata No kubo/0.18.1/675f8bd/docker '/dns4/pinning-service-1.ipfs-swarm.use1.pinata.cloud/tcp/4001/p2p/12D3KooWFxvYLyabpcDtDLYd16nYygG3YS2FV6GYU9ut6M3rr5We'
crust.io TBD TBD TBD
4everland No go-ipfs/0.11.0/67220ed '/dnsaddr/node-1.ipfs.4everland.net/p2p/12D3KooWQ85aSCFwFkByr5e3pUCQeuheVhobVxGSSs1DrRQHGv1t'
scaleway No Error: connect QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt failure: failed to dial: no good addresse [1m13.599s] '/dnsaddr/delivery.ipfs.labs.scw.cloud/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt'
filebase Sometimes (more than a few get stuck in pinning status for > 30m) kubo/0.19.2/afb27ca/filebase '/ip4/15.204.161.17/tcp/4001/p2p/12D3KooWErRwP21vdVD7rkaivnMfxXAnXs3djX4D5JNfzAYFmrCM' '/ip4/141.95.85.123/tcp/4001/p2p/12D3KooWNWgNz5QSL2D2YxfKcYo6HyEpFuRnW2fRYewrDmjWJECi' '/ip4/198.244.201.187/tcp/4001/p2p/12D3KooWC9L4RjPGgqpzBUBkcVpKjJYofCkC5i5QdQftg1LdsFb2'

NOTE: when running npm run start-node against filebase, retries needs to be bumped up so that the node is still running when filebase kicks off their task of pulling the content. Joshua Noble on slack "Pinning is not immediate, it goes into a queue."

Tests

Not started yet.. I think in order to test this properly, we will need to stand up more than a mock pinning provider that involves delegate and origin dials.

@SgtPooki SgtPooki linked an issue Sep 14, 2023 that may be closed by this pull request
@SgtPooki
Copy link
Contributor Author

@achingbrain can you help me figure out how to log incoming dial attempts for the browser example? I can't seem to get it to succeed unless the pinning provider is already aware of the CID (via npm run start-node -- "some string") and then running the browser example with the same string.

@achingbrain
Copy link
Contributor

Something like this?

helia.libp2p.addEventListener('connection:open', (evt) => {
  const conn = evt.detail

  if (conn.direction === 'inbound') {
    // .. some code here
  }
})

@SgtPooki
Copy link
Contributor Author

@achingbrain ok so that will listen for new connections, but I doubt we will get new inbound connections to the browser (currently) because only webrtc can provide that functionality and that doesn't exist in (m)any backend IPFS nodes yet.

Does any event fire when a connection comes through on an already open connection with a p2p-circuit relay? I imagine this is where we need to listen for connection attempts to helia browser nodes.

@BigLep
Copy link

BigLep commented Sep 15, 2023

@SgtPooki : Thanks for doing this. I'll look more a bit later as well, but some initial thoughts:

  1. Thanks for summarizing the situation. Rough that none work...
  2. We should remove Estuary. It is no longer supported/maintained.
  3. Maybe try https://filebase.com/ipfs-pinning/ ?
  4. I would reach out to these groups and ask how they test their service. You can also point to your example.

@SgtPooki
Copy link
Contributor Author

SgtPooki commented Sep 15, 2023

@BigLep No problem.. happy to atleast be narrowing the focus for ipfs/helia#256

We should remove Estuary. It is no longer supported/maintained.

ACK

Maybe try https://filebase.com/ipfs-pinning/ ?

Failing from nodejs :( -- I just created a table with all pinning services and only web3.storage is succeeding from nodejs currently.

Some note-able callouts:

  • web3.storage is the only successful pinning service from NodeJS in this example.
  • pinata can be connected to, but pinning doesn't succeed. Also, their handling of "origins" is flagging valid multiaddrs as invalid, so i have to explicitly filter out all multiaddrs containing quic
  • 4everland can be connected to, but pinning doesn't succeed. They're running a very old version of go-ipfs
  • scaleway & filebase: pinning providers' returned delegates cannot be connected to from Helia nor the latest kubo version, nor ipfs-shipyard/vole
  • crust.io requires web3 wallet and i need new credentials to investigate further

I would reach out to these groups and ask how they test their service. You can also point to your example.

With most pinning services leaving something to be desired, I will reach out to them all, point them to this issue, and start work on other tasks. I will consider ipfs/helia#256 blocked until we have some pinning services with webtransport.

@BigLep
Copy link

BigLep commented Sep 15, 2023

Thanks for the update here @SgtPooki. This is tough to see, but at least we're engaged with reality.

A couple of things:

  1. I believe there are shared slack channels with most/all of these or that ecosystem/outercore can help us get in touch. It's probably worth requesting connections to these groups in #ipfs-ecosystem.
  2. Is pinning service compliance (https://github.com/ipfs-shipyard/pinning-service-compliance ) showing similar issues that multiple providers don't work (either from the browser or nodejs)?

@SgtPooki
Copy link
Contributor Author

I believe there are shared slack channels with most/all of these or that ecosystem/outercore can help us get in touch. It's probably worth requesting connections to these groups in #ipfs-ecosystem.

Will request in ipfs-ecosystem.

Is pinning service compliance (https://github.com/ipfs-shipyard/pinning-service-compliance ) showing similar issues that multiple providers don't work (either from the browser or nodejs)?

it's showing failures in estuary for pinning content, and crust.io authentication errors, but other services are showing successes primarily because we're using inlineCids. So data retrieval through the IPFS network is not required. IIRC, pinning-service-compliance was primarily intended to validate that services were implementing the pinning-services spec properly, not that the pinning services themselves worked well when doing IPFS network connectivity.

@SgtPooki
Copy link
Contributor Author

SgtPooki commented Sep 15, 2023

slack threads or github issues with pinning providers & further discussions.

I will link any issues/tickets that come up, but for now, here are some slack threads where I'm trying to get in touch with the teams:

Service Link
#ipfs-ecosystem help request https://filecoinproject.slack.com/archives/C05PGBP697E/p1694802122998339
web3.storage https://filecoinproject.slack.com/archives/C027XP5BTGB/p1694708735768809
Pinata https://filecoinproject.slack.com/archives/C015WB84K4M/p1694800671205179
crust.io crustio/ipfs-w3auth-pinning-service#36
Scaleway https://filecoinproject.slack.com/archives/C03KQ8MC62Y/p1694802383522219?thread_ts=1694512602.404439&cid=C03KQ8MC62Y
4EVERLAND TBD
filebase https://filecoinproject.slack.com/archives/C025RV6JLLB/p1694820067603299

@SgtPooki
Copy link
Contributor Author

note: filebase did some updates and the returned delegates are now dialable:

╭─    ~/code/work/protocol.ai/ipfs-shipyard/vole 
╰─ ✔ ❯ ./vole libp2p identify /ip4/15.204.161.17/tcp/4001/p2p/12D3KooWErRwP21vdVD7rkaivnMfxXAnXs3djX4D5JNfzAYFmrCM
PeerID: "12D3KooWErRwP21vdVD7rkaivnMfxXAnXs3djX4D5JNfzAYFmrCM"
Protocol version: "ipfs/0.1.0"
Agent version: "kubo/0.19.2/afb27ca/filebase"
Listen addresses:
	- "/ip4/15.204.161.17/tcp/4001/p2p/12D3KooWErRwP21vdVD7rkaivnMfxXAnXs3djX4D5JNfzAYFmrCM"
Protocols:
	- "/ipfs/bitswap"
	- "/ipfs/bitswap/1.0.0"
	- "/ipfs/bitswap/1.1.0"
	- "/ipfs/bitswap/1.2.0"
	- "/ipfs/id/1.0.0"
	- "/ipfs/id/push/1.0.0"
	- "/ipfs/kad/1.0.0"
	- "/ipfs/lan/kad/1.0.0"
	- "/ipfs/ping/1.0.0"
	- "/libp2p/circuit/relay/0.2.0/stop"
	- "/x/"

╭─    ~/code/work/protocol.ai/ipfs-shipyard/vole
╰─ ✔ ❯ ./vole libp2p identify /ip4/141.95.85.123/tcp/4001/p2p/12D3KooWNWgNz5QSL2D2YxfKcYo6HyEpFuRnW2fRYewrDmjWJECi
PeerID: "12D3KooWNWgNz5QSL2D2YxfKcYo6HyEpFuRnW2fRYewrDmjWJECi"
Protocol version: "ipfs/0.1.0"
Agent version: "kubo/0.19.2/afb27ca/filebase"
Listen addresses:
	- "/ip4/141.95.85.123/tcp/4001/p2p/12D3KooWNWgNz5QSL2D2YxfKcYo6HyEpFuRnW2fRYewrDmjWJECi"
Protocols:
	- "/ipfs/bitswap"
	- "/ipfs/bitswap/1.0.0"
	- "/ipfs/bitswap/1.1.0"
	- "/ipfs/bitswap/1.2.0"
	- "/ipfs/id/1.0.0"
	- "/ipfs/id/push/1.0.0"
	- "/ipfs/kad/1.0.0"
	- "/ipfs/lan/kad/1.0.0"
	- "/ipfs/ping/1.0.0"
	- "/libp2p/circuit/relay/0.2.0/stop"
	- "/x/"

╭─    ~/code/work/protocol.ai/ipfs-shipyard/vole 
╰─ ✔ ❯ ./vole libp2p identify /ip4/198.244.201.187/tcp/4001/p2p/12D3KooWC9L4RjPGgqpzBUBkcVpKjJYofCkC5i5QdQftg1LdsFb2
PeerID: "12D3KooWC9L4RjPGgqpzBUBkcVpKjJYofCkC5i5QdQftg1LdsFb2"
Protocol version: "ipfs/0.1.0"
Agent version: "kubo/0.19.2/afb27ca/filebase"
Listen addresses:
	- "/ip4/198.244.201.187/tcp/4001/p2p/12D3KooWC9L4RjPGgqpzBUBkcVpKjJYofCkC5i5QdQftg1LdsFb2"
Protocols:
	- "/ipfs/bitswap"
	- "/ipfs/bitswap/1.0.0"
	- "/ipfs/bitswap/1.1.0"
	- "/ipfs/bitswap/1.2.0"
	- "/ipfs/id/1.0.0"
	- "/ipfs/id/push/1.0.0"
	- "/ipfs/kad/1.0.0"
	- "/ipfs/lan/kad/1.0.0"
	- "/ipfs/ping/1.0.0"
	- "/libp2p/circuit/relay/0.2.0/stop"
	- "/x/"

@2color
Copy link
Contributor

2color commented Jan 4, 2024

Some updates:

  • web3.storage is discontinuing support for the Pinning API spec (source)
  • Estuary is discontinued
  • Unclear whether nft.storage will continue supporting the pinning API spec. I think they're running on the same infra as web3.storage, so it may also be discontinued.

Copy link

vercel bot commented Oct 10, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
verified-fetch ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 10, 2024 11:17am

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: create helia-remote-pinning example
4 participants