Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

0.20.0-rc2: rc1 & rc2 fail to boot up with WebTransport in Addresses.Announce #9838

Closed
3 tasks done
ylempereur opened this issue Apr 25, 2023 · 4 comments · Fixed by #9857
Closed
3 tasks done

0.20.0-rc2: rc1 & rc2 fail to boot up with WebTransport in Addresses.Announce #9838

ylempereur opened this issue Apr 25, 2023 · 4 comments · Fixed by #9857
Assignees
@hacdias
Labels
kind/bug A bug in existing code (including security flaws) P0 Critical: Tackled by core team ASAP

Comments

@ylempereur
Copy link

Checklist

Installation method

ipfs-update or dist.ipfs.tech

Version

Kubo version: 0.20.0-rc2
Repo version: 13
System version: amd64/darwin
Golang version: go1.19.8

Config

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "http://webui.ipfs.io.ipns.localhost:8080",
        "http://localhost:3000",
        "http://127.0.0.1:5001",
        "https://webui.ipfs.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [
      "/ip4/127.0.0.1/tcp/4001",
      "/ip4/127.0.0.1/udp/4001/quic",
      "/ip4/127.0.0.1/udp/4001/quic-v1",
      "/ip4/127.0.0.1/udp/4001/quic-v1/webtransport",
      "/ip4/75...170/tcp/4001",
      "/ip4/75...170/udp/4001/quic",
      "/ip4/75...170/udp/4001/quic-v1",
      "/ip4/75...170/udp/4001/quic-v1/webtransport",
      "/ip6/::1/tcp/4001",
      "/ip6/::1/udp/4001/quic",
      "/ip6/::1/udp/4001/quic-v1",
      "/ip6/::1/udp/4001/quic-v1/webtransport",
      "/ip6/2001:...:5c59/tcp/4001",
      "/ip6/2001:...:5c59/udp/4001/quic",
      "/ip6/2001:...:5c59/udp/4001/quic-v1",
      "/ip6/2001:...:5c59/udp/4001/quic-v1/webtransport"
    ],
    "AppendAnnounce": null,
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/127.0.0.1/tcp/4001",
      "/ip4/127.0.0.1/udp/4001/quic",
      "/ip4/127.0.0.1/udp/4001/quic-v1",
      "/ip4/127.0.0.1/udp/4001/quic-v1/webtransport",
      "/ip4/192.168.2.9/tcp/4001",
      "/ip4/192.168.2.9/udp/4001/quic",
      "/ip4/192.168.2.9/udp/4001/quic-v1",
      "/ip4/192.168.2.9/udp/4001/quic-v1/webtransport",
      "/ip6/::1/tcp/4001",
      "/ip6/::1/udp/4001/quic",
      "/ip6/::1/udp/4001/quic-v1",
      "/ip6/::1/udp/4001/quic-v1/webtransport",
      "/ip6/2001:...:5c59/tcp/4001",
      "/ip6/2001:...:5c59/udp/4001/quic",
      "/ip6/2001:...:5c59/udp/4001/quic-v1",
      "/ip6/2001:...:5c59/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt"
  ],
  "DNS": {
    "Resolvers": null
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false,
      "Interval": 10
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooW...V"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": null,
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": [
      {
        "Addrs": [],
        "ID": "12D3KooW...T"
      }
    ]
  },
  "Pinning": {
    "RemoteServices": {
      "Web3_Storage": {
        "API": {
          "Endpoint": "https://api.web3.storage"
        },
        "Policies": {
          "MFS": {
            "Enable": false,
            "PinName": "",
            "RepinInterval": ""
          }
        }
      },
      "pinata": {
        "API": {
          "Endpoint": "https://api.pinata.cloud/psa"
        },
        "Policies": {
          "MFS": {
            "Enable": false,
            "PinName": "",
            "RepinInterval": ""
          }
        }
      }
    }
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "6h0m0s",
    "Strategy": "all"
  },
  "Routing": {
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.128.0/ipcidr/17",
      "/ip4/192.168.64.0/ipcidr/18",
      "/ip4/192.168.32.0/ipcidr/19",
      "/ip4/192.168.16.0/ipcidr/20",
      "/ip4/192.168.8.0/ipcidr/21",
      "/ip4/192.168.4.0/ipcidr/22",
      "/ip4/192.168.0.0/ipcidr/23",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Limits": {}
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

The daemon fails to finish booting if I try to use 20 rc1 or rc2. If I switch back to the 0.19.1 binary, it boots fine.

Here is the log from an rc2 boot:

> ipfs daemon --migrate --enable-gc --routing=auto
Initializing daemon...
Kubo version: 0.20.0-rc2
Repo version: 13
System version: amd64/darwin
Golang version: go1.19.8

Computed default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "8.6 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 30720

Theses can be inspected with 'ipfs swarm resources'.

It simply hangs there and does nothing until I kill it.

Here is the log if I switch back to 0.19.1 (I removed some lines in the middle):

> ipfs daemon --migrate --enable-gc --routing=auto     
Initializing daemon...
Kubo version: 0.19.1
Repo version: 13
System version: amd64/darwin
Golang version: go1.19.8

Computed default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "8.6 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 30720

Theses can be inspected with 'ipfs swarm resources'.

Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/127.0.0.1/udp/4001/quic-v1
...
Swarm announcing /ip6/::1/tcp/4001
Swarm announcing /ip6/::1/udp/4001/quic
Swarm announcing /ip6/::1/udp/4001/quic-v1
Swarm announcing /ip6/::1/udp/4001/quic-v1/webtransport
API server listening on /ip4/127.0.0.1/tcp/5001
WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready

Please advise.
@ylempereur ylempereur added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels Apr 25, 2023
@hacdias
Copy link
Member

hacdias commented May 3, 2023

@ylempereur hey! I'm sorry you're experiencing this. I cannot reproduce your issue on macOS. I tried your configuration (adapting where needed due to secrets being removed). What version of macOS are you running? For how long did you try running the daemon before you killed it?

@ylempereur
Copy link
Author

@hacdias First, I just upgraded to 0.19.2 and there's no problem there.

Next, I reran rc2 and waited 15 mins; still stuck (0.19.2 boots in under 2 seconds). I tried the following from another terminal before I killed it:

> ipfs id
Error: lock /Users/yves/.ipfs/repo.lock: someone else has the lock
>

Also, one ctl-C doesn't kill it, it takes two.

I'm running "macOS Ventura 13.3.1 (a)" on an Intel Mac (16GB RAM).

If you create a debugging/logging branch, I can build it and post the log here.

For reference, while I'm not a go dev, I'm a back end dev with 40+ years of experience. In fact, I was a Mac dev from '84 to '98, and the first half of that was entirely in 68K assembler. So, I can follow directions 😬

@hacdias
Copy link
Member

hacdias commented May 4, 2023 •
edited
Loading

@ylempereur just updated to 13.3.1 (a). I'm using a M1 laptop and I can run both arm64 and amd64 binaries. There's something else going on here. Since you're at it, would you mind trying building from source: https://github.com/ipfs/kubo#build-from-source?

I want to understand if the problem is:

1. Current code
2. The built binaries
3. Your configuration
4. Your laptop

@guseggert has managed to pin-point the issue: removing the WebTransport announce addresses from the configuration fixes the issue. Thanks for reporting. We'll work on finding the root cause to fix it.

Reproducible on macOS, Linux and Windows.

Libp2p upgrade caused the issue: 3c363eb

@hacdias hacdias changed the title 0.20.0-rc2: rc1 & rc2 fail to boot up properly on macOS 0.20.0-rc2: rc1 & rc2 fail to boot up with WebTransport in Addresses.Announce May 4, 2023
@hacdias hacdias added P0 Critical: Tackled by core team ASAP and removed need/triage Needs initial labeling and prioritization labels May 4, 2023
@p-shahi
Copy link

p-shahi commented May 4, 2023

Thread in #libp2p-implementers public slack channel: https://filecoinproject.slack.com/archives/C03K82MU486/p1683208653452199 and potential fix: libp2p/go-libp2p#2268

@BigLep BigLep mentioned this issue May 4, 2023
Closed
@hacdias hacdias mentioned this issue May 5, 2023
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hacdias hacdias

Labels
kind/bug A bug in existing code (including security flaws) P0 Critical: Tackled by core team ASAP
Projects
No open projects
IPFS Shipyard Team
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: deadlock on retrieving WebTransport addresses ipfs/kubo
3 participants
@ylempereur @hacdias @p-shahi