95% of DHT find provider requests are trying to find relays

[Stebalien]

When combined with the issues outlined in libp2p/go-libp2p-kad-dht#146, this is spamming the network. Worse, we're doing this constantly.

Every 15 minutes, if we're connected to fewer than 5 relays, we search for relays 5 3 times until we give up, waiting 30s between each search.

That means:

1. 30 seconds searching, 30 seconds waiting. 5 times (5 minutes).
2. 10 minutes waiting.

That means that a full third of the network (assuming they have auto-relay enabled) is asking for this key at any given time unless they happen to find 5 3 stable nodes.

---

Key issues:

• Requiring 5 is probably too many.
• This module makes no effort to differentiate between stable/unstable relays. This means that crappy relays with low connection limits could get picked repeatedly, causing relay users to constantly have to try to find more relays.

[Stebalien]

For now, let's just hard-code a set of relays and turn this off.

[vyzo]

It tries to find 3 relays, but we can reduce this to 1 or 2.

[vyzo]

And yeah, it should try to estimate relay stability to pick good relays.

[Stebalien]

It tries to find 3 relays, but we can reduce this to 1 or 2.

Ah, got it. It just tries 5 times. I'm pretty sure one of the issues here is that there are groups running relays with low connection limits. That's why we need to make sure we pick a stable relay.

In the mean-time, I'd like to switch to hard-coded relays.

[vyzo]

I don't think hard-coded relays are a very good idea, we can't just route all the traffic to our 3 medium relays and call it day. Who else is a stable relay operator?
I think it's better to try and estimate relay stability and pick accordingly.

[Stebalien]

The current issue is both stability and libp2p/go-libp2p-kad-dht#146. I'm not kidding when I say that 95% of the DHT requests are trying to find relays.

[vyzo]

This is a problem indeed, but hard-coding relays doesn't seem like the right solution. What relays to hard code? We don't have operating guidelines or known relay operators we can contact, and we (as pl) can't run enough relays to support the entire network.

[Stebalien]

I completely agree. I just want to hard-code them for now until we fix the DHT. Right now, we have a nasty feedback loop where a significant portion of the relays' bandwidth is likely waisted by these queries.

[aschmahmann]

@Stebalien @vyzo is the problem simply that the DHT discovery queries for floodsub:/libp2p/relay are taking too long/never ending? If so it really doesn't seem super unreasonable to limit the number of required relays since probably if we find one we'll find multiple. If we're concerned about DHT churn fragmenting the list of relays we can potentially up the publishing on the limited number of relays.

Another approach might be to have a pubsub channel where the relays are published. Hard coding a few peers in the pubsub channel for relay advertisements should be really easy and not eat too many resources.

[raulk]

This search needs to be adaptive. We should start with target N (proposal: 2) and we increment or decrement the target adaptively based on how many relays we found in the last round, bounded by a min=1 and max=5.

If the network can't give you 5 results because they are no 5 relays globally, you are constantly asking the impossible. You need to scale back.

@aschmahmann using pubsub makes sense if the peer anyway supports pubsub. But I wouldn't want to make autorelay (part of the lower stack of libp2p protocols) dependent on a higher-level protocol.

Can we use the routing-helpers to attach pubsub-based and DHT-based discovery in daisy chain?

[aschmahmann]

But I wouldn't want to make autorelay (part of the lower stack of libp2p protocols) dependent on a higher-level protocol.

For sure, and using interfaces like Discovery or ContentRouting make sense to make this swappable as you suggest. However, it's worth noting that using the DHT as Discovery for a total list of values is currently pretty wonky since AFAIK the DHT doesn't cap the number of Provider Records per key, which can lead to some peers getting hammered with requests in addition to if multiple peers contain parts of the total value you basically end up with the issues that IPNS over DHT suffers from.

Can we use the routing-helpers to attach pubsub-based and DHT-based discovery in daisy chain?

@raulk Yes and No.

Yes, if you use the parallel or tiered routers and call SearchValue then you call both the DHT and PubSub in parallel every time you do a request. We could then leverage stateful validators to return the union of relays returned from the DHT and PubSub.

No, if you use the approach above without also doing something like your adaptive scaling (or just setting the number of relays to be small) we still end up doing many DHT requests in addition to listening to the pubsub channel (e.g. if we keep looking for 5 relays and they just aren't there). However, we could create a new utility/wrapper that has some linear or exponential backoff to wrap the DHT requests. Additionally, to distribute a persistent piece of data (e.g. a list of relays) over pubsub you need, at the very least, some chunks of the changes from libp2p/go-libp2p-pubsub-router#33 (e.g. repeated rebroadcasting is necessary like in go-ds-crdt, however, in the above PR which relies on libp2p/go-libp2p-pubsub#190 we are more bandwidth efficient and lower latency).

[raulk]

I believe we have fixed this problem. If not, do you mind reopening, @Stebalien?

[Stebalien]

We have. Thanks for closing.