Some subspaces missing from the directory of very large Spaces over federation

[apetresc]

Description

When opening up the room directory for a very large space such as #community:matrix.org from a federating homeserver (matrix.apetre.sc in my case, but I've reproduced it from a few others as well), only a subset of the subspaces are visible.

Steps to reproduce

• Open the room directory of #community:matrix.org from a homserver other than matrix.org
• I'd expect to see 38 rooms and 11 spaces, but instead I see 54 rooms and 3 spaces:
  

Other non-matrix.org users on the Spaces feedback room have experienced the same issue with the exact same room/subspace counts that I was seeing, so it's unlikely to be just a cached bad value.

Clearing cache and reloading does not help. Trying with different instances of Element does not help either.

Version information

• Homeserver: matrix.apetre.sc

If not matrix.org:

• Version: v1.36.0

• Install method: Docker

• Platform: Linux

[karmanyaahm]

Looking at #twim:matrix.org this bug seems to not be happening to everyone, so here's my result that seems to work fine. Note that this page takes about 10 seconds to load each time I navigate to it.

(Screenshot is cut off due to the long list)

• Homeserver: matrix.malhotra.cc
  If not matrix.org:
• Version: v1.36.0
• Install method: Docker
• Platform: Linux

[apetresc]

Your results are much better but still appear to be missing one space. There should be 11, I believe?

[apetresc]

@t3chguy gave me some insight into what's likely happening here. The site-to-site API doesn't have any implementation for paginating the room directory. So if you're on a federating server, the results you get back are undefined, it just depends on what happened to be in the first page of results for whatever subset of homeservers you happened to query.

If you have already joined some subspaces (maybe directly, maybe from a previous session back when that subspace did fit in the first page of results), then those subspaces get queried directly even if they're not part of the first page of global directory results, which is why some users who have already joined a bunch of subspaces may see more/all of them.

Still, a new user with a fresh cache visiting #community:matrix.org today is likely to see the numbers I showed above.

[karmanyaahm]

If you have already joined some subspaces [...] from a previous session back when that subspace did fit in the first page of results), then those subspaces get queried directly

I'm in 6 subspaces, so the result makes sense according to this.

[clokep]

Pagination for this API isn't implemented yet, so this is somewhat expected, although confusing.

[clokep]

Also I don't think this has anything to do with federation, even on matrix.org you don't get the full response.

[clokep]

MSC2946 is the canonical place to track this work btw. We first need to finish the design, then implement it.

[clokep]

I'm going to close this in favor of #10495, which includes pagination.