Implement updated spaces summary API

[clokep]

MSC2946 was recently rewritten based on some changes in design and lessons learned during initial implementation. We need to implement the changes and test them, etc.

The main design changes are:

• The API now has a max_depth instead of a max_rooms_per_space.
• The API now has pagination (controlled via limit and from in the requests and next_token in the response).
• The responses now embed the child events with the room.
• The server-server API is significantly different.

This is enough changes that I think we'll need to add new API endpoints / handlers for this code. I think we're going to want to keep backwards compatibility (for at least a few releases) so can't quite get rid of the old code yet. This will be a bit annoying, but should get cleaned up when we switch to the stable version.

I think some rough steps of what we'll want to do are:

1. Refactor some of the existing code.
2. Implement a version without federation (but with pagination) using in-memory storage.
3. Hook up federation APIs to fetch unknown rooms.
4. Handle any pending TODOs.
5. Handle persisting to the database (for multiple workers).
6. Expire old persisted data.
7. Explore caching of requests made over federation.
8. Attempt to fetch from the old federation APIs and translate.
9. Explore whether pagination tokens can be shared between users (with results cached / reused).

[clokep]

We also need to update allowed_spaces -> allowed_room_ids.

[clokep]

The local version of this is done now, with the federation APIs up for review.

I'm wondering about caching the requests made out over federation -- is this something that we do anywhere else? Ideally we would cache them so that they could be used by multiple workers...but again I'm not sure this is something we do? Maybe if the request was routed via the federation_reader or something that would make sense?

[clokep]

Also see matrix-org/matrix-js-sdk#1818 for the Element Web side.

[clokep]

Explore whether pagination tokens can be shared between users (with results cached / reused).

I've thought about this a bit and I don't think the state can be easily shared as what rooms are accessible depends on the requester. We did add a ResponseCache in #10549, but that's there to de-duplicate requests not to re-use responses. (It might make sense to hold onto the responses for a few minutes, but we would then need to generate stable pagination tokens. I'm not convinced this is worthwhile at the moment.)

Federation responses can be easily cached and should be, but that's a separate task.

[clokep]

I'm going to call this "done" and handle the follow-ups as part of #10523, which are all to do with optimizations / performance.

This feature will be coming out in Synapse 1.41.0, which should have an RC available this week.