| Lifecycle Stage | Maturity | Status | Latest Revision |
|---|---|---|---|
| 1A | Working Draft | Active | r0, 2022-11-16 |
Authors: @marcopolo
Interest Group: @marcopolo, @mxinden, @marten-seemann
The perf protocol represents a standard benchmarking protocol that we can use
to talk about performance within and across libp2p implementations. This lets us
analyze peformance, guide improvements, and protect against regressions.
The /perf/1.0.0 protocol (from here on referred to as simply perf) is a
client driven set of benchmarks. To not reinvent the wheel, this perf protocol
is almost identical to Nick Bank's QUIC Performance
Internet-Draft
but adapted to libp2p.
The protocol first performs an upload of a client-chosen amount of bytes. Once
that upload has finished, the server sends back as many bytes as the client
requested.
The bytes themselves should be a predetermined arbitrary set of bytes. Zero is fine, but so is random bytes (as long as it's not a different set of random bytes, because then you may be limited by how fast you can generate random bytes).
The protocol is as a follows:
Client:
- Open a libp2p stream to the server.
- Tell the server how many bytes we want the server to send us as a single big-endian uint64 number. Zero is a valid number, so is the max uint64 value.
- Write some amount of data to the stream. Zero is a valid amount.
- Close the write side of our stream.
- Read from the read side of the stream. This should be the same number of bytes as we told the server in step 2.
Server, on handling a new perf stream:
- Read the big-endian uint64 number. This is how many bytes we'll send back in step 3.
- Read from the stream until we get an
EOF(client's write side was closed). - Send the number of bytes defined in step 1 back to the client. This MUST NOT be run concurrently with step 2.
- Close the stream.
The above protocol is flexible enough to run the following benchmarks and more. The exact specifics of the benchmark (e.g. how much data to download or for how long) are left up to the benchmark implementation. Consider these rough guidelines for how to run one of these benchmarks.
Other benchmarks can be run with the same protocol. The following benchmarks have immediate usefulness, but other benchmarks can be added as we find them useful. Consult the QUIC Performance Internet-Draft for some other benchmarks (called scenarios in the document).
For an upload test, the client sets the the server response size to 0 bytes, writes some amount of data and closes the stream.
For a download test, the client sets the server response size to N bytes, and closes the write side of the data.
The measurements are gathered and reported by the client by measuring how many bytes were transferred by the total time it took from stream open to stream close.
A timer based variant is also possible where we see how much data a client can upload or download within a specific time. For upload it's the same as before and the client closes the stream after the timer ends. For download, the client should request a response size of max uint64, then close the stream after the timer ends.
This benchmark measures connection setup efficiency. A transport that takes many RTTs will perform worse here than one that takes fewer.
To run this benchmark:
- Set up N clients
- Each client opens K connections/s to a single server
- once a connection is established, the client closes it and establishes another one.
Handshakes per second are calculated by taking the total number of connections successfully established and divide it by the time period of the test.
Since this protocol lets clients ask servers to do significant work, it SHOULD NOT be enabled by default in any implementation. Users are advised not to enable this on publicly reachable nodes.
Authentacting by Peer ID could mitigate the security concern by only allowing trusted clients to use the protocol. Support for this is left to the implementation.
As mentioned above, this document is inspired by Nick Bank's: QUIC Performance Internet-Draft