Skip to content

Commit

Permalink
Updated the README.md with a how-to for the RDMA-example
Browse files Browse the repository at this point in the history
  • Loading branch information
Maximilian committed Sep 16, 2024
1 parent fdffa2f commit 608fe64
Show file tree
Hide file tree
Showing 2 changed files with 32 additions and 1 deletion.
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,37 @@ Coyote already comes with a number of pre-configured example applications that c
#### perf_local

#### rdma_service
To run this example, two servers with a U55C-accelerator card each and connected via the same network (both the FPGAs and the servers independently) are required. One of the two FPGAs takes the role of a RDMA-client which initiates transactions and sets the parameters for networking experiment, while the other FPGA acts as a RDMA-server, responding to the initiated transfers. Both sides require the same bitstream `rdma_perf`, which can be built with this name-parameter to the CMakeList exactly as described above. The software-version however is different for the two sides and can be built with the name-parameters `rdma_server` and `rdma_client` respectively. If compiled successfully, the software-build-directory on both machines will hold an executable `bin/test` that can be used as following:

##### Server
The server needs to be initiated first. Since it basically just runs a passive daemon in the background, it's execution can be triggered via

~~~~
$ ./test
~~~~

Furthermore, one needs to find out the IP-address that can be used to exchange meta-information between client and server (CPUs) before initiating RDMA-transfers via FPGAs. The server-IP-address needs to be given to the client as argument and can be figured out via

~~~~
$ ifconfig
~~~~

##### Client
After the background-daemon for the Server is running, the client-software can be started. Different than on the other side, this requires some more arguments:
* `-t`: IP-address of the Server (-CPU) for meta-exchange (QP-exchange and experiment-parameters).
* `-w`: Controller to either issue WRITE (=1) or READ (=0) operations.
* `-n`: Minimum message size for experimentation.
* `-x`: Maximum message size. The experimentation size will be iterated between these two boundaries.
* `-r`: Number of throughput-repetitions that are executed per message size. If this value is set to 0, no throughput-experiments will be executed.
* `-l`: Number of latency-repetitions that are executed per message size. If this value is set to 0, no latency-experiments will be executed.

Therefore, the software-call has to be
~~~~
$ ./test -t <server-side IP-address> -w <1 for WRITE, 0 for READ> -n <minimum message-size> -x <maximum message-size> -r <# of throughput-exchanges per message-size> -l <# of latency-exchanges per message-size>
~~~~

It is important to know that latency-measurements are conducted using a ping-pong-style communication (single RDMA-transactions from server and client exchanged in an alternating pattern repeated `l`-times), while throughput is measured with block transfers (`r` transactions from the client, followed by `r` transactions from the server).


#### reconfigure_shell

Expand Down
2 changes: 1 addition & 1 deletion program_coyote.sh
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ if [ $DRV_INSERT -eq 1 ]; then
echo "***"
echo "** IP_ADDRESS: $DEVICE_1_IP_ADDRESS_HEX_0"
echo "** MAC_ADDRESS: $DEVICE_1_MAC_ADDRESS_0"
sgutil program driver -i $DRV_PATH -p ip_addr=$DEVICE_1_IP_ADDRESS_HEX_0,mac_addr=$DEVICE_1_MAC_ADDRESS_0
sgutil program driver -m $DRV_PATH -p ip_addr=$DEVICE_1_IP_ADDRESS_HEX_0,mac_addr=$DEVICE_1_MAC_ADDRESS_0
# sgutil program driver -m $DRV_PATH
echo "***"
echo "** Driver loaded "
Expand Down

0 comments on commit 608fe64

Please sign in to comment.