From 608fe64dec6724f266d0193cc379d4ccfed0b776 Mon Sep 17 00:00:00 2001 From: Maximilian Date: Mon, 16 Sep 2024 11:48:15 +0200 Subject: [PATCH] Updated the README.md with a how-to for the RDMA-example --- README.md | 31 +++++++++++++++++++++++++++++++ program_coyote.sh | 2 +- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 0aecc1ab..382f3371 100644 --- a/README.md +++ b/README.md @@ -126,6 +126,37 @@ Coyote already comes with a number of pre-configured example applications that c #### perf_local #### rdma_service +To run this example, two servers with a U55C-accelerator card each and connected via the same network (both the FPGAs and the servers independently) are required. One of the two FPGAs takes the role of a RDMA-client which initiates transactions and sets the parameters for networking experiment, while the other FPGA acts as a RDMA-server, responding to the initiated transfers. Both sides require the same bitstream `rdma_perf`, which can be built with this name-parameter to the CMakeList exactly as described above. The software-version however is different for the two sides and can be built with the name-parameters `rdma_server` and `rdma_client` respectively. If compiled successfully, the software-build-directory on both machines will hold an executable `bin/test` that can be used as following: + +##### Server +The server needs to be initiated first. Since it basically just runs a passive daemon in the background, it's execution can be triggered via + +~~~~ +$ ./test +~~~~ + +Furthermore, one needs to find out the IP-address that can be used to exchange meta-information between client and server (CPUs) before initiating RDMA-transfers via FPGAs. The server-IP-address needs to be given to the client as argument and can be figured out via + +~~~~ +$ ifconfig +~~~~ + +##### Client +After the background-daemon for the Server is running, the client-software can be started. Different than on the other side, this requires some more arguments: +* `-t`: IP-address of the Server (-CPU) for meta-exchange (QP-exchange and experiment-parameters). +* `-w`: Controller to either issue WRITE (=1) or READ (=0) operations. +* `-n`: Minimum message size for experimentation. +* `-x`: Maximum message size. The experimentation size will be iterated between these two boundaries. +* `-r`: Number of throughput-repetitions that are executed per message size. If this value is set to 0, no throughput-experiments will be executed. +* `-l`: Number of latency-repetitions that are executed per message size. If this value is set to 0, no latency-experiments will be executed. + +Therefore, the software-call has to be +~~~~ +$ ./test -t -w <1 for WRITE, 0 for READ> -n -x -r <# of throughput-exchanges per message-size> -l <# of latency-exchanges per message-size> +~~~~ + +It is important to know that latency-measurements are conducted using a ping-pong-style communication (single RDMA-transactions from server and client exchanged in an alternating pattern repeated `l`-times), while throughput is measured with block transfers (`r` transactions from the client, followed by `r` transactions from the server). + #### reconfigure_shell diff --git a/program_coyote.sh b/program_coyote.sh index b28b6081..b0e72de1 100644 --- a/program_coyote.sh +++ b/program_coyote.sh @@ -56,7 +56,7 @@ if [ $DRV_INSERT -eq 1 ]; then echo "***" echo "** IP_ADDRESS: $DEVICE_1_IP_ADDRESS_HEX_0" echo "** MAC_ADDRESS: $DEVICE_1_MAC_ADDRESS_0" - sgutil program driver -i $DRV_PATH -p ip_addr=$DEVICE_1_IP_ADDRESS_HEX_0,mac_addr=$DEVICE_1_MAC_ADDRESS_0 + sgutil program driver -m $DRV_PATH -p ip_addr=$DEVICE_1_IP_ADDRESS_HEX_0,mac_addr=$DEVICE_1_MAC_ADDRESS_0 # sgutil program driver -m $DRV_PATH echo "***" echo "** Driver loaded "