Layer 3
OpenSwitch OPX supports unicast routing over Linux interfaces using routes in the Linux kernel routing table. Applications can also use the CPS API to configure routes. This information describes how to configure Layer 3 unicast routing to provision the NPU.
The routing subsystem manages the forwarding information base (FIB), and programs routes with resolved next-hops using ARP/Neighbor table entries received from the Linux kernel.
A routing table entry consists of a destination IP address prefix and at least one next-hop address or a Linux interface.
$ ip route show
default dev eth0 scope link
3.3.3.0/24 dev e101-003-0 proto kernel scope link src 3.3.3.1
$ ip route add 11.10.10.0/24 dev e101-003-0
$ ip route show
default dev eth0 scope link
3.3.3.0/24 dev e101-003-0 proto kernel scope link src 3.3.3.1
11.10.10.0/24 dev e101-003-0 scope link
$ ip route add 30.30.30.0/24 via 3.3.3.3
$ ip route show
default dev eth0 scope link
3.3.3.0/24 dev e101-003-0 proto kernel scope link src 3.3.3.1
30.30.30.0/24 via 3.3.3.3 dev e101-003-0
$ ip route delete 11.10.10.0/24
$ ip route show
default dev eth0 scope link
3.3.3.0/24 dev e101-003-0 proto kernel scope link src 3.3.3.1
To add a persistent static route that is saved after a reboot, configure the route in the /etc/network/interfaces file.
You can add, delete, or modify the IPv6 routes and next-hops in the IPv6 routing table.
$ ip -6 route add 5::5/64 via 3::3
$ ip -6 route show
3::/64 dev e101-003-0 proto kernel metric 256
5::/64 via 3::3 dev e101-003-0 metric 1024
$ ip monitor
30.30.30.0/24 via 3.3.3.3 dev e00-3
3::/64 via 3::3 dev e101-003-0 metric 1024
5::/64 via 3::3 dev e101-003-0 metric 1024
OPX uses ARP and neighbor table entries to resolve adjacencies by using the host IP address-to-MAC address binding. In Linux, the ARP table is used for IPv4 routing, and the neighbor table is used for IPv6 routing.
$ arp -n
Address HWtype HWaddress Flags Mask Iface 3.3.3.4 ether 90:b1:1c:f4:9d:44 C
$ ip -6 neighbor
$ ifconfig e101-003-0 inet6 add 3::1/64
$ ifconfig e101-003-0
e101-003-0 Link encap:Ethernet HWaddr 90:b1:1c:f4:a8:ea
inet addr:3.3.3.1 Bcast:3.3.3.255 Mask:255.255.255.0
inet6 addr: 3::1/64 Scope:Global
inet6 addr: fe80::92b1:1cff:fef4:a8ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:532 errors:0 dropped:0 overruns:0 frame:0
TX packets:173 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:46451 (45.3 KiB) TX bytes:25650 (25.0 KiB)
$ ip -6 neighbor show
3::3 dev e101-003-0 lladdr 90:b1:1c:f4:9d:44 router REACHABLE
$ ping6 3::3
PING 3::3(3::3) 56 data bytes
64 bytes from 3::3: icmp_seq=1 ttl=64 time=1.74 ms
$ tcpdump -i e101-003-0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on e101-003-0, link-type EN10MB (Ethernet), capture size 262144 bytes
04:30:17.053115 IP6 3::1 > 3::3: ICMP6, echo request, seq 8, length 64
The Linux networking stack supports ECMP by adding multiple next-hops to the route.
$ ip route add 40.40.40.0/24 nexthop via 3.3.3.6 nexthop via 4.4.4.7
$ ip route show
default dev eth0 scope link
3.3.3.0/24 dev e101-003-0 proto kernel scope link src 3.3.3.1
40.40.40.0/24
nexthop via 3.3.3.6 dev e101-003-0 weight 1
nexthop via 4.4.4.7 dev e101-004-0 weight 1
The Linux kernel provides limited support for IPv6 multi-path routing.
When you configure an IP address, use any Linux utility command such as ip addr add or ifconfig to configure an interface.
$ ip addr add 10.1.1.1/24 dev e101-007-0
$ ip addr add 11.1.1.1/24 dev e101-001-0
$ ip addr add 10.1.1.2/24 dev e101-007-0
$ ip addr add 12.1.1.1/24 dev e101-001-0
$ ip addr show e101-007-0
16: e101-007-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group
default qlen 500
link/ether 74:e6:e2:f6:af:87 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.1/24 scope global e101-007-0
valid_lft forever preferred_lft forever
inet6 fe80::76e6:e2ff:fef6:af87/64 scope link
valid_lft forever preferred_lft forever
$ ip addr show e101-001-0
10: e101-001-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group
default qlen 500
link/ether 74:e6:e2:f6:af:81 brd ff:ff:ff:ff:ff:ff
inet 11.1.1.1/24 scope global e101-001-0
valid_lft forever preferred_lft forever
inet6 fe80::76e6:e2ff:fef6:af81/64 scope link
valid_lft forever preferred_lft forever
$ ip addr show e101-007-0
16: e101-007-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group
default qlen 500
link/ether 74:e6:e2:f6:ba:87 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 scope global e101-007-0
valid_lft forever preferred_lft forever
inet6 fe80::76e6:e2ff:fef6:ba87/64 scope link
valid_lft forever preferred_lft forever
$ ip addr show e101-001-0
10: e101-001-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group
default qlen 500
link/ether 74:e6:e2:f6:ba:81 brd ff:ff:ff:ff:ff:ff
inet 12.1.1.1/24 scope global e101-001-0
valid_lft forever preferred_lft forever
inet6 fe80::76e6:e2ff:fef6:ba81/64 scope link
valid_lft forever preferred_lft forever
$ ip link set dev e101-007-0 up
$ ip link set dev e101-001-0 up
$ ip route add 12.1.1.0/24 via 10.1.1.2
$ ip route add 11.1.1.0/24 via 10.1.1.1
$ ping 11.1.1.2
$ arp -n
Address HWtype HWaddress Flags Mask Iface
11.1.1.2 ether 00:00:00:1d:9a:bd C e101-001-0
10.1.1.2 ether 74:e6:e2:f6:ba:87 C e101-007-0
$ arp -n
Address HWtype HWaddress Flags Mask Iface
10.1.1.1 ether 74:e6:e2:f6:af:87 C e101-007-0
12.1.1.2 ether 00:00:00:1d:9a:be C e101-001-0
See Programming examples for information on how to program routines using the CPS API.
Virtual routing and forwarding (VRF) allows multiple instances of a routing table to coexist within the same router, at the same time. VRF improves functionality by allowing network paths to be segmented without using multiple devices. The control and data planes are isolated in each VRF, achieving unique routing and forwarding intelligence per VRF.
In OPX, VRF is based on the Linux Stretch (4.9) kernel namespace concept, along with the slave MAC-VLAN interface approach.
This command attaches a new MAC-VLAN link to a physical interface, then moves the interface into the namespace.
$ opx-config-vrf --create ---vrf blue --port e101-001-0
Configuration successful...
root@OPX:~# ip netns exec blue ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
39: v-e101-001-0@if12: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:0c:29:a2:7e:87 brd ff:ff:ff:ff:ff:ff link-netnsid 0
If using FRR, run Zebra with –n option to set the VRF backend based on Linux network namespaces.
$ opx-config-vrf --show
Key: 1.292.19136521.
ni/network-instances/network-instance/name = blue
Key: 1.292.19136521.
ni/network-instances/network-instance/name = default
root@OPX:~# ip netns list
default
blue (id: 1)
root@OPX:~# vtysh
Hello, this is FRRouting (version 5.0.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
OPX# sh vrf
vrf blue id 1 netns /run/netns/blue
First remove ports from the VRF, then delete the VRF.
$ opx-config-vrf --removeport --vrf blue --port e101-001-0
Configuration Successful...
root@OPX:~# opx-config-vrf --delete --vrf blue
Configuration Successful...
root@OPX:~# ip netns
root@OPX:~#
To enable dynamic routing, configure BGP and OSPF using open-source routing stacks, such as Quagga, Bird, and other third-party applications.