Raw socket checkers

A collection of check scripts that uses RAW sockets to force the destination MAC address, suitable to be used for load balancers (LB) in direct routing mode (LVS-DR) to ensure that the real server is indeed answering to packets that have as destination IP the Virtual IP (VIP).

The problem

Load balancers in direct routing mode (LVS-DR) usually works with this setup:

• The load balancer has the VIP assigned to one interface as an alias.
• The incoming traffic into the LB to the VIP is modified at Layer 2 in the ethernet frame to change the destination MAC address from the one of the VIP interface on the LB itself to the one of the real server's interface to which the LB decided to route the traffic.
• The real servers have the VIP assigned to their loopback interface as an alias in order to answer to traffic that has the VIP as destination IP.
• The real servers are configured to not announce and reply to ARP requests the IPs in their loopback interface.

IF one of the real server is missing the configuration of the VIP as an alias of the lo interface, usually this is what happens:

• All the incoming traffic to that real server is rejected as not recognized by the Kernel as valid.
• The checks of LB management softwares like ldirectord or keepalived will not fail because their checks are performed using the real server IP as destination address.
• The statistics on the LB don't show any anomaly because from the LB's perspective the traffic was properly routed to the real server and because there is no NAT in place the responses from the real servers go directly to the client without any knowledge of it by the LB.
• Any monitor on the VIP service will not clearly detect the issue and at most there will be some flapping checks when the request is routed to the failing real server.
• The LB will continue to route the traffic to the real server until a manual intervention will fix the situation.
• The whole check is following a path that is different from the real traffic.

Available alternatives

Configuration management tools

While the enforcement of the VIP configuration on the loopback of the real servers could be done with any configuration management tool, it will usually not ensure that the interface is actually up at all time.

Alarming tools

While an alarming tool could obviously check the status of the loopback interface on the real server at all time, it will still just raise an alarm and a manual or automatic intervention would be needed. Also in the case of an automatic intervention the usual delay between checks in the alarming tools will not avoid that some traffic got lost.

The proposed solution

The proposed solution is to check the real servers using the same kind of packets that characterize the normal routed traffic from the LB. To achieve that a RAW socket is used to bypass both the ARP and the Kernel networking stack that would not allow to use the VIP as the destination IP in the checks given that the VIP is configured and announced on the LB itself and the packets would not exit the LB at all. The real server MAC address is resolved through ARP (for IPv4) and will be used as the destination MAC address for all the packets. The destination IP instead will be set to the VIP. RAW sockets are used in order to be able to set all those values manually, by-passing the Kernel TCP stack.

Caveat

In case of a cluster of LBs, usually the checks on the real servers are performed by all the LBs, not only by the master one. Because the LBs in stand-by/backup mode don't have the VIP configured, the Kernel would reject the traffic in response to the checks performed using the RAW sockets. In this scenario the possible solutions are:

1. Block with iptables the outgoing reset packets sent by the Kernel.
2. Add the VIP address to the loopback interface to all the LBs in the cluster.
3. Fallback to standard checks using the real server IP on stand-by/backup LBs

The first two solutions requires a manual configuration on the LBs, while the third one requires that the check is aware of the LB role. In order to make the checks aware of the role of the LB to use the third solution, an option is available on all checks (-r) that allow to specify the path of a file where the LB role is saved. The software that is in charge of changing the LB role will need to update this file accordingly.

List of available checks

Because of the usage of RAW sockets, all checks need to be run as root or have the CAP_NET_RAW capability.

• TCP check
• HTTP GET check (BETA)
• HTTP check (BETA)

Dependencies

• GNU C compiler (i.e. gcc package).
• GNU C Library: Development Libraries and Header Files (i.e. install libc6-dev package (or equivalent) on Debian based distros, glibc-devel on RedHat based ones).
• [HTTP checks only] OpenSSL development files (i.e. install libssl-dev package on Debian based distros, openssl-devel on RedHat based ones).

Compilation

To compile all checks just run:

make

• The binaries are created in the bin/ directory.
• All the other build artifacts are created in the build/ directory.
• Run make clean to clean all compiled artifacts.
• The checks can be compiled individually, using make tcp, make http_get or make http.
• All checks were tested on Debian 8 (jessie) and Fedora 23 Server.

TODO

• Add Tests.
• Extract common parameters and options to a common parser.
• Support fragmented responses in HTTP checks.
• Support packet loss in HTTP checks.
• Accept also HTTP responses with LF only.
• Add IPv6 versions of the checks.
• Evaluate options to eventually support SSL.

Useful resources

• C Language Examples of IPv4 and IPv6 Raw Sockets for Linux by P. David Buchan. A very useful resource about RAW sockets in C with examples for all major network protocols.