TestMeltdownSpectre

Runtime differences caused by Meltdown/Spectre patches

In the course of the publication of the Meltdown and Spectre patches by system vendors, voices about performance degredation got louder. Since the patches have a bad influence on transitions from user- to kernel-space and back, this influence also affects LIKWID. The influences on LIKWID are independent of the used access mode since LIKWID always runs in user-space while access to the hardware counter registers is only allowed at kernel-space.

Access Modes

There are currently three different access modes:

• AccessDaemon: The accessDaemon is a distinct process started by the LIKWID library. This daemon is used for priviledge escalation because commonly only root users can are allowed to open/read/write/close the MSR and PCI device files. The common call sequence is: LIKWID AccessDaemon <MSR/PCI devices> Kernel <MSR/PCI devices> AccessDaemon LIKWID
• DirectAccess: This mode can be used if the users have enough priviledges to access the MSR and PCI device files directly. The common call sequence is: LIKWID <MSR/PCI devices> Kernel <MSR/PCI devices> LIKWID
• perf_event: With this mode LIKWID does not measure the counters, it hands over the work to the perf_event Kernel subsystem. In contraast to the other two modes the perf_event mode is not handled by the access-layer because we need higher level information for instructing perf_event which are not available at access-layer level. The common call sequence is: LIKWID perf_event/Kernel LIKWID

First measurements

In one of our system we patched almost all nodes (except two) with the Meltdown/ Spectre patches. The two unpatched nodes are taken out of the job scheduler to be accessible for testing purposes. I have root access to these machines so I can only test AccessDaemon and perf_event access mode, let's start with AccessDaemon

Where did I add timing?

In the LIKWID library is a socalled access-layer which abstracts the access mode. The library uses always HPMread() and HPMwrite() and based on the configuration, the access-layer uses the AccessDaemon or the direct mode. I added timing calls (RDTSC) to both of these functions and print out the result afterwards.

How did I measure

I ran LIKWID (4.3.1) simply using likwid-perfctr -C 0 -g L3 hostname in an infinite loop and save the printed timings.

Results

The results can be seen in the following picture. Hosts e0301 and e0302 are not patched while e0136 has the patches installed. The measurements are in cycles per call and plotted is the minimum, maximum and mean with errorbars and the median.

The minimum execution time is increased by the patches by around 5000 cycles for reads, an increase of 17%, and about 1000 cycles for writes, an increase of 4%. When just looking at the mean, the call time increase is 5% for reads and 7% for writes, pretty moderate, but the median reveals the picture: For reads the increase is about 32% and for writes 24%! A general increase of 30% was noticed also by my colleagues testing at a higher level (MarkerAPI calls). Interestingly, the maximal call time decreases with the patches by 29% for reads and 7% for writes but further investigation is needed to find the reason.

Conclusion

The call time increase is significant but it's not sure where the main contribution comes from. It can be the communication with the AccessDaemon or the access to the registers itself. Further measurements at a deeper level (DirectAccess) are needed to reduce the influences of participating code.