Build

Introduction

Likwid is built using GNU make and Perl. Besides the Linux kernel and the standard C library, all required dependencies are shipped with the archive (Lua and hwloc). It should build on any Linux distribution with a recent GCC or CLANG compiler and 2.6 or newer Linux kernel without any changes.

There is one generic top level Makefile and one .mk configuration file for each compiler (at the moment GCC, CLANG and ICC). Please note that we test LIKWID only with GCC. CLANG and ICC is only tested for basic functionality.

There is one exception: If you want to use LIKWID on a Intel Xeon Phi card you have to choose the MIC as Compiler in config.mk, which is based on Intel ICC compiler.

Directory structure

All source files are in the src directory. All header files are located in src/includes . Lua application source files are in src/applications. All external tools, namely HWLOC and Lua, are located in ext/. The bench/ folder contains all files of the benchmarking suite of LIKWID.

All build products are generated in the directory ./TAG, where TAG is the compiler configuration, default ./GCC.

Configuration

Usually the only thing you have to configure is the PREFIX install path in the build config file config.mk in the top directory.

Changing color of likwid-pin output

Depending on the background of your terminal window you can choose a color for likwid-pin output. It can be set in the build config file config.mk at the variable COLOR. Possible colors are: NONE, BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN or WHITE.

Usage of access daemon likwid-accessD

Usually on your own system, you can use LIKWID with direct access to the MSR files. If you install LIKWID on a shared system as a HPC compute cluster you may consider to use the access daemon likwid-accessD. This is a proxy application which was implemented with security in mind and performs address checks for allowed access. Using the access daemon, the measurements involve more overhead, especially if you use likwid-perfctr in timeline mode or with the marker API.

To enable using the access daemon, configure in config.mk:

1. Set BUILDDAEMON to true
2. Configure the path to the access daemon binary at ACCESSDAEMON
3. Set the ACCESSMODE to accessdaemon

ACCESSMODE can be direct, accessdaemon and sysdaemon (not yet officially supported). You can overwrite the default setting on the command line with the -M commandline option.

If you want to access Uncore performance counters that are located in the PCI memory range, like they are implemented in Intel SandyBridge EP and IvyBridge EP and newer architectures, you have to use the access daemon or have root privileges because access to the PCI space is only permitted for high privileged users.

Usage of frequency daemon likwid-setFreq

The application likwid-setFrequencies uses another daemon to modify the frequency of CPUs. The daemon is build and later installed if BUILDFREQ is set to true in config.mk.

Build LIKWID as shared library

Per default the LIKWID library is build as a shared library. You need the library if you want to use the Marker API. You can also use the LIKWID modules like perfmon directly. In some settings it is necessary to build LIKWID as a shared library. To do so set SHARED_LIBRARY to true in config.mk.

Instrument likwid-bench for usage with likwid-perfctr

likwid-bench is instrumented for use with likwid-perfctr. This allows you to measure various metrics of your likwid-bench kernels. Enable instrumentation by setting INSTRUMENT_BENCH to true in config.mk.

Enabling Fortran interface for Marker API

If you want to use the Marker API in Fortran programs, LIKWID offers a native Fortran90 interface. To enable it, set FORTRAN_INTERFACE to true in config.mk. More information using LIKWID with Fortran look at Marker API with Fortran90.

Dependencies

Although we tried to minimize the external dependencies of LIKWID, some advanced tools or only specific tool options require external packages.

• likwid-perfscope uses the Perl script feedGnuplot to forward the real-time data to gnuplot. feedGnuplot is included into LIKWID, but gnuplot itself is not.
• likwid-agent provided multiple backends to output the periodically measured data. The syslog backend requires the shell tool logger to be installed. The RRD backend requires rrdtool and the GMetric backend the gmetric tool, part of the Ganglia Monitoring System.
• In order to create the HTML documentation of LIKWID, the tool Doxygen is required.

Build targets

You have to edit config.mk to configure your build and install path.

The following make targets are available:

• make - Build everything
• make likwid-bench - Build likwid-bench
• make likwid-accessD - Build likwid-accessD
• make likwid-setFreq - Build likwid-setFreq
• make docs - Create HTML documentation using doxygen
• make clean - Remove the object file directory ./GCC, keep the executables
• make distclean - Remove all generated files
• make local - Adjust paths in Lua scripts to work from the build directory. Requires the daemons to be already installed. Mainly used for testing.

The build system has a working dependency tracking, therefore make clean is only needed if you change the Makefile configuration.

Installing

NOTE: The pinning functionality and the daemons only work if configured in config.mk and installed with make install. If you do not use the pinning functionality the tools can be used without installation.

• make install - Installs the executables, libraries, man pages and headers to the PREFIX path you configured in config.mk.
• make uninstall - Delete all installed files. If you changed the INSTALLED_PREFIX in config.mk you can use:
• make move - Copy all files from the PREFIX path to the INSTALLED_PREFIX path defined in config.mk
• make uninstall_moved - Delete all installed and moved files from the INSTALLED_PREFIX path

Normal configuration in config.mk sets INSTALLED_PREFIX = PREFIX, so in most cases, make && (sudo) make install is sufficient to build and install LIKWID.

Setting up access for hardware performance monitoring

Hardware performance monitoring on x86 is enabled using model-specific registers (MSR). MSR registers are special registers not part of the instruction set architecture. To read and write to these registers the x86 ISA provides special instructions. These instructions can only be executed in protected mode or in other words only kernel code can execute these instructions. Fortunately, any Linux kernel 2.6 or newer provides access to these registers via a set of device files. This allows to implement all of the functionality in user space. Still it does not allow to use those more advanced features of hardware performance monitoring which require to setup interrupt service routines or kernel located memory.

Per default only root has read/write access to these MSR device files. In order to use the LIKWID tools, which need access to these files (likwid-perfctr, likwid-powermeter and likwid-agent) as standard user, you need to setup access rights to these files.

likwid-perfctr, likwid-powermeter and likwid-agent require the Linux msr kernel module. This module is part of most standard distro kernels. You have to be root to do the initial setup.

- Check if the `msr` module is loaded with `lsmod | grep msr`. There should be an output.
- It the module is not loaded, load it with `modprobe msr`. For automatic loading at startup consult your distros documentation how to do so.
- Adopt access rights on the MSR device files for normal user. To grant access to anyone,  you can use `chmod o+rw /dev/cpu/*/msr`. This is only recommended on single user desktop systems.

As in general access to MSRs is not desired on security sensitive systems, you can either implement a more sophisticated access rights settings with e.g. setgid. A common solution used on many other device files, e.g. for audio, is to introduce a group and make a chown on the MSR device files to that group. Now if you execute likwid-perfctr with setgid on that group, the executing user can use the tool but cannot directly write or read the MSR device files.

Some distributions backported the capabilities check for the MSR device to older kernels. If there are problems with accessing the MSR devices for older kernels with file system permissions set to read&write, please check your kernel code (arch/x86/kernel/msr.c) for the backport and set the MSR capabilities in case.

A secure solution is to use the access daemon likwid-accessD, which encapsulates the access to the MSR device files and performs a address check for allowed registers.

Some newer kernels implement the so-called capabilities, a fine-grained permission system that can allow access to the MSR files for common users. On the downside, it may be not enough anymore to set the suid-root flag for the access daemon, the executable must be registerd at the libcap.

sudo setcap cap_sys_rawio+ep EXECUTABLE

This is only possible on local file systems. A feasible way is to use the likwid-accessD for all accesses and just enable the capabilities for this one binary. This will enable the usage for all LIKWID tools and also for all instrumented binaries. If the likwid-perfctr utility should only be used in wrapper mode, it is suitable to set the capabilities for likwid-perfctr only. Please remember to set the file permission of the MSR device file to read/write for all users, even if capabilites are configured correctly.

Installation on Intel Xeon Phi (KNC)

In order to build LIKWID for the Xeon Phi, set the COMPILER option in config.mk to MIC and build it on a host that has a MIC-aware Intel compiler and the MPSS installed. Moreover, there might be the problem that the required Intel libraries are not present on the Xeon Phi and you need to copy them. Commonly the libraries are libimf.so, libsvml.so, libirng.so and libintlc.so.5. LIKWID assumes that they are copied in the same path as the LIKWID library liblikwid.so ($INSTALLED_PREFIX/lib). If you install them somewhere different, please set the RPATHS variable in config.mk or make/include_MIC.mk to point to the location.

Moreover, the Intel Xeon Phi has some limit of running processes and since every CPU needs its access daemon, LIKWID would start many processes. Therefore, LIKWID allows only direct access to the MSR device files on Xeon Phi. In order to allow this, set the suid-root bit for the Lua interpreter (sudo chmod u+s $INSTALLED_PREFIX/bin/likwid-lua).

Installation on Intel Xeon Phi (KNL)

The successor of the KNC has more similarities with normal system CPUs. Therefore, no special compiler flags are needed during the build process, the default compiler GCC can be used. The only thing that needs to be adjusted is the number of supported hardware threads. In config.mk the variable MAX_NUM_THREADS must be at least the number of hardware threads, so for a 72 core KNL you have to set the variable to minimum 288.