You may begin to notice after writing a couple of package template files a
pattern emerge for some packages. For example, you may find yourself writing
an install()
method that invokes: configure
, cmake
,
make
, make install
. You may also find yourself writing
"prefix=" + prefix
as an argument to configure
or cmake
.
Rather than having you repeat these lines for all packages, Spack has
classes that can take care of these patterns. In addition,
these package files allow for finer grained control of these build systems.
In this section, we will describe each build system and give examples on
how these can be manipulated to install a package.
.. graphviz:: digraph G { node [ shape = "record" ] edge [ arrowhead = "empty" ] PackageBase -> Package [dir=back] PackageBase -> MakefilePackage [dir=back] PackageBase -> AutotoolsPackage [dir=back] PackageBase -> CMakePackage [dir=back] PackageBase -> PythonPackage [dir=back] }
The above diagram gives a high level view of the class hierarchy and how each
package relates. Each subclass inherits from the PackageBaseClass
super class. The bulk of the work is done in this super class which includes
fetching, extracting to a staging directory and installing. Each subclass
then adds additional build-system-specific functionality. In the following
sections, we will go over examples of how to utilize each subclass and to see
how powerful these abstractions are when packaging.
We've already seen examples of a Package
class in our walkthrough for writing
package files, so we won't be spending much time with them here. Briefly,
the Package class allows for arbitrary control over the build process, whereas
subclasses rely on certain patterns (e.g. configure
make
make install
) to be useful. Package
classes are particularly useful
for packages that have a non-conventional way of being built since the packager
can utilize some of Spack's helper functions to customize the building and
installing of a package.
As we have seen earlier, packages using Autotools
use configure
,
make
and make install
commands to execute the build and
install process. In our Package
class, your typical build incantation will
consist of the following:
def install(self, spec, prefix):
configure("--prefix=" + prefix)
make()
make("install")
You'll see that this looks similar to what we wrote in our packaging tutorial.
The Autotools
subclass aims to simplify writing package files and provides
convenience methods to manipulate each of the different phases for a Autotools
build system.
Autotools
packages consist of four phases:
autoreconf()
configure()
build()
install()
Each of these phases have sensible defaults. Let's take a quick look at some
the internals of the Autotools
class:
$ spack edit --build-system autotools
This will open the AutotoolsPackage
file in your text editor.
Note
The examples showing code for these classes is abridged to avoid having long examples. We only show what is relevant to the packager.
.. literalinclude:: _spack_root/lib/spack/spack/build_systems/autotools.py :emphasize-lines: 2,4,28-37 :lines: 138-158,589-617 :linenos:
Important to note are the highlighted lines. These properties allow the
packager to set what build targets and install targets they want for their
package. If, for example, we wanted to add as our build target foo
then we can append to our build_targets
property:
build_targets = ["foo"]
Which is similar to invoking make in our Package
make("foo")
This is useful if we have packages that ignore environment variables and need a command-line argument.
Another thing to take note of is in the configure()
method.
Here we see that the prefix
argument is already included since it is a
common pattern amongst packages using Autotools
. We then only have to
override configure_args()
, which will then return it's output to
to configure()
. Then, configure()
will append the common
arguments
Packagers also have the option to run autoreconf
in case a package
needs to update the build system and generate a new configure
. Though,
for the most part this will be unnecessary.
Let's look at the mpileaks
package.py file that we worked on earlier:
$ spack edit mpileaks
Notice that mpileaks is a Package
class but uses the Autotools
build system. Although this package is acceptable let's make this into an
AutotoolsPackage
class and simplify it further.
.. literalinclude:: tutorial/examples/Autotools/0.package.py :language: python :emphasize-lines: 9 :linenos:
We first inherit from the AutotoolsPackage
class.
Although we could keep the install()
method, most of it can be handled
by the AutotoolsPackage
base class. In fact, the only thing that needs
to be overridden is configure_args()
.
.. literalinclude:: tutorial/examples/Autotools/1.package.py :language: python :emphasize-lines: 25,26,27,28,29,30,31,32 :linenos:
Since Spack takes care of setting the prefix for us we can exclude that as
an argument to configure
. Our packages look simpler, and the packager
does not need to worry about whether they have properly included configure
and make
.
This version of the mpileaks
package installs the same as the previous,
but the AutotoolsPackage
class lets us do it with a cleaner looking
package file.
Packages that utilize Make
or a Makefile
usually require you
to edit a Makefile
to set up platform and compiler specific variables.
These packages are handled by the Makefile
subclass which provides
convenience methods to help write these types of packages.
A MakefilePackage
class has three phases that can be overridden. These include:
edit()
build()
install()
Packagers then have the ability to control how a Makefile
is edited, and
what targets to include for the build phase or install phase.
Let's also take a look inside the MakefilePackage
class:
$ spack edit --build-system makefile
Take note of the following:
.. literalinclude:: _spack_root/lib/spack/spack/build_systems/makefile.py :language: python :emphasize-lines: 60,64,69 :lines: 40-111 :linenos:
Similar to Autotools
, MakefilePackage
class has properties
that can be set by the packager. We can also override the different
methods highlighted.
Let's try to recreate the Bowtie package:
$ spack create -f https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip
==> This looks like a URL for bowtie
==> Found 1 version of bowtie:
1.2.1.1 https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip
==> How many would you like to checksum? (default is 1, q to abort) 1
==> Downloading...
==> Fetching https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip
######################################################################## 100.0%
==> Checksummed 1 version of bowtie
==> This package looks like it uses the makefile build system
==> Created template for bowtie package
==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/bowtie/package.py
Once the fetching is completed, Spack will open up your text editor in the
usual fashion and create a template of a MakefilePackage
package.py.
.. literalinclude:: tutorial/examples/Makefile/0.package.py :language: python :linenos:
Spack was successfully able to detect that Bowtie
uses Make
.
Let's add in the rest of our details for our package:
.. literalinclude:: tutorial/examples/Makefile/1.package.py :language: python :emphasize-lines: 10,11,13,14,18,20 :linenos:
As we mentioned earlier, most packages using a Makefile
have hard-coded
variables that must be edited. These variables are fine if you happen to not
care about setup or types of compilers used but Spack is designed to work with
any compiler. The MakefilePackage
subclass makes it easy to edit
these Makefiles
by having an edit()
method that
can be overridden.
Let's take a look at the default Makefile
that Bowtie
provides.
If we look inside, we see that CC
and CXX
point to our GNU
compiler:
$ spack stage bowtie
Note
- As usual make sure you have shell support activated with spack:
source /path/to/spack_root/spack/share/spack/setup-env.sh
$ spack cd -s bowtie
$ cd spack-src
$ vim Makefile
CPP = g++ -w
CXX = $(CPP)
CC = gcc
LIBS = $(LDFLAGS) -lz
HEADERS = $(wildcard *.h)
To fix this, we need to use the edit()
method to write our custom
Makefile
.
.. literalinclude:: tutorial/examples/Makefile/2.package.py :language: python :emphasize-lines: 23,24,25 :linenos:
Here we use a FileFilter
object to edit our Makefile
. It takes
in a regular expression and then replaces CC
and CXX
to whatever
Spack sets CC
and CXX
environment variables to. This allows us to
build Bowtie
with whatever compiler we specify through Spack's
spec
syntax.
Let's change the build and install phases of our package:
.. literalinclude:: tutorial/examples/Makefile/3.package.py :language: python :emphasize-lines: 28,29,30,31,32,35,36 :linenos:
Here demonstrate another strategy that we can use to manipulate our package
We can provide command-line arguments to make()
. Since Bowtie
can use tbb
we can either add NO_TBB=1
as a argument to prevent
tbb
support or we can just invoke make
with no arguments.
Bowtie
requires our install_target
to provide a path to
the install directory. We can do this by providing prefix=
as a command
line argument to make()
.
Let's look at a couple of other examples and go through them:
$ spack edit esmf
Some packages allow environment variables to be set and will honor them.
Packages that use ?=
for assignment in their Makefile
can be set using environment variables. In our esmf
example we
set two environment variables in our edit()
method:
def edit(self, spec, prefix):
for var in os.environ:
if var.startswith('ESMF_'):
os.environ.pop(var)
# More code ...
if self.compiler.name == 'gcc':
os.environ['ESMF_COMPILER'] = 'gfortran'
elif self.compiler.name == 'intel':
os.environ['ESMF_COMPILER'] = 'intel'
elif self.compiler.name == 'clang':
os.environ['ESMF_COMPILER'] = 'gfortranclang'
elif self.compiler.name == 'nag':
os.environ['ESMF_COMPILER'] = 'nag'
elif self.compiler.name == 'pgi':
os.environ['ESMF_COMPILER'] = 'pgi'
else:
msg = "The compiler you are building with, "
msg += "'{0}', is not supported by ESMF."
raise InstallError(msg.format(self.compiler.name))
As you may have noticed, we didn't really write anything to the Makefile
but rather we set environment variables that will override variables set in
the Makefile
.
Some packages include a configuration file that sets certain compiler variables,
platform specific variables, and the location of dependencies or libraries.
If the file is simple and only requires a couple of changes, we can overwrite
those entries with our FileFilter
object. If the configuration involves
complex changes, we can write a new configuration file from scratch.
Let's look at an example of this in the elk
package:
$ spack edit elk
def edit(self, spec, prefix):
# Dictionary of configuration options
config = {
'MAKE': 'make',
'AR': 'ar'
}
# Compiler-specific flags
flags = ''
if self.compiler.name == 'intel':
flags = '-O3 -ip -unroll -no-prec-div'
elif self.compiler.name == 'gcc':
flags = '-O3 -ffast-math -funroll-loops'
elif self.compiler.name == 'pgi':
flags = '-O3 -lpthread'
elif self.compiler.name == 'g95':
flags = '-O3 -fno-second-underscore'
elif self.compiler.name == 'nag':
flags = '-O4 -kind=byte -dusty -dcfuns'
elif self.compiler.name == 'xl':
flags = '-O3'
config['F90_OPTS'] = flags
config['F77_OPTS'] = flags
# BLAS/LAPACK support
# Note: BLAS/LAPACK must be compiled with OpenMP support
# if the +openmp variant is chosen
blas = 'blas.a'
lapack = 'lapack.a'
if '+blas' in spec:
blas = spec['blas'].libs.joined()
if '+lapack' in spec:
lapack = spec['lapack'].libs.joined()
# lapack must come before blas
config['LIB_LPK'] = ' '.join([lapack, blas])
# FFT support
if '+fft' in spec:
config['LIB_FFT'] = join_path(spec['fftw'].prefix.lib,
'libfftw3.so')
config['SRC_FFT'] = 'zfftifc_fftw.f90'
else:
config['LIB_FFT'] = 'fftlib.a'
config['SRC_FFT'] = 'zfftifc.f90'
# MPI support
if '+mpi' in spec:
config['F90'] = spec['mpi'].mpifc
config['F77'] = spec['mpi'].mpif77
else:
config['F90'] = spack_fc
config['F77'] = spack_f77
config['SRC_MPI'] = 'mpi_stub.f90'
# OpenMP support
if '+openmp' in spec:
config['F90_OPTS'] += ' ' + self.compiler.openmp_flag
config['F77_OPTS'] += ' ' + self.compiler.openmp_flag
else:
config['SRC_OMP'] = 'omp_stub.f90'
# Libxc support
if '+libxc' in spec:
config['LIB_libxc'] = ' '.join([
join_path(spec['libxc'].prefix.lib, 'libxcf90.so'),
join_path(spec['libxc'].prefix.lib, 'libxc.so')
])
config['SRC_libxc'] = ' '.join([
'libxc_funcs.f90',
'libxc.f90',
'libxcifc.f90'
])
else:
config['SRC_libxc'] = 'libxcifc_stub.f90'
# Write configuration options to include file
with open('make.inc', 'w') as inc:
for key in config:
inc.write('{0} = {1}\n'.format(key, config[key]))
config
is just a dictionary that we can add key-value pairs to. By the
end of the edit()
method we write the contents of our dictionary to
make.inc
.
CMake is another common build system that has been gaining popularity. It works
in a similar manner to Autotools
but with differences in variable names,
the number of configuration options available, and the handling of shared libraries.
Typical build incantations look like this:
def install(self, spec, prefix):
cmake("-DCMAKE_INSTALL_PREFIX:PATH=/path/to/install_dir ..")
make()
make("install")
As you can see from the example above, it's very similar to invoking
configure
and make
in an Autotools
build system. However,
the variable names and options differ. Most options in CMake are prefixed
with a '-D'
flag to indicate a configuration setting.
In the CMakePackage
class we can override the following phases:
cmake()
build()
install()
The CMakePackage
class also provides sensible defaults so we only need to
override cmake_args()
.
Let's look at these defaults in the CMakePackage
class in the _std_args()
method:
$ spack edit --build-system cmake
.. literalinclude:: _spack_root/lib/spack/spack/build_systems/cmake.py :language: python :lines: 167-300 :emphasize-lines: 87,96 :linenos:
Some CMake
packages use different generators. Spack is able to support
Unix-Makefile generators as well as Ninja generators.
If no generator is specified Spack will default to Unix Makefiles
.
Next we setup the build type. In CMake
you can specify the build type
that you want. Options include:
empty
Debug
Release
RelWithDebInfo
MinSizeRel
With these options you can specify whether you want your executable to have the debug version only, release version or the release with debug information. Release executables tend to be more optimized than Debug. In Spack, we set the default as Release unless otherwise specified through a variant.
Spack then automatically sets up the -DCMAKE_INSTALL_PREFIX
path,
appends the build type (RelWithDebInfo
default), and then specifies a verbose
Makefile
.
Next we add the rpaths
to -DCMAKE_INSTALL_RPATH:STRING
.
Finally we add to -DCMAKE_PREFIX_PATH:STRING
the locations of all our
dependencies so that CMake
can find them.
In the end our cmake
line will look like this (example is xrootd
):
$ cmake $HOME/spack/var/spack/stage/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/xrootd-4.6.0 -G Unix Makefiles -DCMAKE_INSTALL_PREFIX:PATH=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_FIND_FRAMEWORK:STRING=LAST -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=FALSE -DCMAKE_INSTALL_RPATH:STRING=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/lib:$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/lib64 -DCMAKE_PREFIX_PATH:STRING=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/cmake-3.9.4-hally3vnbzydiwl3skxcxcbzsscaasx5
We can see now how CMake
takes care of a lot of the boilerplate code
that would have to be otherwise typed in.
Let's try to recreate callpath:
$ spack create -f https://github.com/llnl/callpath/archive/v1.0.3.tar.gz
==> This looks like a URL for callpath
==> Found 4 versions of callpath:
1.0.3 https://github.com/LLNL/callpath/archive/v1.0.3.tar.gz
1.0.2 https://github.com/LLNL/callpath/archive/v1.0.2.tar.gz
1.0.1 https://github.com/LLNL/callpath/archive/v1.0.1.tar.gz
1.0 https://github.com/LLNL/callpath/archive/v1.0.tar.gz
==> How many would you like to checksum? (default is 1, q to abort) 1
==> Downloading...
==> Fetching https://github.com/LLNL/callpath/archive/v1.0.3.tar.gz
######################################################################## 100.0%
==> Checksummed 1 version of callpath
==> This package looks like it uses the cmake build system
==> Created template for callpath package
==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/callpath/package.py
which then produces the following template:
.. literalinclude:: tutorial/examples/Cmake/0.package.py :language: python :linenos:
Again we fill in the details:
.. literalinclude:: tutorial/examples/Cmake/1.package.py :language: python :linenos: :emphasize-lines: 9,13,14,18,19,20,21,22,23
As mentioned earlier, Spack will use sensible defaults to prevent repeated code
and to make writing CMake
package files simpler.
In callpath, we want to add options to CALLPATH_WALKER
as well as add
compiler flags. We add the following options like so:
.. literalinclude:: tutorial/examples/Cmake/2.package.py :language: python :linenos: :emphasize-lines: 26,30,31
Now we can control our build options using cmake_args()
. If defaults are
sufficient enough for the package, we can leave this method out.
CMakePackage
classes allow for control of other features in the
build system. For example, you can specify the path to the "out of source"
build directory and also point to the root of the CMakeLists.txt
file if it
is placed in a non-standard location.
A good example of a package that has its CMakeLists.txt
file located at a
different location is found in spades
.
$ spack edit spades
root_cmakelists_dir = "src"
Here root_cmakelists_dir
will tell Spack where to find the location
of CMakeLists.txt
. In this example, it is located a directory level below in
the src
directory.
Some CMake
packages also require the install
phase to be
overridden. For example, let's take a look at sniffles
.
$ spack edit sniffles
In the install()
method, we have to manually install our targets
so we override the install()
method to do it for us:
# the build process doesn't actually install anything, do it by hand
def install(self, spec, prefix):
mkdir(prefix.bin)
src = "bin/sniffles-core-{0}".format(spec.version.dotted)
binaries = ['sniffles', 'sniffles-debug']
for b in binaries:
install(join_path(src, b), join_path(prefix.bin, b))
Python extensions and modules are built differently from source than most applications. These modules are usually installed using the following line:
$ pip install .
We can write package files for Python packages using the Package
class,
but the class brings with it a lot of methods that are useless for Python packages.
Instead, Spack has a PythonPackage
subclass that allows packagers
of Python modules to be able to invoke pip
.
We will write a package file for Pandas:
$ spack create -f https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz
==> This looks like a URL for pandas
==> Warning: Spack was unable to fetch url list due to a certificate verification problem. You can try running spack -k, which will not check SSL certificates. Use this at your own risk.
==> Found 1 version of pandas:
0.19.0 https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz
==> How many would you like to checksum? (default is 1, q to abort) 1
==> Downloading...
==> Fetching https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz
######################################################################## 100.0%
==> Checksummed 1 version of pandas
==> This package looks like it uses the python build system
==> Changing package name from pandas to py-pandas
==> Created template for py-pandas package
==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/py-pandas/package.py
And we are left with the following template:
.. literalinclude:: tutorial/examples/PyPackage/0.package.py :language: python :linenos:
As you can see this is not any different than any package template that we have written. We have the choice of providing build options or using the sensible defaults
Luckily for us, there is no need to provide build args.
Next we need to find the dependencies of a package. Dependencies are usually
listed in setup.py
. You can find the dependencies by searching for
install_requires
keyword in that file. Here it is for Pandas
:
# ... code
if sys.version_info[0] >= 3:
setuptools_kwargs = {
'zip_safe': False,
'install_requires': ['python-dateutil >= 2',
'pytz >= 2011k',
'numpy >= %s' % min_numpy_ver],
'setup_requires': ['numpy >= %s' % min_numpy_ver],
}
if not _have_setuptools:
sys.exit("need setuptools/distribute for Py3k"
"\n$ pip install distribute")
# ... more code
You can find a more comprehensive list at the Pandas documentation.
By reading the documentation and setup.py
we found that Pandas
depends on python-dateutil
, pytz
, and numpy
, numexpr
,
and finally bottleneck
.
Here is the completed Pandas
script:
.. literalinclude:: tutorial/examples/PyPackage/1.package.py :language: python :linenos:
It is quite important to declare all the dependencies of a Python package. Spack can "activate" Python packages to prevent the user from having to load each dependency module explicitly. If a dependency is missed, Spack will be unable to properly activate the package and it will cause an issue. To learn more about extensions go to spack extensions.
From this example, you can see that building Python modules is made easy
through the PythonPackage
class.
Although we won't get in depth with any of the other build systems that Spack supports, it is worth mentioning that Spack does provide subclasses for the following build systems:
IntelPackage
SconsPackage
WafPackage
RPackage
PerlPackage
QMakePackage
Each of these classes have their own abstractions to help assist in writing
package files. For whatever doesn't fit nicely into the other build-systems,
you can use the Package
class.
Hopefully by now you can see how we aim to make packaging simple and robust through these classes. If you want to learn more about these build systems, check out Implementing the installation procedure in the Packaging Guide.