RFE: Declarative build system support

[pmatilai]

Vast majority of software projects use a common, known build system where the steps to build and install only differ by configuration details. In rpm itself we have common macros for autotools-based projects (%configure, %make_build, %make_install), and distros have added macros to do the same for various other build systems, including language-specific ecosystems. These simplify life greatly but packagers still need to fill in specs with this redundant boilerplate for each and every project, that boilerplate just has to go away in the common case.

We should implement means to fill up the boilerplate by simply defining the project build system type, eg "Buildsystem: autotools". This would automatically populate the relevant sections with content defined in "autotools" configuration except where a section is already defined by the spec. Specifying an unknown build system would return an explicit error, ie the build systems would be "registered" by dropping a file in a known location rather than just macros defined wherever.

Actual build and install steps are common for all well-behaved packages, but configuration often has project-specific values so while a default configuration should be supplied, it needs be easy to override without reverting to all-manual spec foo. Hence we need at least #1086 for this.

Similarly there's often a need for some more or less minor tweaks to upstream build/install process, this must not require reverting to manually redefining all that. So I suppose we need pre- and post-section scripts, eg %build(post) where such manual tweaks can be placed.

Finally, there are multiple levels of possible default templates: %clean is rather universal and we already supply a default for that as a special case, and similarly %prep could quite univerally default to %autosetup -p1. Both should be overridable by buildsystem specifics though.

As an more practical example, here's an excerpt from a typical autotools project :

%prep
%autosetup

%build
%configure
%make_build

%install
%make_install

All that would simply become:

BuildSystem: autotools

This would be defined by eg /usr/lib/rpm/buildsystem/autotools file contents which could be just a macro file looking somewhat like this:

%__autotools_configure\
%configure

%__autotools_build\
%make_build\

%__autotools_install\
%make_install

...and eg cmake definition would only really differ from that on the configure section. Python etc would use their own macros as distros have defined. This would also serve as an API to which the various language-specific build systems etc can be developed for so the definitions are cross-distro shareable.

Crucial design goals:

• minimize spec boilerplate and redundancy
• building blocks must be equally usable within templates and manual use
• minimize need for spec section override (eg provide pre/post slots for minor customization needs)
• spec override must be possible for individual sections
• no magic, what happens must be easily inspectable (rpmspec --parse etc)

[ignatenkobrain]

Sometimes build can involve multiple buildsystems, would be nice to have that covered.

[pmatilai]

The intention here is to make simple things simple while keeping arbitrary things possible, multiple build systems within a project falls into the "arbitrary" category in my books. However those cases will still benefit from having the interface standardized.

[jasontibbitts]

Wow, this is quite interesting and somewhat similar to things I've brainstormed about for a number of years now I think that this has the possibility to simplify packaging for a significant portion of at least the Fedora packaging set.

However, one problem we've struggled with is that when you hide too much, it becomes difficult for people to understand what to do when a package doesn't fit the mold. Would it be possible for this to simply define defaults for %build or %install or whatnot which would be used if those sections aren't present in the specfile? That way it would be simpler to handle that one package that needs a file removed after %make_install without having to completely switch away from using the templates.

[pmatilai]

Would it be possible for this to simply define defaults for %build or %install or whatnot which would be used if those sections aren't present in the specfile? That way it would be simpler to handle that one package that needs a file removed after %make_install without having to completely switch away from using the templates.

That's exactly the idea, sorry if it wasn't clear from the description. But note that in addition to being able to override any single section, I'm suggesting new pre/post sections for each to minimize the need to do override the entire section. So in order to remove that one pesky unwanted file, you wouldn't revert back to manual %install at all, you'd just add

%install(post)
rm -f /stupid/file

FWIW, I've been thinking and talking about this since at least 2013 too, just haven't gotten around to describe it in writing, much less implement.

[nim-nim]

FYI I’ve converged on this pattern for my own packaging macros

%foometa → munge upstream metadata into rpm metadata
%foopkg -a → create package headers
%fooprep -a → prep actions
%foo_generate_buildrequires -a → dynbr actions
%foobuild -a → build
%fooinstall -a → install
%foocheck -a → check
%foofiles -a → create files section

https://pagure.io/fonts-rpm-macros/
https://pagure.io/go-rpm-macros/

You need to remember, that a spec file may match several build patterns at once (because upstream is shipping a complex project with multiple components of different kinds), and that the result, will be split over multiple subpackages (because the usual result of automating, is enforcing technical component boundaries, not social project boundaries).

Example of multipattern project:
https://src.fedoraproject.org/rpms/golang-x-image/blob/5301a8411cb2d76351fcf83ccf8550d86c9cb215/f/golang-x-image.spec

[voxik]

Looking at the PR and thinking how I would use it for Rubygems, I wonder if something like Autobuild: rubygems would also generate something like BuildRequires: rubygems-devel to pull in the associated /usr/lib/rpm/buildsystem/rubygems. But given that the rubygems-devel cannot be deferred, it would need some nomenclature or virtual provides or what not.

[pmatilai]

Oh, that's a very good point! We'll need to add some kind of hook to allow for that.

[voxik]

On related note, will the autobuild macros be required to build SRPM? I hope that is not the case, similarly as e.g. the BRs are not required.

[pmatilai]

The current autobuild macros relate to the executed scripts which aren't involved in SRPM creation so from that POV no.

The notion that buildrequires are not required for creating an SRPM is rather thin ice though, as the spec needs to be parseable to create an SRPM, and arbitrarily large chunks of the spec could come from macros in the buildrequires. If we add BuildRequires creation to autobuild, and then those macros are not present during SRPM creation, the buildrequires will be missing from the SRPM...

[pmatilai]

To recoup the very useful discussion and ideas from the now withdrawn draft PR:
To make things properly declarative, overriding sections should not be a part of the plan at all. Instead there should be a way to declare independent build options one by one. And the "auto" in the name needs to go.

What I'll be looking into now is:

• BuildSystem: a singleton tag to declare the used build system for the package (tag name open to speculation still, Build is kinda tempting too)
• BuildOption(section): a repeatable tag to declare options one by one to pass to the build system for a given section
• BuildOption: without parentheses is equal to BuildOption(conf) which is by far the most common place needing options

This nicely allows grouping options along with other related conditionals, eg

%if %{with foo}
BuildRequires: libfoo-devel
BuildOption: --enable-foo
%endif

Support for a buildsystem is implemented by defining macros that describe the required for each relevant section. Said macros are responsible for passing the options from the BuildOption tags to the buildsystem as appropriate.

Be sure to check out the #2620 draft for examples and pros/cons of alternatives etc.

[pmatilai]

The above specification is now implemented in #2774, more or less.

[LecrisUT]

Found out about this new feature. How does one go about constructing such BuildSystem, do they just define a %buildsystem_<buildsystem>_conf macros?

I am considering the case of MPI packaging and it would be a neat approach to for looping all mpi variants. One thing that would be needed is to expand BuildOption(section) for specific variant, e.g.: BuildOption(conf-serial).

Alternatively, would the BuildOption work with shell/macro commands, e.g.:

# Using env variable
BuildOption(conf): $([[ -z "$MPI_SUFFIX" ]] && echo "-DUSE_MPI=ON" || echo "-DUSE_MPI=OFF")
# Using macro
BuildOption(conf): -DUSE_MPI=%{?mpi_variant:ON}%{?!mpi_variant:OFF}

The former would work even if the macros %configure macro is expanded with shell for loops, but the latter is more manageable.

[pmatilai]

It's documented here: https://rpm-software-management.github.io/rpm/manual/buildsystem.html#supporting-new-build-systems

But closed tickets are not a good place for questions, please open a new thread in https://github.com/rpm-software-management/rpm/discussions