improve support for out-of-tree kernel modules #1220
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tjkirch
approved these changes
Nov 20, 2020
sam-aws
approved these changes
Nov 20, 2020
zmrow
approved these changes
Nov 20, 2020
We use squashfs archives for files that must be included, but which are rarely or never accessed on most running systems. zstd offers compression ratios similar to xz, and decompression speeds like lz4. This saves space while keeping reads fast. Signed-off-by: Ben Cressey <bcressey@amazon.com>
Previously, we included host programs like `objtool` which are built with the default `gcc` compiler and not our cross-compiler toolchain. This works as long as the running system matches our build host, but would break if we began building x86_64 images on an aarch64 system. The reverse is not true today, but only because `objtool` is not yet required for the arm64 target. Ideally, we'd be able to cross-compile these host programs, but that isn't supported by the kernel's build system, and would be hard to implement. For example, `fixdep` is both a tool we'd want to ship, meaning it would need to be cross-compiled, and a tool that's used to build `objtool`, meaning it couldn't be cross-compiled and still run on the build host. Instead we push the problem out to the downstream consumer, who can be relied on to have a compiler that can build native versions of the host programs. This requires shipping all the headers, tools, and scripts needed to run `make prepare`. For compatibility with solutions like DKMS, which do not expect to run anything but the module build, we add a minimal prepare target to this path so that the host programs will be automatically rebuilt. We also make some edits and exclude some files to avoid dependencies on bison, flex, and OpenSSL. Signed-off-by: Ben Cressey <bcressey@amazon.com>
The squashfs filesystem is meant to be used on a running host, while a tarball is easier to work with when assembling a combined archive that also includes our toolchain. Signed-off-by: Ben Cressey <bcressey@amazon.com>
Apply the same options we use for the kernel-devel squashfs. Signed-off-by: Ben Cressey <bcressey@amazon.com>
With the changes to our packaging of kernel development sources, any out-of-tree module builds will need to run `make prepare` in order to compile dependencies like `objtool`. These binaries need to land in the same directory tree as the other development files we ship. Using an overlayfs mount allows writes to the otherwise read-only content from the squashfs. We purge the upper directory on reboot so changes do not persist across system upgrades. Signed-off-by: Ben Cressey <bcressey@amazon.com>
This directory needs to be writable in order to build out-of-tree modules inside a superpowered container. Signed-off-by: Ben Cressey <bcressey@amazon.com>
To support compiling out-of-tree modules ahead of time, rather than on a running Bottlerocket host, we need to provide two things: the kernel development sources, such as headers and Makefiles; and the toolchain we use to build our kernel. Our toolchain is built separately as part of our cross-compiling SDK, and it's possible, if unlikely, that we would ship two releases with the same kernel version built with a different GCC. It's also possible that variants will use different kernels, so we cannot have just one development kit per release. This is not yet supported, but we need the ecosystem to anticipate the requirement for a per-variant, per-architecture kit. The build target combines the archives from the toolchain matching the SDK we used to build the kernel, and kernel development sources from the most recent build. This produces a single artifact that can be uploaded for later retrieval by a consumer that knows the variant, architecture, and version that they are targeting. Signed-off-by: Ben Cressey <bcressey@amazon.com>
tjkirch
approved these changes
Dec 1, 2020
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue number:
#1141
Description of changes:
This overhauls our packaging of kernel development sources, so that they can be used on an architecture that differs from the build host, and can be extracted for use outside of a running host.
Testing done:
Tests were repeated across this matrix:
Images were tested by launching nodes, logging into a Fedora-based admin container, building the Wireguard out-of-tree module, and then running
make prepare.x86_64 images were additionally tested by using Helm to install Falco, and verifying that the module could be built by DKMS and loaded:
The kmod kits were tested on Fedora 32 and Amazon Linux 2, by building the Wireguard and ZFS out-of-tree modules, using snippets like this:
The modules were then loaded on a running instance to confirm they were built correctly.
x86_64 kmod kits were additionally tested by building the Falco module.
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.