Image Definitions as a middle abstraction layer

[supakeen]

In the past year we've split out what we call our image definitions from the osbuild-composer project into a separate project called images.

Image definitions are what determine what goes into a given image for a given distribution at a given version, architecture, workload, etc. They also determine how all those things are assembled together into a certain artifact.

Splitting out images has helped greatly in reducing time spent on CI while developing osbuild-composer and it receives regular pull requests from distribution owners such as RHEL for Edge and Fedora IoT which use osbuild-composer to build their images.

Over the past year we've also started building a live installer for Fedora Workstation. Currently we are building that as a side-compose which means we aren't the main compose but are testing our build system to ensure stability.

While doing so we had tentatively set our goal to be able to build three new Fedora editions in the near future as side-composes as well. To show our capabilities, to find our weaknesses, and to explore how such a system would work.

While implementing for this there is constant tension between the abstraction level that blueprints offer and the policies for customizations that are enforced on them. We don't want blueprints to be able to break image builds in osbuild-composer and we also don't want blueprints to become a one-on-one mapping to stages in osbuild as that would turn blueprints into the new manifests. We do want blueprints to be a straightforward and failsafe way to make adjustments to images, not to redefine images.

During FOSDEM and before we've talked about the idea to introduce a new abstraction layer called definitions. Definitions would partly replace a large swathe of the Go code currently living in images. They would be in declarative format and likely structured as a tree of files that can be combined into images. These definitions would be packaged inside the distribution(s) they define much like the current fedora-kickstarts used by lorax and fedora-kiwi-descriptions used by kiwi.

This solves a variety of problems:

1. Definitions are allowed to break builds.
2. Definitions can redefine distributions or create entirely new ones.
3. Definitions do not need to be upstreamed and thus can be more easily hacked on and shared locally.
4. Definitions are under the ownership of distributions and thus can be controlled by the processes distributions use for changes.

It also introduces new problems. Currently we have quite a bit of conditional code in images, we'd need to find a way around that, at least for those conditions that do not involve distribution version numbers.

Let's take this clean slate to think about what would be a good format for all of this.

I'd like to stress that it is important to find a gradual path towards our end goal, during FOSDEM we discussed that package sets are a very good initial candidate.

As a secondary thing I'd like to treat this format as a public specification that might garner input from other image build tooling or that could be used by other image build tooling. In this regard it would be great if we can not make these definitions lists-of-stages-with-their-options in the long run. Perhaps that is a pipe dream but let's not couple too tightly.

Some ideas about the possibility of sharing formats is in its inception days in the image build description specification where any discussion about sharing this could take place.

[supakeen]

Note, the format being used is a good candidate to perhaps consider shared between build tools. Currently we use both TOML (in blueprints) and JSON (in manifests). While I'm very much on the side of "let's not add another format" the reality is that we already have two and should pick something that is best suited to this. I've started an issue in the image build description specification to see what others think about formats.

[achilleas-k]

Personally, I'm not too bothered by this decision. Most of the candidate formats we tend to bring up (JSON, TOML, YAML) are interchangeable and more or less compatible with each other. It's not an unimportant decision but it's also one we can finalise at the last moment perhaps. Unless we start considering "smarter" formats.

[supakeen]

Yea, this is mostly to keep in mind that it's a good idea to not do custom-TOML directives or the like which are very likely difficult to share.

[bcl]

I think that trying to share is just going to limit what we can accomplish. Any of the formats are fine as long as all you want to do it list packages or set options. This breaks down when you need to add logic of some sort. I think it is better to focus on allowing users to customize and add their own definitions than to try to share definitions between projects.

Lorax has had a lot of success using mako templates for the boot.iso creation process. It has allowed lorax to define a command set, but left it open to using logic directly in the templates when needed.

Anaconda and kickstart are a similar example, except that kickstart doesn't have any logic capabilites other than using an interpreter for the scripting sections. This has also been pretty successful, but the extensive use of scripting show there needs to be something in-between blueprints and 'just use bash' :)

One possible solution, similar to mako templates, is go's https://pkg.go.dev/text/template package which might be used as the basis of a definition format. Or as a pre-parser for a TOML/JSON format.

[achilleas-k]

The following was something I wrote in chat and I'm copying it here and following it up with some notes and thoughts I had about this from a few months ago.

So as I see it what we want out of this is (very roughly) a way to externally define the thing we currently call ImageType. But we also want to simplify it because there's a weird mix of declarative and procedural stuff that goes into creating an image type. Also a lot of the declarative stuff is half baked.

I think there's two things going on here:

1. Make the definitions declarative
2. Pull them out of the go code into a configuration

No. 2 is the goal state for Fedora, to make it easier for builders outside the project, distro owners that aren't code owners, to define images, but no. 1 is the important part IMO. Also, when we work towards no. 1 it should happen for all distros, even if no. 2 only happens to Fedora (for now).

I think we can get pretty far by taking image config and OS customizations and figuring out which component of the definition they belong to and how they trickle down into stage options. Because those two things combined are pretty much everything that defines the OS pipeline for most images.

---

The following is a collection of notes from my initial work towards the image config and OS customization break-down I mentioned above. These were written with a specific model in mind, (the [PLATFORM, DISTRO, ENVIRONMENT, WORKLOAD] model), but that's of course something we're not bound by. I think it's relevant here because it demonstrates one approach to this whole thing which involves looking at each option/config/customization and figuring out how it affects the image build and whether it should be abstracted away, made an option, or bundled with other similar options/configs/customizations.

The following also assumes a world where pre-defined WORKLOADS exist and the user-supplied blueprints are a custom version of those workloads.

Sample customisations and which component they relate to:

• ISO.ISOLinux: uses ISOLinux bootloader - PLATFORM
• AnacondaISOTree.ISOLinux: see above
• AnacondaISOTree.Users: enables kickstart options and adds users to kickstart file - WORKLOAD
• OSTreeCommit.OSVersion: set version metadata on commit - DISTRO
• EFIBootTree.UEFIVendor: DISTRO
• QCOW2.Compat: DISTRO
  • Actually, Platform dependent: The virtualisation platform is qemu / libvirt v(whatever's on the old distro)
• VPC.ForceSize: DISTRO
• Tar.{Format,ACLs,SELinux,Xattrs}: PLATFORM? (for GCP)
• Time sync servers: ENVIRONMENT (e.g., EC2 chrony servers)

Different blueprint types

Each WORKLOAD can have a separate blueprint spec that supports different options. The current blueprint represents the Custom Workload.

OSCustomizations

It's important to note that a lot of OSCustomizations (and ImageConfig options) are simply stage options that are set pretty high in the layers and trickle all the way down to the stages. With the new abstractions, we need to decouple the purpose and meaning of a configuration option from the effect it has on one or more stages. It's not a one-to-one mapping. An option might require multiple stages to implement, or it might just be a single string that is included in multiple stages. While the stages might look like they are related to one component/layer (e.g., WORKLOAD), the option itself might be related to another (e.g., PLATFORM). Conversely, multiple options, each from a different component/layer, might converge onto a single stage, because a stage affects the image in multiple ways (e.g., packages or systemd services). Package addition, for example, can be the result of an option from any component/layer, but it all converges into a single rpm stage.

Some options can exist in multiple components/layers if they can be overridden or extended. An ENVIRONMENT might define a default option for NTP servers, but a user or built-in WORKLOAD might be allowed to override it. Alternatively, an ENVIRONMENT might define systemd services that are required to boot successfully and the workload might include extra services.

Below I'm marking each option with the name of a component/layer, but for the ones that are stage options in particular, a lot might change in the process.

Source

type OSCustomizations struct {
	// Packages to install in addition to the ones required by the
	// pipeline.
	// WORKLOAD or REMOVE: what would require adding arbitrary packages other than a workload (whether built-in or custom)?
	ExtraBasePackages []string

	// Packages to exclude from the base package set. This is useful in
	// case of weak dependencies, comps groups, or where multiple packages
	// can satisfy a dependency. Must not conflict with the included base
	// package set.
	// WORKLOAD or REMOVE: what would require excluding arbitrary packages other than a workload (whether built-in or custom)?
	ExcludeBasePackages []string

	// Additional repos to install the base packages from.
	ExtraBaseRepos []rpmmd.RepoConfig

	// Containers to embed in the image
	// WORKLOAD: currently, only user-defined (custom) workloads include containers.
	Containers []container.Spec

	// KernelName indicates that a kernel is installed, and names the kernel
	// package.
	// WORKLOAD: this is modified by the user, usually to install the rt kernel
	KernelName string

	// KernelOptionsAppend are appended to the kernel commandline
	KernelOptionsAppend []string

	// KernelOptionsBootloader controls whether kernel command line options
	// should be specified in the bootloader configuration. Otherwise they are
	// specified in /etc/kernel/cmdline (default).
	// This should only be used for RHEL 8 and CentOS 8 images that use grub
	// (non s390x).  Newer releases (9+) should keep this disabled.
	// PLATFORM
	KernelOptionsBootloader bool

	GPGKeyFiles []string

	// WORKLOAD: built-in or custom
	Language         string
	Keyboard         *string
	X11KeymapLayouts []string
	Hostname         string
	Timezone         string
	EnabledServices  []string
	DisabledServices []string
	DefaultTarget    string

	// SELinux policy, when set it enables the labeling of the tree with the
	// selected profile
	// ENVIRONMENT
	SElinux string

	// Only used on RHEL 7
	SELinuxForceRelabel *bool

	// Do not install documentation
	// WORKLOAD
	ExcludeDocs bool

	// WORKLOAD or ENVIRONMENT depending on purpose.  For example, the
	// "ec2-user" for AWS should be defined by the environment.
	Groups []users.Group
	Users  []users.User

	// WORKLOAD
	Firewall *osbuild.FirewallStageOptions

	// PLATFORM: bootloader configuration; hardware dependent
	Grub2Config *osbuild.GRUB2Config

	// PLATFORM: hardware (network) configs
	Sysconfig []*osbuild.SysconfigStageOptions

	// PLATFORM or ENVIRONMENT: used for AMIs to set NAutoVTs
	SystemdLogind []*osbuild.SystemdLogindStageOptions

	// ENVIRONMENT: depends on the deployment environment of the image
	CloudInit []*osbuild.CloudInitStageOptions

	// PLATFORM: hardware
	Modprobe []*osbuild.ModprobeStageOptions

	// PLATFORM: hardware
	DracutConf []*osbuild.DracutConfStageOptions

	// ENVIRONMENT
	// Only supports overriding the environment of the nm-cloud-setup.service
	// for automatically setting up network manager in cloud environments
	SystemdUnit []*osbuild.SystemdUnitStageOptions

	// ENVIRONMENT or WORKLOAD
	// Select identity and authentication sources
	Authselect    *osbuild.AuthselectStageOptions
	SELinuxConfig *osbuild.SELinuxConfigStageOptions

	// PLATFORM: hardware related
	Tuned *osbuild.TunedStageOptions

	// WORKLOAD
	Tmpfilesd []*osbuild.TmpfilesdStageOptions

	// WORKLOAD
	PamLimitsConf []*osbuild.PamLimitsConfStageOptions
	Sysctld       []*osbuild.SysctldStageOptions

	// WORKLOAD: usually release ver, but could be other things
	DNFConfig []*osbuild.DNFConfigStageOptions

	// WORKLOAD
	DNFAutomaticConfig *osbuild.DNFAutomaticConfigStageOptions

	// Same as DNFConfig
	YUMConfig *osbuild.YumConfigStageOptions
	YUMRepos  []*osbuild.YumReposStageOptions

	// WORKLOAD
	SshdConfig *osbuild.SshdConfigStageOptions

	// ENVIRONMENT: GCP stuff
	GCPGuestAgentConfig *osbuild.GcpGuestAgentConfigOptions

	// WORKLOAD
	AuthConfig *osbuild.AuthconfigStageOptions

	// ENVIRONMENT: used in azure
	PwQuality *osbuild.PwqualityConfStageOptions

	// WORKLOAD
	OpenSCAPConfig *osbuild.OscapRemediationStageOptions

	// ENVIRONMENT (default for image type) and WORKLOAD (overrides)
	NTPServers []osbuild.ChronyConfigServer

	// ENVIRONMENT: Azure agent
	WAAgentConfig *osbuild.WAAgentConfStageOptions

	// PLATFORM: hardware thingies
	UdevRules *osbuild.UdevRulesStageOptions

	// ENVIRONMENT (default for image type) and WORKLOAD (overrides)
	LeapSecTZ *string

	FactAPIType string

	Subscription *distro.SubscriptionImageOptions
	RHSMConfig   map[distro.RHSMSubscriptionStatus]*osbuild.RHSMStageOptions

	// VERY WORKLOAD
	// Custom directories and files to create in the image
	Directories []*fsnode.Directory
	Files       []*fsnode.File
}

[supakeen]

We are currently implementing a proof of concept approach to this in the otk repository.