Principles of Image Builder

[achilleas-k]

Or:

• What even is Image Builder?
• Who is Image Builder for?
• The overarching vision of the Image Builder project.

First, a disclaimer

A lot of things I state as fact below are simply my own observations while working on these projects for the last few years. For context, almost all of my time is spent in images plus-and-minus one level, meaning osbuild-composer, images, and osbuild. It may very well be that I have a different view of things than others, whether they work on the same components or not. Please point out anything you feel is inaccurate or downright wrong in my statements. That's actually a big part of what this discussion is for.

Principles of $PROJECT

Last year we defined the principles of the component projects that make up Image Builder. You can find each list in the README for each project, osbuild, images, osbuild-composer, and image-builder-frontend. The purpose of this exercise was to make sure everyone was on the same page with respect to the rules we try to follow when working on the project. It was during a time when we had new people joining the team, as well as an increase in external contributors, and it became clear that, for some of us that had been around longer, these principles were somehow shared and agreed upon, but were never written down.

Principles, in this respect, are very different from things like code styles and contributor guides. The idea behind these is to communicate the role and nature of each project, the things users should expect from them (whether those are end users or developers). Statements like osbuild-composer's "If a blueprint can be built, it should also boot" serve to both guide development—if you add a new blueprint feature, you should think about how to validate it against all other possible customizations, image definitions, etc—and also define the user experience. Practically, it might be impossible to adhere to it completely—we can't control for strange interactions between user-defined packages for example—but it becomes a rule we can point to when deciding how tightly to control user inputs.

What even is Image Builder?

We are lacking this same idea for the broader Image Builder project. Unlike with each individual component, I feel that there's less implicit agreement about this. I suspect that a lot of us have different answers for the questions:

• Who is the ideal user of Image Builder?
  • I guess some might talk about Personas, but let's not go there.
• Is the (type of) user of Image Builder in Insights different from the (type of) user of Image Builder on-prem?
• Where does Image Builder fit in the broader world?
  • People mean a lot of different things when using the term. Some image builders are for defining and building a distribution from scratch, while others have a very tight focus around specific use cases or devices. I'm sure we all agree that Image Builder fits somewhere between those two, but I'm also sure that we wouldn't completely agree on where.

I'm certain the people who started the Image Builder project had answers for all of the above and maybe they all agreed on those answers as well. Most of them aren't working on the project any more (😢) but more importantly, things change. I do think it's important to revisit where it all started and why, we don't want to end up recreating the problem this project was meant to solve¹, but I also think it's worth reexamining them (or at least, answering them).

The drift

I think there are times when we drift away from things that might have been considered principles in the past. That's not bad, but it should be conscious. I feel like some of the things we're doing today would have been considered too much 2 or 3 years ago when the project was younger and the scope was smaller. That is also fine and perfectly normal.

Let's keep this under 1000 words

The crux of this whole post is:

Is there a drift in the principles and scope of Image Builder? Is this done consciously when designing features? Is everyone aware of this and are we all on the same page?

I think the answer to the last question there is "no". But as I mentioned in the disclaimer, maybe it's just me.

What do I want?

Whether it's just me or not, I think we would benefit from things like:

• Agreeing on and writing down the principles for Image Builder as a whole.
• Referring to them when designing new features and expanding the scope.
• Discussions that consider the whole stack when working on something new.
  • This isn't just about what will go in the service or not, but also whether a new feature makes sense in the service vs on-prem, what it will look like, how it will interact with everything else, whether it's compatible with some long-term plan involving integrations with other services that not everyone is aware of, etc.

Footnotes

1. For people with access to internal projects on issues.redhat.com, look up RHELBU-292. For everyone else, here's the stated Market Problem:

As a system administrator, I find it expensive and difficult to provide a consistent and compliant environment for my workloads across all my target footprints.

I need to comply with corporate policies, regulations, and the dependencies of my applications. To do this simply, I want to create a gold image. Each infrastructure, like VMware, AWS, or bare metal, requires their own formats for that gold image. Creating that gold image and making that image available to all those infrastructures is time-consuming and easy to mess up. As a result, my gold images are always out of date and my developers end up making up their own.

[schuellerf]

As image builder is mainly used to support the workflow and user-experience in RHEL and Fedora (golden) image creation and customization I was wondering if this is some kind of principle or just something that happens to be and to be changed.
(view of a newbie to the project :-) )

[achilleas-k]

I'd say it is a principle, but on its own it doesn't cover everything. There's a very wide range of features that can fit into that description that make for very different products.

[supakeen]

I've restarted writing this post many times and decided to just have a bunch of slightly related sections with my random thoughts.

Let me start with the questions asked at the beginning of the post as I understand it. Not necessarily how things are presented to the world.

What even is Image Builder? Image Builder is a team that works on tooling to build operating system artifacts.

Who is Image Builder for? Image Builder is for everyone that wants to produce, or needs to consume operating systems.

The overarching vision of the Image Builder project. Provide the necessary tools for people that want to produce, or need to consume operating systems.

---

First, let me start with some of my ideas surrounding the above.

I feel that we're not doing a great job of presenting Image Builder. The stack we work with is large and varied, the public name we use is generally osbuild and not Image Builder. Are we a project, a tool, a team, a webservice?

My thought on that is: we are a team that provides many tools.

As for who is Image Builder for? For me we have two targeted 'demographics':

• System Administrators that want a simple(r) way to build artifacts with some medium customization such as: custom users, custom services, custom software. These should be served either by the webservice or by using osbuild-composer on their own network.
• Distribution Maintainers that want to manage all the details themselves and iterate quickly on their distribution and trying out changes to it and its artifacts.

Historically we have focused mostly on the first part of this, however, the tooling we build has its uses for the second part as well. Lately we've been focusing on that as well but I'm very biased towards that group.

The two groups are partially at odds with each other in the context of our team. Giving more flexibility to distribution maintainers has meant making things harder on system administrators.

---

The presentation of Image Builder is not the greatest. Naming discussions are great so I'll bring one up:

Why are we using the osbuild name so often? It's confusing. Is it Image Builder or is it osbuild? If it wasn't so hard to change names I'd go with the following things:

• osbuild.org -> image-builder.org

• github.com/osbuild -> github.com/image-builder(-project|-team)

• osbuild-composer -> composer

• weldr-client -> composer-client

• images -> composer-images

• image-builder-frontend -> service-frontend

• image-builder-api -> service-api

• osbuild-getting-started -> getting-started

We'd get a bit of hierarchy that way and have 3 clear and distinct areas: service, composer, osbuild.

[achilleas-k]

At the risk of getting sidetracked into naming: I think to the wider world, osbuild is a more unique name that differentiates this image builder from other ones. I wouldn't drop the name because that uniqueness disappears. Image builder is the descriptive name of the project as a whole, but more importantly the product. I think this is (perhaps unfortunately) a common thing especially in the FOSS world, where a technology or project has a unique name but get turned into a product for a company under a generic name.

Red Hat's Image Builder is a collection of projects and technologies, with osbuild at its core. The Image Builder frontend / UI isn't the osbuild frontend / UI (at least not right now), because it's tightly coupled with the service that Red Hat provides.

I see it the same way I see the relationship between the names Anaconda and Installer. Granted, the org/team there is usually referred to as "The installer team", and you could go around talking about "the Fedora installer" and everyone would know what you're talking about, but Anaconda names a specific project (Python distribution confusions aside) and stack that differentiates it from any other installers (present or future).

In conclusion: I wouldn't want to drop the osbuild name from the projects, but I also wouldn't mind if "Red Hat Image Builder" became a more widely used name for the team / project / product / whatever.

[schuellerf]

I'd also argue that a more unique name helps in identification of the tool/project and with search engines. Hence "image builder" is not ideal and osbuild is pretty unique. But I agree there are several places where some name-cleanup can/should be done.

[achilleas-k]

I purposefully skipped any mention of a recent scenario that got me thinking about all this. I was afraid that bringing up specific features might influence the discussion to become too technical. I think I might have over-corrected though and perhaps it's not clear what I'm really on about.

The following is a combination of real events (or my interpretation of them) with some imaginary yet plausible, in my opinion, outcomes.

The real part

We've been getting requests to make partitioning more flexible. This goes beyond the recent requests for things like LVM naming customizations. Users have been asking for filesystem type customizations, control over LVM creation, etc, for years now. While addressing a recent user request, we decided that we can satisfy a lot of the issues and past requests (as well as anticipated future requests) by going "all in" and exposing configuration options for designing more-or-less the whole filesystem layout. This is a major shift from the existing filesystem customizations we support, which only allow adding mountpoints with a minimum size (no filesystem type selection) and a very recently added high-level switch for automatically creating LVM layouts vs plain partitioning.

I think this decision and shift happened only semi-consciously. I'm sure everyone on the team would say they're on board with the whole thing if asked, but I also think that we're making a mistake by not having that conversation in the first place.

Perhaps this conversation did happen, it might have been discussed by higher level decision makers without my knowledge. I doubt it, since I feel like the scope of the feature grew while we (myself and a couple of others on the team) were designing it, but maybe I'm interpreting events incorrectly.

The imaginary part

Now imagine this feature gets merged, added to osbuild-composer, it becomes a feature supported by the blueprints on-prem. Then when it comes time to add it to the service, different decisions are made. People in charge of the scope of features on the service, along with UI/UX designers, decide to address the underlying user request with a smaller scope, perhaps a set of high-level switches and options that translate internally to a small set of advanced partitioning customizations.

This isn't a problem in itself. I do wonder how I would explain to a user the reason why there's a big difference in feature complexity between the two. We would probably say "the target audience for osbuild-composer is different from the audience of the service", for example. But the real reason in this case would be "well the backend people weren't talking to the service people when designing this"... and that's bad.

Note that I don't think any of this will happen, just that it could, perhaps for a different feature in the future. That's fine, we're allowed to have differences in scope for different faces of the project, as long as it's done consciously and not in the ad-hoc, bottom-up way that I think it's done now.

[ondrejbudai]

I think there are multiple phases when working on a new feature:

The first phase is about defining requirements: someone (or a group of people) needs to take inputs from various stakeholders, and distill them into a set of requirements, and a set of "anti-requirements" (we usually refer to them as "out of scope" items). This is how formal features (tracked in Jira) are born.

The second phase is about designing the feature: someone needs to take a look at the requirements, and create a design or an architectural draft of it.

Yes, there are also other phases, but I consider them out of scope. Also, the phases don't happen sequentially. There are a lot of software engineering theories discussing these matters.

Of course, the phases are interconnected, it doesn't make sense to set requirements that are not implementable. An example from the past: we got asked to make bib rootless. After considering several architectures, we rejected that request because the current technology cannot reasonably do it in all environments and configuration scenarios that we want to support.

---

My question is: are you more concerned about the first phase, or the second phase?

Regarding the partitioning that you mentioned: I'm happy to take full responsibility for the first phase outcomes (a set of requirements). It's actually my role, I'm a product owner. See one of my responsibilities from the linux org role api (happy to send a link to Red Hatters):

Works with PM to refine Features, works with the team(s) to break down Features into Epics, and Epics into User Stories (etc).

I'm actually pretty happy with how it went with this feature. It's fairly well-written in Jira (RHELBU-2851, RH-internal only, sorry), I discussed it with both PM and with backend engineers (Achilleas, Michael) to verify that it's doable, and I have pretty strong arguments why this set of requirements is needed. There are features where I definitely did a worse job.

I'm questioning myself if I should have involved also someone from the frontend side... We know of users that are going to use this feature without a graphical frontend, and they requested majority (if not all) of requirements. And to be clear, the feature is only targeting bootc-image-builder (for now). So I'm really not sure if the frontend is relevant here. I hope this doesn't sound too arrogant, but you simply have to draw a line somewhere when discussing features. I'm happy to hear your thoughts.

The second phase was primarily done by you, while Michael and I were doing design reviews. While the original feature was targeting bootc-image-builder, it's pretty obvious that the same data structures will be used by everything in the stack. Thus, I think that we should have involved the whole stack in these considerations. The question is who should represent the entire stack? Should it be an architect? A board of architects? The whole team?

I think I don't want to get the whole team to ack on every feature because it simply doesn't scale: We have 13 engineers working on the project + 2 managers + 3 QEs + various other teams collaborating with us on the project. So I think we should either find a person that is happy to be responsible for the architecture, or find a person representing each area and form a board. I'm not saying that others won't be heard, but I simply don't want to gather 13+ acks on every architectural design.

[achilleas-k]

First of all: Thank you! This is exactly the kind of conversation I was hoping would happen.

I think there are multiple phases when working on a new feature:

The first phase is about defining requirements: someone (or a group of people) needs to take inputs from various stakeholders, and distill them into a set of requirements, and a set of "anti-requirements" (we usually refer to them as "out of scope" items). This is how formal features (tracked in Jira) are born.

The second phase is about designing the feature: someone needs to take a look at the requirements, and create a design or an architectural draft of it.

Yes, there are also other phases, but I consider them out of scope. Also, the phases don't happen sequentially. There are a lot of software engineering theories discussing these matters.

Of course, the phases are interconnected, it doesn't make sense to set requirements that are not implementable. An example from the past: we got asked to make bib rootless. After considering several architectures, we rejected that request because the current technology cannot reasonably do it in all environments and configuration scenarios that we want to support.

My question is: are you more concerned about the first phase, or the second phase?

Second phase. My concern revolves around how things in the second phase can change when people don't have full (or good enough) knowledge of everything discussed in the first phase.

For example, if I start working on a feature and decide to add a couple of things that are considered "out of scope", because they're easy and fit well with how the work turned out, whether those "out of scope" items are written in the ticket or not, will I get push back for adding them and changing the nature of the feature?

If it was decided that something is out of scope because it would grow the amount of work required and I discover that it actually doesn't (it's not a lot more work to add this tiny little bit), perhaps it's fine to keep them. If, on the other hand, an item is considered out of scope because it goes against what we're trying to achieve with a feature (it makes it too complicated, even though it didn't take a lot of work to implement), if it was decided that this out of scope item is something we don't want users to be able to do (because it doesn't fit with our "vision", or it conflicts with other planned features I'm not aware of), will I be told to drop it?

Regarding the partitioning that you mentioned: I'm happy to take full responsibility for the first phase outcomes (a set of requirements). It's actually my role, I'm a product owner. See one of my responsibilities from the linux org role api (happy to send a link to Red Hatters):

Works with PM to refine Features, works with the team(s) to break down Features into Epics, and Epics into User Stories (etc).

I'm actually pretty happy with how it went with this feature. It's fairly well-written in Jira (RHELBU-2851, RH-internal only, sorry), I discussed it with both PM and with backend engineers (Achilleas, Michael) to verify that it's doable, and I have pretty strong arguments why this set of requirements is needed. There are features where I definitely did a worse job.

I love the work you do when fleshing out features. You are absolutely right to be happy with how it went. My example with the partitioning is not a criticism of anything specific to this feature, I just felt like it served as a good example (when adding the imaginary bits of the story).

I'm questioning myself if I should have involved also someone from the frontend side... We know of users that are going to use this feature without a graphical frontend, and they requested majority (if not all) of requirements. And to be clear, the feature is only targeting bootc-image-builder (for now). So I'm really not sure if the frontend is relevant here. I hope this doesn't sound too arrogant, but you simply have to draw a line somewhere when discussing features. I'm happy to hear your thoughts.

I don't think it's arrogant at all. You're right. But you mentioning that it was a feature for BIB actually raises a core issue with what I want to discuss (more below).

The second phase was primarily done by you, while Michael and I were doing design reviews. While the original feature was targeting bootc-image-builder, it's pretty obvious that the same data structures will be used by everything in the stack. Thus, I think that we should have involved the whole stack in these considerations. The question is who should represent the entire stack? Should it be an architect? A board of architects? The whole team?

These are good questions and my answer is basically: I don't know, but it should be someone (or someones). Maybe it should be you. Maybe it already is you and I just don't know it. I think most of this discussion would be over if you responded with "don't worry about all this, I got it, I'm on it". In which case I would thank you and move on, though I'd probably ask that we increase the transparency by, for example, having a well defined scope for what we should and shouldn't be doing for the project as a whole, so I can have some guidance when working on the next ticket. I'm happy to be a code monkey grabbing tickets and following them to the letter, but I also like having an idea of where my work fits in the grand scheme of things and how over-arching decisions are made.

This illustrates my issue very well. I did the second phase. I decided "let's put this in osbuild-composer, it'll be great". The decision was made pretty ad-hoc and it was driven by my personal desire to make partitioning in osbuild-composer more advanced. My question to you is, did you consider whether the blueprints we designed were appropriate for the future of osbuild-composer? Maybe you did, but I also understand if you didn't. Perhaps you thought "Achilleas knows what he's doing, he understands the project, and if he wants to blow up the scope, that's fine", but even then, I don't want to make that decision it with no consideration for how it fits in the team's idea of what our project is for.

We, as a team, since forever, have been pretty self-organising in a lot of ways and that's all fun and a really nice way to work, but I think in the long run it can lead to contradictions or scope changes that cause whiplash, that weren't really planned but just happened.

I think I don't want to get the whole team to ack on every feature because it simply doesn't scale: We have 13 engineers working on the project + 2 managers + 3 QEs + various other teams collaborating with us on the project. So I think we should either find a person that is happy to be responsible for the architecture, or find a person representing each area and form a board. I'm not saying that others won't be heard, but I simply don't want to gather 13+ acks on every architectural design.

I agree, getting everyone to sign off is unsustainable. I don't know what the perfect solution is, but I think regardless of who or what ends up in this role, there's a first step we can take which is agreeing on the general idea of what we might consider in and out of scope for our project. The same way we did the principles for each project: none of us are architects of osbuild, but all of us get a bit uncomfortable and push back when we're reviewing a PR that bends or breaks one of them. I don't think I've ever seen this happen on a global, project-wide scale.