Including files in blueprints

[supakeen]

There are quite a few requests for an easier way to include files in blueprints, currently we support inline file contents through the files customization.

Papercuts

These file customizations need their full data to be put inline in the blueprint. There's a few papercuts with this approach:

Binary Data

TOML strings have some escaping rules; however, putting in binary data requires application-specific encoding (such as base64) which we currently don't have.

Many Files

Having to copy-paste a billion files into the blueprint and then keeping them up to date with where they came from is a huge chore.

Ideas

Some ideas have been proposed to deal with this issue:

Blueprint Preprocessor

A thing that lives in weldr-client to parse a blueprint and replace custom directives with their content; an example idea would be:

@include-file(src, dst, ...)

which would expand to the normal file data in a blueprint as we have now.

Blueprint and Associated Data

An alternative to the preprocessor to pre-empt some problems (mentioned later) is to put the blueprint and all data it needs into a single archive file (format can be chosen later); the proposed @include-file directive can only refer to files included in the same archive.

Crystal Ball

Using our crystal ball we can predict issues with some of the approaches suggested here; these will need to be resolved.

Blueprint Preprocessor

Where does it run, we're deprecating the weldr API in osbuild-composer and work is ongoing to convert blueprints to cloudapi compose requests which is being discussed: https://github.com/orgs/osbuild/discussions/2

Summarizing that: we will likely be converting the blueprint server side where there is no access to any local files referred to by the blueprint.

Other Problems

Whatever we do, the requests to the cloudapi will get large; this might not be a problem, or it might be. Where more of a problem lies is how to send all that data to the workers after a build is started.

We'll likely also need a format for the osbuild side of things; copying all the files into the manifest leads to huge manifests. Perhaps a manifest and associated data is in place there as well, where the other files in the archive can be exposed as an input?

[lavocatt]

I don't remember how much of blueprints have been designed, but could blueprints be an archive instead of a text file? And there would be the toml file at the root of the archive and files in their directories alongside with it?

[supakeen]

That's the idea of the 'blueprint and associated data' part; an archive with a blueprint.toml at its root and other files that can be referred to :)

[lavocatt]

ah okay, I understood the files would be inlined in base64. Monday morning misreading :D

[thozza]

My understanding that at the end of the day for the sake of the API call, the data would be in fact inlined in base64... But the thing is that this is not really nice nor manageable for users. But I may have old information 😇

[kingsleyzissou]

So two questions:

1. would we have to define the file customization and then do the include or would the include take care of both?
2. How would we handle file permissions for the second approach? Would we set that on the source file or as a parameter in the include directive?

[supakeen]

1. would we have to define the file customization and then do the include or would the include take care of both?

I can see pros and cons for either approach; perhaps having the directive autofill parts of the file customization while allowing to pass them explicitly as well.

2. How would we handle file permissions for the second approach? Would we set that on the source file or as a parameter in the `include` directive?

This is a good question that @thozza also brought up. I'd assume the latter.

[achilleas-k]

I'd go for the latter and be explicit about it, and falling back to the default when it's not specified (umask based).

[thozza]

I'm not a fan of using any properties of the file from the local FS and would be in favor of listing them explicitly in the BP. The reason is that the owner user / group which should be used in the image may not even exist on the local FS where the BP and custom file are located.

[supakeen]

I'd also like to add that @bcl proposed a 3rd approach which was to use RPMs to package these files in a (I believe) custom repository.

Personally I like the simplicity of this but I'm torn because it requires people that build images to familiarize themselves with the RPM format and repositories which might be a big hurdle to 'just' place some files into an artifact.

[kingsleyzissou]

Not sure if we'd ever support other distros at this point but that would be a limitation if we ever had to cross that bridge. Just an additional con to consider

[thozza]

While this is reasonable idea, I think that it could get really complicated. Unless the user provides the RPM served by a repository (which they can do already today), osbuild-composer would have to take those files and so the whole work. This involves generating an osbuild manifest, which would effectively include a dedicated pipeline for building an RPM with the user-provided files (involves also generating the SPEC file), so that the package could be installed into the image.

I think that this approach is totally valid and technically possible, but I'm not sure if it is worth the effort (from the time and supportability PoV). Especially if osbuild does not really care about how it embeds the file into the image and the whole "dance" with building the RPM as part of the image build seems like unnecessary additional work.

However, using RPM would add one very nice feature - the user would be able to determine the ownership of a file on the system using rpm -qf <path>.

[bcl]

The way I made this easier in lorax-composer was to let the user point to a git repo and composer would make the rpm for them. The rpmfluff package is pretty helpful for things like that.

[thozza]

Yeah, I can imagine that if the user already provides a git repository with a SPEC file, it would things a bit easier. Thanks for this additional context.

[ondrejbudai]

My issues with the "blueprint and associated data" proposal is that it will inevitable lead to creating of a blueprint pre-pre-processor - a script that will take a blueprint and a directory and output a tarball, that's ready for being sent to the pre-preprocessor.

Thus, if we just allow blueprints to include any files from the cwd, we will save ourselves time coding a pre-pre-processor. Also, people are already used to build images using a recipe and files from a directory: Dockerfiles.

RPMs are nice, but they require too much setup imho.

Finally, regarding the "custom directive": I think we should still use a plain TOML/JSON, otherwise you cannot simply deal with blueprints using standard tools and libraries. Thus, I suggest something like this:

[[customizations.files]]
path = "/etc/foobar"
mode = "0644"
user = "root"
group = "root"
#data = "this how you include file content currently"
data-include = "foobar"

This would instruct weldr-client to load the file specified in data-include, put its content to the data key and remove the data-include key.

[bcl]

And for reference here is the work I've done previously - https://github.com/bcl/osbuild/commits/master-copy-file that was never merged (and not quite done, I think? I need to reread all the commits to refresh my memory).

[bcl]

Another idea is to follow the example of rpm creation and use a separate upload for large files, and refer to them in the blueprint using a custom URI of some kind.

[supakeen]

Right, there could be a blob storage and blueprints can only refer to this blob storage but that would still come with a bunch of file management woes. How would you approach this for the on-premise?