Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advanced partitioning customizations (COMPOSER-2355) #926

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
f39037a
blueprint: partitioning customization
achilleas-k Aug 27, 2024
805ece4
disk: NewCustomPartitionTable() function
achilleas-k Sep 3, 2024
68fb75a
disk: basic test structure for NewCustomPartitionTable()
achilleas-k Sep 9, 2024
548e871
disk: more testing for NewCustomPartitionTable()
achilleas-k Sep 9, 2024
ae2d48b
disk: move handling of the /boot partition
achilleas-k Sep 9, 2024
e0ba877
pkg/{distro,platform}: move BootMode to platform
achilleas-k Sep 6, 2024
8b5653d
disk: add bios and esp partitions based on boot mode
achilleas-k Sep 6, 2024
80ce76d
disk: add root partition / lv / subvol if missing
achilleas-k Sep 6, 2024
766911f
disk: create root partition for empty customizations
achilleas-k Sep 9, 2024
7e9ca23
blueprint: include minsize in partitioning customization
achilleas-k Sep 9, 2024
f88e6e2
disk: add defaultType to NewCustomPartitionTable()
achilleas-k Sep 9, 2024
397efb0
disk: use defaultType when missing fs type customization
achilleas-k Sep 9, 2024
9b75803
disk: define FSType enum for all supported types
achilleas-k Sep 9, 2024
4a689db
disk: relayout the partition table in NewCustomPartitionTable()
achilleas-k Sep 9, 2024
51c44a5
disk: CustomPartitionTableOptions struct
achilleas-k Sep 9, 2024
986b64c
disk: add PartitionTableType to CustomPartitionTableOptions
achilleas-k Sep 10, 2024
d7b9dd4
disk: add "gpt", "dos", and "" types to tests
achilleas-k Sep 10, 2024
24b83ca
disk: generate UUIDs in NewCustomPartitionTable()
achilleas-k Sep 10, 2024
d844a8a
disk: add RequiredSizes to CustomPartitionTableOptions
achilleas-k Sep 12, 2024
4e7a7ee
blueprint: remove GetFilesystemsMinSize()
achilleas-k Sep 11, 2024
2a0bc1b
blueprint: add getter for PartitioningCustomization
achilleas-k Sep 11, 2024
9a5f8dd
blueprint: add CheckPartitioningPolicy()
achilleas-k Sep 11, 2024
92f089b
distro/fedora: pass all customizations to getPartitionTable()
achilleas-k Sep 11, 2024
5da5eae
distro/fedora: enable the new partitioning functionality
achilleas-k Sep 11, 2024
bcfb63a
distro/fedora: add minimum directory sizes to image types
achilleas-k Sep 18, 2024
d50bf94
distro/fedora: check partitioning customizations
achilleas-k Sep 11, 2024
84d94e0
blueprint: validate partitioning customizations
achilleas-k Sep 11, 2024
d9fe555
disk: refactor LVM Logical Volume name generation
achilleas-k Sep 12, 2024
541f95f
disk: use vg.CreateLogicalVolume() in NewCustomPartitionTable()
achilleas-k Sep 11, 2024
c7c3486
disk: generate volume group names when unspecified
achilleas-k Sep 12, 2024
837ce9b
test: add partitioning build configs
achilleas-k Sep 11, 2024
9451486
blueprint: add extra validation for mountpoints and fstypes
achilleas-k Sep 19, 2024
01b0ab7
disk: private resolver for CustomPartitionTableOptions.DefaultFSType
achilleas-k Sep 19, 2024
5bf3416
disk: split NewCustomPartitionTable() into smaller functions
achilleas-k Sep 19, 2024
f6fd14a
disk: more custom partition table tests
achilleas-k Sep 19, 2024
0266934
disk: add error tests for NewCustomPartitionTable()
achilleas-k Sep 19, 2024
6b1f037
disk: reword invalid pt type error message
achilleas-k Sep 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pkg/blueprint/customizations.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ type Customizations struct {
Firewall *FirewallCustomization `json:"firewall,omitempty" toml:"firewall,omitempty"`
Services *ServicesCustomization `json:"services,omitempty" toml:"services,omitempty"`
Filesystem []FilesystemCustomization `json:"filesystem,omitempty" toml:"filesystem,omitempty"`
Partitioning *PartitioningCustomization `json:"partitioning,omitempty" toml:"partitioning,omitempty"`
InstallationDevice string `json:"installation_device,omitempty" toml:"installation_device,omitempty"`
FDO *FDOCustomization `json:"fdo,omitempty" toml:"fdo,omitempty"`
OpenSCAP *OpenSCAPCustomization `json:"openscap,omitempty" toml:"openscap,omitempty"`
Expand Down
89 changes: 88 additions & 1 deletion pkg/blueprint/filesystem_customizations.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,60 @@ import (
"github.com/osbuild/images/pkg/pathpolicy"
)

// TODO: validate input:
// - Duplicate mountpoints
// - No mixing of btrfs and LVM
// - Only one swap partition or file

type PartitioningCustomization struct {
MinSize uint64 `json:"minsize,omitempty" toml:"minsize,omitempty"`
Plain *PlainFilesystemCustomization `json:"plain,omitempty" toml:"plain,omitempty"`
LVM *LVMCustomization `json:"lvm,omitempty" toml:"lvm,omitempty"`
Btrfs *BtrfsCustomization `json:"btrfs,omitempty" toml:"btrfs,omitempty"`
}

type PlainFilesystemCustomization struct {
Filesystems []FilesystemCustomization `json:"filesystems,omitempty" toml:"filesystems,omitempty"`
}

type FilesystemCustomization struct {
Mountpoint string `json:"mountpoint,omitempty" toml:"mountpoint,omitempty"`
Mountpoint string `json:"mountpoint" toml:"mountpoint"`
MinSize uint64 `json:"minsize,omitempty" toml:"minsize,omitempty"`
Label string `json:"label,omitempty" toml:"label,omitempty"`
Type string `json:"type,omitempty" toml:"type,omitempty"`
}

type LVMCustomization struct {
VolumeGroups []VGCustomization `json:"volume_groups,omitempty" toml:"volume_groups,omitempty"`
}

type VGCustomization struct {
// Volume group name
Name string `json:"name" toml:"name"`
// Size of the partition that contains the volume group
MinSize uint64 `json:"minsize" toml:"minsize"`
LogicalVolumes []LVCustomization `json:"logical_volumes,omitempty" toml:"logical_volumes,omitempty"`
}

type LVCustomization struct {
// Logical volume name
Name string `json:"name,omitempty" toml:"name,omitempty"`
FilesystemCustomization
}

type BtrfsCustomization struct {
Volumes []BtrfsVolumeCustomization
}

type BtrfsVolumeCustomization struct {
// Size of the btrfs partition/volume.
MinSize uint64 `json:"minsize" toml:"minsize"`
ondrejbudai marked this conversation as resolved.
Show resolved Hide resolved
Subvolumes []BtrfsSubvolumeCustomization
}

type BtrfsSubvolumeCustomization struct {
Name string `json:"name" toml:"name"`
Mountpoint string `json:"mountpoint" toml:"mountpoint"`
}

func (fsc *FilesystemCustomization) UnmarshalTOML(data interface{}) error {
Expand All @@ -24,6 +75,24 @@ func (fsc *FilesystemCustomization) UnmarshalTOML(data interface{}) error {
return fmt.Errorf("TOML unmarshal: mountpoint must be string, got %v of type %T", d["mountpoint"], d["mountpoint"])
}

switch d["type"].(type) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we don't test this right now? It's fine if we don't, I would love to have a test for this and also one that checks that the json strings are what we expect but I don't want this PR to become bigger, I can offer to do it as a followup :)

Copy link
Member Author

@achilleas-k achilleas-k Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to existing tests.

case nil:
// empty allowed
case string:
fsc.Type = d["type"].(string)
default:
return fmt.Errorf("TOML unmarshal: type must be string, got %v of type %T", d["type"], d["type"])
}

switch d["label"].(type) {
case nil:
// empty allowed
case string:
fsc.Label = d["label"].(string)
default:
return fmt.Errorf("TOML unmarshal: label must be string, got %v of type %T", d["label"], d["label"])
}

switch d["minsize"].(type) {
case int64:
fsc.MinSize = uint64(d["minsize"].(int64))
Expand Down Expand Up @@ -54,6 +123,24 @@ func (fsc *FilesystemCustomization) UnmarshalJSON(data []byte) error {
return fmt.Errorf("JSON unmarshal: mountpoint must be string, got %v of type %T", d["mountpoint"], d["mountpoint"])
}

switch d["type"].(type) {
case nil:
// empty allowed
case string:
fsc.Type = d["type"].(string)
default:
return fmt.Errorf("JSON unmarshal: type must be string, got %v of type %T", d["type"], d["type"])
}

switch d["label"].(type) {
case nil:
// empty allowed
case string:
fsc.Label = d["label"].(string)
default:
return fmt.Errorf("JSON unmarshal: label must be string, got %v of type %T", d["label"], d["label"])
}

// The JSON specification only mentions float64 and Go defaults to it: https://go.dev/blog/json
switch d["minsize"].(type) {
case float64:
Expand Down
22 changes: 22 additions & 0 deletions pkg/blueprint/filesystem_customizations_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ import (
var allFieldsFsc = blueprint.FilesystemCustomization{
Mountpoint: "/data",
MinSize: 1234567890,
Label: "data",
Type: "xfs",
}

func TestFilesystemCustomizationMarshalUnmarshalTOML(t *testing.T) {
Expand Down Expand Up @@ -61,6 +63,16 @@ func TestFilesystemCustomizationUnmarshalTOMLUnhappy(t *testing.T) {
minsize = "20 KG"`,
err: "toml: line 0: TOML unmarshal: minsize is not valid filesystem size (unknown data size units in string: 20 KG)",
},
{
name: "label not string",
input: "label=13\nmountpoint=\"/\"",
err: "toml: line 0: TOML unmarshal: label must be string, got 13 of type int64",
},
{
name: "fstype not string",
input: "type=true\nmountpoint=\"/\"",
err: "toml: line 0: TOML unmarshal: type must be string, got true of type bool",
},
}

for _, c := range cases {
Expand Down Expand Up @@ -93,6 +105,16 @@ func TestFilesystemCustomizationUnmarshalJSONUnhappy(t *testing.T) {
input: `{ "mountpoint": "/", "minsize": "20 KG"}`,
err: "JSON unmarshal: minsize is not valid filesystem size (unknown data size units in string: 20 KG)",
},
{
name: "label not string",
input: `{ "mountpoint": "/", "label": 13}`,
err: "JSON unmarshal: label must be string, got 13 of type float64",
},
{
name: "type not string",
input: `{ "mountpoint": "/", "type": true}`,
err: "JSON unmarshal: type must be string, got true of type bool",
},
}

for _, c := range cases {
Expand Down
49 changes: 49 additions & 0 deletions pkg/disk/disk.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package disk

import (
"encoding/hex"
"fmt"
"io"
"math/rand"
"reflect"
Expand Down Expand Up @@ -58,6 +59,54 @@ const (
XBootLDRPartitionGUID = "BC13C2FF-59E6-4262-A352-B275FD6F7172"
)

// FSType is the filesystem type enum.
//
// There should always be one value for each filesystem type supported by
// osbuild stages (stages/org.osbuild.mkfs.*) and the unset/none value.
type FSType uint64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about making this a string alias, and use descriptive values? Should make debugging a tad bit simpler. Are there downsides (apart from slightly worse performance)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's not much downside other than making it "easy" to use strings in place of the actual value sometimes, which can make it hard to change or track. Having the enum backed by a uint says, a bit more clearly, "the underlying value doesn't matter", which I find encourages better use of the enum.

But I can go either way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fwiw, I had the same thought and tried it out and it was the same about the amount of code/complexity, golang just does not have a good way of doing this it feels so I did not comment (maybe I should have).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have any strong feelings either way? Should I go with strings?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally don't have strong feelings. :)


const (
FS_NONE FSType = iota
FS_VFAT
FS_EXT4
FS_XFS
FS_BTRFS
)

func (f FSType) String() string {
switch f {
case FS_NONE:
return ""
case FS_VFAT:
return "vfat"
case FS_EXT4:
return "ext4"
case FS_XFS:
return "xfs"
case FS_BTRFS:
return "btrfs"
default:
panic(fmt.Sprintf("unknown or unsupported filesystem type with enum value %d", f))
}
}

func FSTypeFromString(s string) (FSType, error) {
mvo5 marked this conversation as resolved.
Show resolved Hide resolved
switch s {
case "":
return FS_NONE, nil
case "vfat":
return FS_VFAT, nil
case "ext4":
return FS_EXT4, nil
case "xfs":
return FS_XFS, nil
case "btrfs":
return FS_BTRFS, nil
default:
return FS_NONE, fmt.Errorf("unknown or unsupported filesystem type name: %s", s)
}
}

// Entity is the base interface for all disk-related entities.
type Entity interface {
// Clone returns a deep copy of the entity.
Expand Down
Loading