Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sabakan-cryptsetup: reformat when TPM is enabled #169

Merged
merged 2 commits into from
Aug 19, 2019
Merged

sabakan-cryptsetup: reformat when TPM is enabled #169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6521899
Update sabakan-cryptsetup specifications.
morimoto-cybozu Aug 19, 2019
e25e426
sabakan-cryptsetup: reformat when TPM 2.0 is enabled
ymmt2005 Aug 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 72 additions & 66 deletions docs/disk_encryption.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,75 +1,81 @@
Disk encryption
===============

In order to mount encrypted disk, Sabakan provides `sabakan-cryptsetup` as follows.

* Sabakan keeps an `encryption key` for each disk which is specified by `sabakan-cryptsetup` argument.

Each node generates a pair of `64 byte` random keys. One is for [cryptsetup][], and another is to encrypt
the other by [one-time pad][]. The node keeps this latter key in the first sector of the metadata partition.

The `encryption key` is stored `sabakan`. Hence, the node cannot decrypt its own storage data without `sabakan`, and vice versa.

`sabakan-cryptsetup` requests `/api/v1/crypts` to manage keys. See details [API](api.md).

If the server supports [TPM][] 2.0, Another `encryption key` is stored in `/dev/tpm0` for more secure disk encryption.

* How to calculate the `encryption key` for [cryptsetup][].

If the server does not support TPM 2.0:

`encryption key` in `sabakan` ^ `encryption key` in the disk

If the server supports TPM 2.0:

`encryption key` in `sabakan` ^ `encryption key` in the disk ^ `encryption key` in the TPM

* Operators need to setup RAID, format filesystem, and mount disk after decrypt disks.

Operators needs to prepare own scripts or systemd units to mount disks.

## Crypt specs

`sabakan-cryptsetup` uses [cryptsetup][] in plain mode, therefore, it is mostly raw dm-crypt.

| Parameter | Value |
| ------------ | ------------------------------ |
| Cipher | AES |
| Chain Mode | XTS |
| IV generator | plain64 |
| Key bits | 512 (256 for AES, 256 for XTS) |

`sabakan-cryptsetup` writes/reads from the disk as metadata. Space for metadata keeps `2 MiB` by `--offset` option of the `cryptsetup`.

| Name | Size(byte) | Description |
| -------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `magic` | 8 | A magic number to detect if disk is encrypted by `sabakan-crypsetup`. |
| `one-time pad` | 64 | An one-time pad as a pair of XOR. It is generated by `crypto/rand` of Go. |
| `ID` | 16 | An unique identifier like an UUID to detect if `encryption key` is registered on the `sabakan`. It is generated by `crypto/rand` of Go. |
This document describes how a physical block device is encrypted
with [`sabakan-cryptsetup`](./sabakan-cryptsetup.md).

## Steps to encrypt a block device

1. `sabakan-cryptsetup` tries to detect [TPM][] 2.0.
If TPM 2.0 is available, it generates a random key and stores it into TPM.
The length of the key is the same as `--keysize` given for `sabakan-cryptsetup`.
2. `sabakan-cryptsetup` reads meta data from the head of the block device.
3. If meta data is _not_ found:
1. Generate two random keys of the specified length.
2. Call [`cryptsetup`][cryptsetup] to setup an encrypted block device using the calculated key.
3. Format the disk using one of the two keys as described in the next section.
4. Store the other key in sabakan.
5. Done.
4. If meta data _is_ found:
1. If meta data indicates that it is formatted w/o TPM:
1. If TPM 2.0 becomes available now, goto 3.
2. If TPM 2.0 remains unavailable, goto 5.
5. Read key ID from the meta data to retrieve key-encryption-key from sabakan.
6. Calculate the disk encryption key with the retrieved key and key stored in meta data, and optionally in TPM.
7. Use the encryption key to call `cryptsetup`.
8. Done.

## Disk encryption keys

`sabakan-cryptsetup` uses bitwise-XOR to calculate the disk encryption key from divided keys.
This is called [one-time pad][].

When TPM 2.0 is available, the keys are divided into three.
When not, the keys are divided into two.

Two keys are generated for a block device. A key in TPM is shared among all block devices.
One of the two keys is stored in the meta data in the block device.
Another key is stored in sabakan using its REST API.

## Disk layout

Disks encrypted with `sabakan-cryptsetup` have 2 MiB of meta data at the beginning.
The meta data itself is not encrypted. The format of meta data is as follows:

| Offset | Length (bytes) | Value |
| ------ | -------------: | ------------------------- |
| 0x0000 | 20 | "\x80sabakan-cryptsetup3" |
| 0x0014 | 1 | Key size (bytes) |
| 0x0015 | 1 | TPM version ID |
| 0x0016 | 1 | Length of cipher name |
| 0x0017 | 105 | cipher name |
| 0x0080 | 16 | Random ID |
| 0x0090 | vary | Key encryption key |

* The maximum length of cipher name is 105.
* Unused areas are filled with `0x88`.
* The size of key encryption key (KEK) is the same as the key size at 0x0014.

### TPM version IDs

| ID | Version |
| ---: | ------------------- |
| 0 | Not exist |
| 1 | 1.2 (not supported) |
| 2 | 2.0 |

### Conversion from old layouts

If the meta data has `\x80sabakan-cryptsetup2` in its first 20 bytes, the meta data
will be automatically converted to the current disk layout without TPM information.

## TPM 2.0

`sabakan-cryptsetup` writes/reads from the `/dev/tpm0` if the server supports [TPM] 2.0.

| Name | Offset | Size(byte) | Description |
| -------------- | ------------ | ---------- | ------------------------------------------------------------------------- |
| `one-time pad` | `0x01000000` | 64 | An one-time pad as a pair of XOR. It is generated by `crypto/rand` of Go. |

## Procedure of the disk encryption and decryption

1. `sabakan-cryptsetup` detect storage devices on the `/dev/disk/by-path` using a glob which is specified as an argument.
2. Read `one-time pad` and `ID`, and send HTTP request to `sabakan` whether an `encryption key` and `ID` of its disk are registered, and read another `one-time pad` from `/dev/tpm0` if exists. If not, `sabakan-cryptsetup` re-encrypts the disk.
1. Generate `one-time pad`, a pair of `key` and `ID`.
2. Write `magic`, `one-time pad` and `ID` to metadata partition.
3. If TPM 2.0 supported `/dev/tpm0` exists, write `one-time pad` to the TPM device.
4. Register XOR value between `one-time pad` as an `encryption key` to `sabakan` with `ID`.
3. Execute `cryptsetup` to create `/dev/mapper/crypt-BYPATH` of each disk as decryption. `BYPATH` is the name of the device, shown in `/dev/disk/by-path`.
4. Setup md device and/or initialize filesystem.
5. Mount filesystem.
6. Ready for apps!

To deploy above procedure for nodes, Operator needs to prepare systemd unit files by Ignition. For example,

- `sabakan-cryptsetup.service` ... An oneshot service executes `sabakan-cryptsetup` to create a device mapper.
- `setup-disk.service` ... Create a filesystem, then mount a device mapper.
| Name | Offset | Size(byte) | Description |
| ---- | ------------ | ---------- | ----------------------------------------------------- |
| Key | `0x01000000` | vary | A random key. It is generated by `crypto/rand` of Go. |

[dm-crypt]: https://gitlab.com/cryptsetup/cryptsetup/wikis/DMCrypt
[one-time pad]: https://en.wikipedia.org/wiki/One-time_pad
Expand Down
27 changes: 6 additions & 21 deletions docs/sabakan-cryptsetup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,11 @@ At the next boot, `sabakan-cryptsetup` will download the encrypted disk encrypti
from sabakan, decrypt it, and setup encrypted disks automatically.

If the server supports [TPM] 2.0, `sabakan-cryptsetup` uses `/dev/tpm0`
as a key for generating disk encryption key.
to store divided disk encryption key.

If `sabakan-cryptsetup` finds that the disk has been encrypted without TPM information
while the server supports TPM 2.0, it reformats the disk.
When you start using a server, you should enable the TPM of that server at the very beginning.

Usage
-----
Expand All @@ -34,7 +38,7 @@ $ sabakan-cryptsetup [flags]
Target disks
------------

`sabakan-cryptsetup` scans `/sys/block` directory and encrypt all disks excluding:
`sabakan-cryptsetup` scans `/sys/block` directory and encrypts all disks excluding:

* Virtual devices (devices not having `/sys/block/*/device`)
* Removable devices (devices whose `/sys/block/*/removable` are not `0`)
Expand All @@ -52,23 +56,4 @@ Crypt device name

For each `/sys/block/<NAME>` device, a dm-crypt device is created as `/dev/mapper/crypt-<NAME>`.

Disk layout
-----------

Disks encrypted with `sabakan-cryptsetup` have 2 MiB of meta data at the beginning.
The meta data itself is not encrypted. The format of meta data is as follows:

| Offset | Length (bytes) | Value |
| ------ | -------------: | ------------------------- |
| 0x0000 | 20 | "\x80sabakan-cryptsetup2" |
| 0x0014 | 1 | Key size (bytes) |
| 0x0015 | 1 | Length of cipher name |
| 0x0016 | 106 | cipher name |
| 0x0080 | 16 | Random ID |
| 0x0090 | vary | Key encryption key |

* The maximum length of cipher name is 106.
* Unused areas are filled with `0x88`.
* The size of key encryption key (KEK) is the same as the key size at 0x0014.

[TPM]: https://en.wikipedia.org/wiki/Trusted_Computing
67 changes: 29 additions & 38 deletions pkg/sabakan-cryptsetup/cmd/driver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package cmd
import (
"context"
"crypto/rand"
"errors"
"io/ioutil"
"net"
"net/http"
Expand All @@ -23,6 +24,9 @@ type Driver struct {
cipher string
keySize int
tpmdev string

// status variables
tpmVersion TpmVersionID
}

// NewDriver creates Driver.
Expand Down Expand Up @@ -60,6 +64,8 @@ func NewDriver(sabakanURL, cipher string, keySize int, tpmdev string, disks []Di
cipher: cipher,
keySize: keySize,
tpmdev: tpmdev,

tpmVersion: TpmNone,
}, nil
}

Expand All @@ -69,49 +75,21 @@ func (d *Driver) Setup(ctx context.Context) error {

_, err := os.Stat(d.tpmdev)
switch {
case err != nil && !os.IsNotExist(err):
return err
case os.IsNotExist(err):
log.Info("no TPM is found. disk encryption proceeds without TPM", map[string]interface{}{
"device": d.tpmdev,
log.FnError: err,
})
default:
case err == nil:
log.Info("TPM is found. disk encryption proceeds with TPM", map[string]interface{}{
"device": d.tpmdev,
})

t, err := newTPMDriver(d.tpmdev)
kek, d.tpmVersion, err = readKeyFromTPM(d.tpmdev)
if err != nil {
return err
}
defer t.device.Close()

err = t.checkTPMVersion20()
if err != nil {
log.Warn("device is not TPM 2.0. disk encryption proceeds without TPM", map[string]interface{}{
"device": d.tpmdev,
log.FnError: err,
})
t.device.Close()
break
}

kek, err = t.readKEKFromTPM()
if err != nil {
log.Info("no TPM key encryption key was found", map[string]interface{}{
log.FnError: err,
})
err := t.allocateNVRAM()
if err != nil {
return err
}
kek, err = t.readKEKFromTPM()
if err != nil {
panic(err)
}
t.device.Close()
}
case os.IsNotExist(err):
log.Info("no TPM is found. disk encryption proceeds without TPM", map[string]interface{}{
"device": d.tpmdev,
log.FnError: err,
})
default:
return err
}

for _, disk := range d.disks {
Expand Down Expand Up @@ -147,6 +125,19 @@ func (d *Driver) setupDisk(ctx context.Context, disk Disk, tpmKek []byte) error
if err != nil {
return err
}
if md.tpmVersion != d.tpmVersion {
if d.tpmVersion == 0 {
log.Error("TPM becomes unavailable", map[string]interface{}{
"tpmversion": md.tpmVersion.String(),
})
return errors.New("TPM unavailable")
}
log.Info("reformat disk because TPM is now available", map[string]interface{}{
"disk": disk.Name(),
"tpmversion": d.tpmVersion.String(),
})
return d.formatDisk(ctx, disk, f, tpmKek)
}

ek, err := d.sabakan.CryptsGet(ctx, d.serial, md.HexID())
if err == nil {
Expand All @@ -165,7 +156,7 @@ func (d *Driver) setupDisk(ctx context.Context, disk Disk, tpmKek []byte) error
}

func (d *Driver) formatDisk(ctx context.Context, disk Disk, f *os.File, tpmKek []byte) error {
md, err := NewMetadata(d.cipher, d.keySize)
md, err := NewMetadata(d.cipher, d.keySize, d.tpmVersion)
if err != nil {
return err
}
Expand Down
Loading