Cannot hibernate bottlerocket instance in AWS

[gilbahat]

What I expected to happen:

I expected bottlerocket to be configurable to allow hibernation as per AWS standards

What actually happened:

bottlerocket ships with kernel_lockdown enabled. while prudent, it has a hibernation policy that requires encrypted swapfiles:
[ 1872.560443] Lockdown: grep: hibernation is restricted; see man kernel_lockdown.7

this policy makes a lot of sense on physical machines but very little sense on cloud instances, where at-rest encryption is done by the cloud provider itself.

How to reproduce the problem:

1. start up a bottlerocket AWS instance, enable hibernation support.
2. enable and log in to the admin container via your preferred method
3. enable a swap file somewhere writable*
4. run pm-hibernate (may need to install pm-utils)
5. observe the error message in dmesg logs

(* while AWS requirements call for hibernation space to be in the root volume, these instructions skip this part because the failure is unrelated. if you want to be prudent about it, change the block device mapping to increase the root volume size and you will get an extra ext4 partition which qualifies)

[foersleo]

Thank you for bringing this up @gilbahat.

We are aware that Bottlerocket is currently not working with AWS hibernation, but we have plans to work towards enabling hibernation this year. Due to the nature of Bottlerocket this unfortunately is not as straight forward as is on a general purpose distribution as you have pointed out.

Especially the situation around encryption and attribution of the hibernation image at resume is something that is an interesting discussion. This point becomes even more interesting once we enable SecureBoot, which we are also working towards in #2501

As I am either way already working on scoping this I will take this one on. Unfortunately we currently do not have a more precise timeline than this year. I will update here with more details and a better timeline whenever I have more information.

I will, however, convert the bug tag to an enhancement tag. Bottlerocket is currently not in the list of supported Linux AMIs for AWS hibernation

[isaacdorfman]

Bottlerocket doesn't allow hibernation because it uses kernel lockdown which prohibits hibernation.
There is a git patch that adds a kernel param called "lockdown_hibernate" which enables hibernation when running with kernel lockdown.

I think it can be a secure option to do the following:
*Add the patch to the list of patches in: packages/kernel-5.15
Add the kernel param to variants/aws-k8s- or otherwise create a new variant that is identical except for this kernel param.
*AWS requires KMS encryption for the volume that will contains the swap therefore it can't be tempered with after hibernation which makes it secure.

[foersleo]

Thanks for your interest in this issue Isaak.

While I had something similar in mind I was not aware of that patch you linked.

For the long run, however, I was tracking the more elaborate efforts that are being discussed upstream for "Encrypted Hibernation". But given the discussion it will probably take a bit more until that arrives in its final form.

Until then I am looking at solutions like the one sketched out by you, and maybe limiting usage to scenarios where we can reason about the implications of softening the lockdown rules, like you have made the case for above.

[isaacdorfman]

Is there any security problem with hibernating into a KMS encrpyted volume?
If so I don't see a problem why the eks variants of bottlerocket shouldn't enable hibernation.

[foersleo]

Sorry for the long pause here. Priorities have shifted around a bit and Hibernation functionality as the team within AWS was working on got de-prioritized. However, we are happy to take contributions, if they do work universally without breaching the security boundaries we have set.

Let me elaborate a bit on this and try to answer the question @isaacdorfman posed:

Yes, from a security standpoint we would be fine hibernating into a KMS encrypted volume. However, as far as I know we can not tell from within a running instance if an EBS volume is encrypted. AWS handles that encryption transparently.
We can hence, unfortunately, not make a blanket statement about hibernation is fine to be enabled. There is just too many variables out of the control of Bottlerocket as a guest OS. To bring that part of the security concerns into the control of the guest OS I would be more interested in an approach that allows for guest controlled encrypted hibernation (as is discussed on the upstream Linux mailing lists I linked before). We could then make a more generic statement and reason about that feature from within the guest and support it depending on necessary facilities being present.

So for now I am putting this issue in icebox state and will monitor the upstream discussions so that we hopefully at some point come back to this and implement a hibernation that does not come with external strings attached.