LXD containers using shiftfs on ZFS or TMPFS broken on 5.15.0-48.54

[aledbf]

Description of the Bug/Problem:

 Error saving key file: rename /etc/docker/.tmp-key.json006026767 /etc/docker/key.json: value too large for defined data

How to Reproduce:
Steps to reproduce the behavior:

1. Go to '...'

Running docker in docker inside a shiftfs mount. No core dump or useful logs.

Expected behavior:

The same scenario works without issues in v5.16

System information:

• OS/Distribution: Ubuntu Jammy
• Kernelversion: 5.17.5-051705-generic

---

Update by @toby63:
Bug is known upstream, see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1990849
Title: LXD containers using shiftfs on ZFS or TMPFS broken on 5.15.0-48.54

[toby63]

Interesting.

Could you give some more details?
Like what filesystems are you using, what Host OS, what Guest OS, what templates, images or configuration options in Docker and/or LXD and what exactly do you do?
You mention running Docker inside Docker, is that really the case?

As you are using Ubuntu and the kernel maintainers might be interested investigating this for their 5.17 branch, I encourage you to report this upstream too:
https://bugs.launchpad.net/ubuntu/+source/linux

---

@brauner

[brauner]

Can you show findmnt from within that container?
How is the shiftfs mount for Docker setup?

[toby63]

@aledbf Could you provide the requested information?

[aledbf]

@aledbf Could you provide the requested information?

I am sorry. I've been OOO and now in Kubecon. I will try to provide the details after the event.

[toby63]

@aledbf Could you provide the requested information?

I am sorry. I've been OOO and now in Kubecon. I will try to provide the details after the event.

Sry if that sounded a bit rude, I just wanted to know whether you are still interested in this issue.
Take your time.

[aledbf]

No, I am sorry for not posting the details before. Thank you for reaching out 🙏

[toby63]

@aledbf Reminder 2.

[aledbf]

Closing. Using 21.04 shiftfs works without issues. I can reproduce the same problem with 21.10 (kernel 5.19.0-21-generic)

[toby63]

@aledbf Well...actually it does not really help to close this issue, if the problem persists with a specific version.

So some questions:

• What kernel do you use in 21.04 (I assume you mean the Ubuntu OS version?)?

  • Am I assuming correctly that you use a "native" Ubuntu kernel that already includes shiftfs (so you don't need the dkms version)?
    If thats the case, it would probably be a problem with missing parts in the vanilla kernel in comparison to the Ubuntu kernel.
    The only problem known to me is a missing patch for overlayfs (see shiftfs incompatible with overlayfs without specific overlayfs version #2 (comment)).

• Kernel 5.19 should already include the sucessor of shiftfs (idmapped mounts), see:
  https://github.com/toby63/shiftfs-dkms#important-info
  So you don't need to use the original shiftfs (this dkms version) for this (the only exception would be, if you use ZFS or some other filesystems that the sucessor does not support yet (AFAIK)).
  Or is the problem occuring with the sucessor?
  In that case this is not the right place to report it, but we can contact the devs.

[aledbf]

What kernel do you use in 21.04 (I assume you mean the Ubuntu OS version?)?

Ubuntu Jammy 5.15.0-47-generic (5.15.0-48 broke shiftfs)
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1990849

Kernel 5.19 should already include the sucessor of shiftfs (idmapped mounts), see:

I want to switch but that change makes some changes incompatible with older kernels.

[toby63]

Ah sry, now I get it, the upstream bug is your bug; ok so I will leave this open, but this is of course a clear upstream issue.

[toby63]

@aledbf Thx for clarifying and answering 👍.

[toby63]

@brauner @tomponline
Is https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1990849 only a problem with the overlayfs fix included in Ubuntu kernels or is this dkms version also affected?
I am asking because xnox mentions in comment 4 that the suspect is a change to overlayfs . Is that confirmed?

[aledbf]

There is a fix for the issue cfe3544e11cc53e0038410a2199ee6afeea3687f

Using https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/lunar/plain/fs/shiftfs.c?h=master-next works as expected (using a 6.1 kernel)

[toby63]

General info:

• Branch k6.1 includes the patch now.
• Branches k5.18 and k5.16 should also include the patch.
• other branches are partly not patched by upstream, but you can try to include the patch manually.
  The patch is intended for the following kernels:
  • Jammy
  • Kinetic
  • Lunar

---

@aledbf Thx for the heads-up.

I was using a different branch of the lunar repo (because IIRC the master-next branch was not ready back then), so I didn't even see the update.
Sadly this kernel stuff is hard to track.