XFN doesn't run on AWS Bottlerocket #3899

passenger31 · 2023-03-23T10:49:18Z

What happened?

xfn container doesn't start when running on the AWS Bottlerocket nodes - returning a possibly misleading message of 'no space left on device'

How can we reproduce it?

create an EKS cluster running Bottlerocket and start with xfn enabled.

to further debug I tried to give it as many privileges as possible:

        xfn:
          args:
          - --debug
          cache:
            sizeLimit: 2Gi
          enabled: true
          securityContext:
            readOnlyRootFilesystem: false
            allowPrivilegeEscalation: true
            privileged: true
            hostPID: true
            hostIPC: true
            hostNetwork: true
            capabilities:
              add: ["SETUID", "SETGID", "SYS_ADMIN", "NET_ADMIN"]
            runAsUser: 0
            runAsGroup: 0

an strace of the xfn binary in the container results in:

from https://manpages.ubuntu.com/manpages/impish/man7/user_namespaces.7.html

ENOSPC (since Linux 4.9; beforehand EUSERS)
              CLONE_NEWUSER was specified in the flags mask, and the call would cause  the  limit
              on the number of nested user namespaces to be exceeded.  See [user_namespaces](https://manpages.ubuntu.com/manpages/impish/man7/user_namespaces.7.html)(7).

the max_user_namespaces is set to 0:

from https://github.com/bottlerocket-os/bottlerocket/blob/develop/SECURITY_GUIDANCE.md

Bottlerocket does not currently support user namespaces. This means that UID 0 (root) inside the container is the same as UID 0 on the host.

I'm not sure if my lead is correct and if this actually relates to the user namespace so I could use a hand to further investigate the strace or any other leads.

What environment did it happen in?

Crossplane version: v1.11.2-stable
Kube version:

Client Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.13-eks-fb459a0", GitCommit:"55bd5d5cb7d32bc35e4e050f536181196fb8c6f7", GitTreeState:"clean", BuildDate:"2022-10-24T20:38:50Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.16-eks-48e63af", GitCommit:"e6332a8a3feb9e0fe3db851878f88cb73d49dd7a", GitTreeState:"clean", BuildDate:"2023-01-24T19:18:15Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Thank you.

The text was updated successfully, but these errors were encountered:

passenger31 · 2023-03-23T11:20:32Z

note: the container doesn't have strace on it, I compiled it statically from source and copied it across to the container in order to debug.

haarchri · 2023-03-23T17:05:48Z

we faced today the same issue with our bottlerocket installation

nabuskey · 2023-03-23T17:59:13Z

Looks like it is related to the number of configured available namespaces. Thank you very much for digging deep and providing relevant information.

This can be configured with

[settings.kernel.sysctl]
"user.max_user_namespaces" = "16384"

Looks like you can do it via cli too. See: bottlerocket-os/bottlerocket#1158

ytsarev · 2023-03-23T18:52:21Z

@nabuskey I quickly checked the Bottlerocket doc

https://github.com/bottlerocket-os/bottlerocket/blob/develop/SECURITY_GUIDANCE.md#do-not-run-containers-as-uid-0

It says that Bottlerocket does not currently support user namespaces.
I am not sure if the info is up to date, but it matches the debug output from the Issue description.

There is a good doc on security requirements for xfn runner https://docs.crossplane.io/knowledge-base/guides/composition-functions/#the-xfn-runner .

It says

Regardless of capabilities xfn always runs each Composition Function as an unprivileged user. That user will appear to be root inside the Composition Function container thanks to user_namespaces(7).

So it looks like if user namespaces are not supported by Bottlerocket in general then xfn runner will not work

negz · 2023-03-23T19:21:02Z

If Bottlerocket doesn't allow user namespaces this unfortunately isn't likely to be something we can fix. This is part of why we think of xfn as the "reference" Composition Function runner and expect that other implementations that make different tradeoffs will exist.

nabuskey · 2023-03-23T21:13:40Z

It does seem to work. Maybe others can test further.

[ssm-user@control]$ cat /proc/sys/user/max_user_namespaces
0
[ssm-user@control]$ exit

➜  test/container k logs -n crossplane-system  crossplane-77b6966474-7wknn crossplane | grep 'no space left on device'
2023-03-23T21:03:24Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562068", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:24Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562079"}, "reason": "ComposeResources"}
2023-03-23T21:03:25Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:25Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}
2023-03-23T21:03:25Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:25Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}
2023-03-23T21:03:26Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:26Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}
2023-03-23T21:03:34Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:34Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}
2023-03-23T21:03:50Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:50Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}
2023-03-23T21:04:22Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:04:22Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}

➜  test/container k delete -f claim.yaml
objectstorage.awsblueprints.io "standard-object-storage" deleted

[ssm-user@control]$ apiclient -u /settings -X PATCH -d '{"kernel": {"sysctl": {"vm.max_map_count": "262144", "user/max_user_namespaces": "16384"}}}'
[ssm-user@control]$ apiclient -u /tx/commit_and_apply -m POST
["settings.kernel.sysctl.user/max_user_namespaces","settings.kernel.sysctl.\"vm.max_map_count\""]
[ssm-user@control]$ cat /proc/sys/user/max_user_namespaces
16384
[ssm-user@control]$ exit

➜  test/container k apply -f claim.yaml
objectstorage.awsblueprints.io/standard-object-storage created
➜  test/container k logs -n crossplane-system  crossplane-77b6966474-7wknn crossplane | grep 'no space left on device'
2023-03-23T21:03:24Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562068", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:24Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562079"}, "reason": "ComposeResources"}
2023-03-23T21:03:25Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:25Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}
2023-03-23T21:03:25Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:25Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}
2023-03-23T21:03:26Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:26Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}
2023-03-23T21:03:34Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:34Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}
2023-03-23T21:03:50Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:03:50Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}
2023-03-23T21:04:22Z	DEBUG	crossplane	cannot compose resources	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/xobjectstorages.awsblueprints.io", "request": "/standard-object-storage-ncnd2", "uid": "8059346e-5890-4902-934b-e407ddcf24f7", "version": "16562081", "name": "standard-object-storage-ncnd2", "error": "cannot run Composition Function pipeline: cannot run function \"quotable\": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device"}
2023-03-23T21:04:22Z	DEBUG	crossplane.events	cannot compose resources: cannot run Composition Function pipeline: cannot run function "quotable": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device	{"type": "Warning", "object": {"kind":"XObjectStorage","name":"standard-object-storage-ncnd2","uid":"8059346e-5890-4902-934b-e407ddcf24f7","apiVersion":"awsblueprints.io/v1alpha1","resourceVersion":"16562081"}, "reason": "ComposeResources"}

No new log messages with no space left on device

Bottlerocket (AMI ID ami-0a49771f6772955e6)

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
nodeGroups:
  - name: ng-bottlerocket
    instanceType: m5.large
    desiredCapacity: 3
    amiFamily: Bottlerocket
    ami: auto-ssm
    iam:
       attachPolicyARNs:
          - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
          - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
          - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
          - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
    ssh:
        allow: false

➜  test/container k get pod -n crossplane-system -o wide
NAME                                       READY   STATUS    RESTARTS   AGE   IP               NODE                                           NOMINATED NODE   READINESS GATES
crossplane-77b6966474-7wknn                2/2     Running   0          31m   192.168.80.137   ip-192-168-69-59.us-west-2.compute.internal    <none>           <none>
crossplane-rbac-manager-5666767cd6-lwftq   1/1     Running   0          31m   192.168.15.161   ip-192-168-16-243.us-west-2.compute.internal   <none>           <none>
➜  test/container k get node ip-192-168-69-59.us-west-2.compute.internal --show-labels
NAME                                          STATUS   ROLES    AGE   VERSION               LABELS
ip-192-168-69-59.us-west-2.compute.internal   Ready    <none>   56m   v1.24.9-eks-4f83af2   alpha.eksctl.io/nodegroup-name=ng-bottlerocket,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-west-2,failure-domain.beta.kubernetes.io/zone=us-west-2c

passenger31 · 2023-03-24T09:47:52Z

Hey @nabuskey - thanks a lot for testing this.

A question if I may - in your screenshot which you mention 'no new log messages with no space left on device' - the log actually shows the error message - is it a bad copy/paste by any chance?

passenger31 · 2023-03-24T11:48:13Z

Looks like it was a copy/paste issue - I've tested myself and the error is gone.
I'm unsure at this stage if there are any other implications so if anyone has any further information it would be greatly appreciated.

Thank you.

nabuskey · 2023-03-24T18:33:04Z

That was my poor attempt at showing no new error messages were generated after making the change to kernel. I should have filtered old logs out.

The particular setting controls how many user namspaces can be created per user. See: https://www.man7.org/linux/man-pages/man7/user_namespaces.7.html

Bottlerocket does not currently support user namespaces might be referring to subuid and subgid. Not too sure tbh.

I agree with Nic though. It is a reference implementation. You don't have to use it, you can have a grpc endpoint that kicks off a k8s job for example.

negz · 2023-05-04T00:39:42Z

Can anyone confirm whether they've been able to get xfn to work on Bottlerocket? If indeed Bottlerocket doesn't support user namespaces per @ytsarev's comment at #3899 (comment) we'll have to close this as not fixable, but @nabuskey indicates that they do seem to work.

mikebryant · 2023-07-13T14:52:27Z

It's working for us on Bottlerocket :)

haarchri · 2023-07-13T16:35:37Z

@mikebryant which Kubernetes Version and node version you using ?

LCaparelli · 2023-08-03T21:42:02Z

We're hitting this as well:

Status:
  Conditions:
    Last Transition Time:  2023-08-03T21:37:10Z
    Message:               cannot compose resources: cannot run Composition Function pipeline: cannot run function "ingress-function": cannot run container: rpc error: code = Unknown desc = cannot start spark: fork/exec /usr/local/bin/xfn: no space left on device
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced

^ from the XR

Running k8s 1.27:

❯ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.26.0
Kustomize Version: v4.5.7
Server Version: v1.27.3-eks-a5565ad

Node info:

System Info:
  Kernel Version:               5.15.117
  OS Image:                     Bottlerocket OS 1.14.2 (aws-k8s-1.27)
  Operating System:             linux
  Architecture:                 amd64
  Container Runtime Version:    containerd://1.6.20+bottlerocket
  Kubelet Version:              v1.27.3-eks-6f07bbc
  Kube-Proxy Version:           v1.27.3-eks-6f07bbc

Composition, XRD

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  labels:
    crossplane.io/xrd: xpocfunctions.foo.bar
  name: xpocfunctions.foo.bar
spec:
  compositeTypeRef:
    apiVersion: foo.bar/v1alpha1
    kind: XPocFunction
  functions:
  - name: ingress-function
    type: Container
    container:
      image: some-account.dkr.ecr.us-east-1.amazonaws.com/poc-function:latest
---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xpocfunctions.foo.bar
spec:
  group: foo.bar
  names:
    kind: XPocFunction
    plural: xpocfunctions
  claimNames:
    kind: PocFunction
    plural: pocfunctions
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        description: XPocFunction is the Schema for the xpocfunctions API
        properties:
          spec:
            type: object
            properties:
              enabled:
                type: boolean
                description: Whether or not this XPocFunction is enabled.

LCaparelli · 2023-08-04T19:53:30Z

Folks, I opened a discussion in Bottlerocket's repo: bottlerocket-os/bottlerocket#3318

From what I've been told there, it seems that the distro does support user namespaces, but limits it to 0 by default. This can be changed via user data, when launching the instance.

For the kernel feature specifically, user namespaces are supported today but disabled by default, meaning that user.max_user_namespaces is set to 0. This can be changed via user data at launch; if you change it, and all you need is the ability to create new user namespaces within a container, then that will work. The default limit of zero is motivated by the security risks of user namespaces: they imply the ability to create child processes with capabilities that the parent process doesn't possess, which can expose a lot more kernel surface area to otherwise untrusted processes.

I still need to test this, but we might be able to make it work.

EDIT: just realized that's what @nabuskey suggested. My bad 🙇

LCaparelli · 2023-08-08T20:51:46Z

Just confirming: it does seem to work fine with the adjustment to the kernel configuration, setting a non-zero limit of user namespaces.

In the TOML settings (launch template's user data), for example:

[settings.kernel.sysctl]
"user.max_user_namespaces" = "16384"

Careful with the quotes on the left side, they're necessary. Otherwise, the dot in the setting name will be interpreted as a new section within the TOML document.

Tested it out with a silly echo function, that just spits out stdin to stdout (needed a new image to make cat - the entrypoint, maybe configuring "command" and "args" would be a nice luxury once we get more important stuff out of the way during alpha/beta).

  functions:
  - name: ingress-function
    type: Container
    container:
      image: lcaparelli/echo-stdin:latest

Again thanks @nabuskey for the insight, sorry for missing it the first time around.

haarchri · 2023-08-14T12:50:43Z

@LCaparelli thanks tested this today in one of our clusters - its working ;)

passenger31 added the bug Something isn't working label Mar 23, 2023

jpoley mentioned this issue Mar 23, 2023

Promote Composition Functions to beta #3751

Closed

negz added composition functions labels Mar 23, 2023

pedjak mentioned this issue Jun 12, 2023

support for fromConnectionSecretKey patch #2772

Closed

phisco mentioned this issue Jun 19, 2023

Require containerized Composition Functions #4138

Closed

haarchri closed this as completed Aug 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XFN doesn't run on AWS Bottlerocket #3899

XFN doesn't run on AWS Bottlerocket #3899

passenger31 commented Mar 23, 2023 •

edited

Loading

passenger31 commented Mar 23, 2023

haarchri commented Mar 23, 2023

nabuskey commented Mar 23, 2023 •

edited

Loading

ytsarev commented Mar 23, 2023

negz commented Mar 23, 2023

nabuskey commented Mar 23, 2023 •

edited

Loading

passenger31 commented Mar 24, 2023

passenger31 commented Mar 24, 2023

nabuskey commented Mar 24, 2023

negz commented May 4, 2023

mikebryant commented Jul 13, 2023

haarchri commented Jul 13, 2023

LCaparelli commented Aug 3, 2023 •

edited

Loading

LCaparelli commented Aug 4, 2023 •

edited

Loading

LCaparelli commented Aug 8, 2023 •

edited

Loading

haarchri commented Aug 14, 2023

XFN doesn't run on AWS Bottlerocket #3899

XFN doesn't run on AWS Bottlerocket #3899

Comments

passenger31 commented Mar 23, 2023 • edited Loading

What happened?

How can we reproduce it?

What environment did it happen in?

passenger31 commented Mar 23, 2023

haarchri commented Mar 23, 2023

nabuskey commented Mar 23, 2023 • edited Loading

ytsarev commented Mar 23, 2023

negz commented Mar 23, 2023

nabuskey commented Mar 23, 2023 • edited Loading

passenger31 commented Mar 24, 2023

passenger31 commented Mar 24, 2023

nabuskey commented Mar 24, 2023

negz commented May 4, 2023

mikebryant commented Jul 13, 2023

haarchri commented Jul 13, 2023

LCaparelli commented Aug 3, 2023 • edited Loading

LCaparelli commented Aug 4, 2023 • edited Loading

LCaparelli commented Aug 8, 2023 • edited Loading

haarchri commented Aug 14, 2023

passenger31 commented Mar 23, 2023 •

edited

Loading

nabuskey commented Mar 23, 2023 •

edited

Loading

nabuskey commented Mar 23, 2023 •

edited

Loading

LCaparelli commented Aug 3, 2023 •

edited

Loading

LCaparelli commented Aug 4, 2023 •

edited

Loading

LCaparelli commented Aug 8, 2023 •

edited

Loading