podman-rootless 2.2 eats ENV variables from the packaging process

[cruwe]

/kind bug

Description

On Ubuntu 20.04, after upgrading to 2.2.0 (or freshly installing, it is a version issue), inside containers in pods created with podman kube play, environment variables from the packaging process (ENV) are missing / invsibile.

Steps to reproduce the issue:

1. Package with

FROM debian:buster
ENV ISTHERE=ANYBODYOUTTHERE

and podman build -t try --format=docker -f Containerfile .

2. Create a pod with

---
apiVersion: v1
kind: Pod
metadata:
  name: try
spec:
  containers:
    - name: try
      image: try
      command:
        - /bin/sleep
        - "3600"

and podman play kube try.yaml.

Describe the results you received:

With 2.1.1 (is podman-rootless=2.1.1~2)

$ podman exec try_pod-try env | sort
HOME=/root
HOSTNAME=try_pod
ISTHERE=ANYBODYOUTTHERE
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TERM=xterm

With 2.2.0 (is podman-rootless=2.2.0~2)

$ podman exec -ti try_pod-try env | sort
container=podman
HOME=/root
HOSTNAME=try_pod
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TERM=xterm

Describe the results you expected:

I'd rather like to have environment variables from the packaging process left intact.

Output of podman version:

Working

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Broken

Version:      2.2.0
API Version:  2.1.0
Go Version:   go1.15.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:
(Freshly upgraded vbox)

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 1
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: morgue
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-56-generic
  linkmode: dynamic
  memFree: 1447694336
  memTotal: 2020876288
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 0.4.3
      commit: 2244b9b6461afeccad1678fac3d6e478c28b4ad6
  swapFree: 0
  swapTotal: 0
  uptime: 40m 13.2s
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/hal9000/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/hal9000/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 6
  runRoot: /run/user/1000/containers
  volumePath: /home/hal9000/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.2.0

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/unknown 2.2.0~2 amd64
podman/unknown 2.2.0~2 arm64
podman/unknown 2.2.0~2 armhf
podman/unknown 2.2.0~2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes, regrettably for several hours, initially thought I broke sth, until I understood what's happening here.

Additional environment details (AWS, VirtualBox, physical, etc.):

Same behaviour on physical as well as on VirtualBox. 2.1 works, 2.2 doesn't.

[mheon]

@nalind @TomSweeneyRedHat PTAL

[rhatdan]

I don't believe this is a Build issue, I just built an image based on the Containerfile and I see the environment variables.

[rhatdan]

Could you do a podman run try printenv
And see if the environment variables are set? If yes then this is a podman play/generate issue.

[mheon]

Ahhh, this could be the play kube rewrite by @baude

[cruwe]

HI ... please excuse if I created extra work, I could have included that information explicitly. It happens only when creating the containers with podman play. podman run is fine.

[cruwe]

Having been pointed in the general direction what to look for,

$ ./bin/podman version
Version:      2.2.0-dev
API Version:  2.0.0
Go Version:   go1.13.8
Git Commit:   9e71fc4552b156dc027004a808d1b74552af4700
Built:        Sat Dec  5 03:30:51 2020
OS/Arch:      linux/amd64

does not eat the ENV var

$ ./bin/podman play kube try.yaml
a container exists with the same name ("try") as the pod in your YAML file; changing pod name to try_pod
Pod:
bd14f2e8c6b2bc3002526c6a9a8fc9e7b7d902014ec53dbe000ad25937099f25
Container:
fa664d023b6ae162c42ad0b940ec801afc78e1de533276d8cbd40b8874bfec94

$ podman exec -ti try_pod-try env | sort   
HOME=/root
HOSTNAME=try_pod
ISTHERE=ANYBODYOUTTHERE
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TERM=xterm

while

$ ./bin/podman version
Version:      2.2.0-dev
API Version:  2.0.0
Go Version:   go1.13.8
Git Commit:   bb07b8c9f1d6b8f24fef91d96cc9c3a3ea381d20
Built:        Sat Dec  5 03:34:11 2020
OS/Arch:      linux/amd64

does:

$ ./bin/podman play kube try.yaml       
a container exists with the same name ("try") as the pod in your YAML file; changing pod name to try_pod
Pod:
17458b4d4ba2140f60cdf3b8f5f4920aa0a920a9d57df363c8b83efd97d4bbcc
Container:
ddd2f9364775f24b407e581239221d3b265d60a9bbd8f2a4a900ca8241b904e8

$ podman exec -ti try_pod-try env | sort          
container=podman
HOME=/root
HOSTNAME=try_pod
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TERM=xterm

Just trying to be intelligent, please take no offence if I should be far off. Does

podman/pkg/specgen/generate/kube/kube.go

Lines 149 to 163 in 8e83799

	// Environment Variables
	envs := map[string]string{}
	for _, env := range containerYAML.Env {
	value := envVarValue(env, configMaps)
	envs[env.Name] = value
	}
	for _, envFrom := range containerYAML.EnvFrom {
	cmEnvs := envVarsFromConfigMap(envFrom, configMaps)
	for k, v := range cmEnvs {
	envs[k] = v
	}
	}
	s.Env = envs

populate the environment for the pod completely? Then the variables packaged into the image are indeed missing, which would explain the observation.

[zhangguanzhang]

seemed related to #5140

[cruwe]

I'd be very grateful if you could release the fix for the (probably unintended) breaking change. We are using the podman play feature regularly.

[ahmadalli]

I've just started using podman and wanted to use podman play as an alternative to docket-compose and I faced this issue :)

[ahmadalli]

is this fix going to be released anytime soon?

[zhangguanzhang]

is this fix going to be released anytime soon?

yes

[rhatdan]

podman 3.0 RC1 should happen at the end of this week.