proxy_set_header doesn't allow overriding "X-Scheme"

[swisspol]

NGINX Ingress controller version:
0.30.0

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:16:51Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.10-eks-bac369", GitCommit:"bac3690554985327ae4d13e42169e8b1c2f37226", GitTreeState:"clean", BuildDate:"2020-02-26T01:12:54Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS / EKS
• OS (e.g. from /etc/os-release): quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
• Kernel (e.g. uname -a):
• Install tools:
• Others: Worker nodes are using standard EKS Linux AMIs

What happened:

I need to set the X-Scheme header on requests forwarded to backend. That's because I want to run pgAdmin which needs this header set when running behind Nginx Ingress Controller with an AWS classic ELB in front of that does TLS termination (so L3 load balancer, not L7). Otherwise, it thinks its URL is http://my-pgadmin.com instead of https://my-pgadmin.com and a bunch of things break.

Here's the ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: httpbin
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Scheme "https";
      proxy_set_header X-Bar foo3;
spec:
  rules:
    - host: jenkins-httpbin.secrethouse.party
      http:
        paths:
        - backend:
            serviceName: httpbin
            servicePort: http

Here's what httpbin returns as received headers:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "jenkins-httpbin.secrethouse.party", 
    "User-Agent": "curl/7.54.0", 
    "X-Bar": "foo4", 
    "X-Forwarded-Host": "jenkins-httpbin.secrethouse.party", 
    "X-Scheme": "http,https"
  }, 
  "origin": "172.31.29.39", 
  "url": "http://jenkins-httpbin.secrethouse.party/get"
}

X-Scheme is set to http,https instead of https.

What you expected to happen:

X-Scheme is set to https. I can't also figure out where is X-Scheme set in the first place by Nginx Ingress controller.

How to reproduce it:

1. Create a deployment & service with some app that echoes HTTP headers like https://hub.docker.com/r/kennethreitz/httpbin/
2. Create an ingress as defined above
3. Make a GET request to ingress and check value of X-Scheme header as received by app

Anything else we need to know:

Other things I tried:

• Setting X-Forwarded-Proto but it's plain ignored
• Using a configmap and proxy-set-headers but setting X-Scheme in there also results in http,https

/kind bug

[swisspol]

I also tried #4096 (comment) and #2724 (comment) to set $pass_access_scheme and $pass_port but it still doesn't work: X-Scheme is always http.

[swisspol]

Based on http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header, it appears that proxy_set_header doesn't override but appends, so the behavior would be expected. But still, I'd like to find a way to set X-Scheme to https.

[teochenglim]

@swisspol #4096 does works for me. =)
I am returning favour because i found him by your post.

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  use-proxy-protocol: "false"
  use-forwarded-headers: "true"

[teochenglim]

or if you using helm3 @swisspol

$ helm upgrade nginx-ingress stable/nginx-ingress \
  --set-string controller.config.use-proxy-protocol="false" \
  --set-string controller.config.use-forwarded-headers="true"

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[brsolomon-deloitte]

Because of the way that nginx.conf is templated out it has become prohibitively difficult to set these headers. They are dictated by the values of pass_port and pass_access_scheme.

One way to hard-override them is with a custom Lua plugin.

See #6358 (comment) for an example.