[nginx] sticky upstream not used

[stp-ip]

With beta.11 it seems the sticky upstream is created next to the default one, but it is actually never used within the server block.

excerpt from nginx.conf:

    upstream sticky-dev-page-main-80 {
        sticky hash=sha1 name=route  httponly;
        server 10.4.1.45:8080 max_fails=0 fail_timeout=0;
        server 10.4.1.42:8080 max_fails=0 fail_timeout=0;
        server 10.4.1.44:8080 max_fails=0 fail_timeout=0;
    }

    upstream dev-page-main-80 {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;
        server 10.4.1.45:8080 max_fails=0 fail_timeout=0;
        server 10.4.1.42:8080 max_fails=0 fail_timeout=0;
        server 10.4.1.44:8080 max_fails=0 fail_timeout=0;
    }
...
    server {
        server_name page.dev.services.example.nl;
        listen 80;
        listen [::]:80;
        set $proxy_upstream_name "-";
...
        location / {
            set $proxy_upstream_name "dev-page-main-80";

            # enforce ssl on server side
            if ($pass_access_scheme = http) {
                return 301 https://$best_http_host$request_uri;
            }
            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;
...
            proxy_pass http://dev-page-main-80;

[stp-ip]

Also as far as I can see the configmap config "enable-sticky-sessions" doesn't matter here at all.

[gmauleon]

I can confirm that in beta.10 too which I was using ATM

[Gacko]

The referenced pull request is not fixing this issue since there is no rewrite in this use case. Or at least it even occurs without rewrites.

I figured out how to make it work without a code change. Defining an explicit path makes the nginx.conf contain sticky upstreams in proxy_pass directives:

   - host: my-host.example.com
     http:
       paths:
-      - backend:
+      - path: /
+        backend:
           serviceName: my-backend-service
           servicePort: http

Here's the complete YAML:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-sticky-ingress
  labels:
    app: my-sticky-app
  annotations:
    ingress.kubernetes.io/affinity: "cookie"
    ingress.kubernetes.io/session-cookie-name: "route"
    ingress.kubernetes.io/session-cookie-hash: "sha1"
spec:
  rules:
  - host: my-host.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: my-backend-service
          servicePort: http

Generated upstreams:

    upstream sticky-default-my-backend-service-http {
        sticky hash=sha1 name=route  httponly;
        server 192.168.1.92:6081 max_fails=0 fail_timeout=0;
        server 192.168.3.114:6081 max_fails=0 fail_timeout=0;
        server 192.168.5.181:6081 max_fails=0 fail_timeout=0;
    }

    upstream default-my-backend-service-http {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;
        server 192.168.1.92:6081 max_fails=0 fail_timeout=0;
        server 192.168.3.114:6081 max_fails=0 fail_timeout=0;
        server 192.168.5.181:6081 max_fails=0 fail_timeout=0;
    }

Generated server block:

server {
        server_name my-host.example.com
        listen 80;
        listen [::]:80;
        set $proxy_upstream_name "-";
        location / {
            set $proxy_upstream_name "sticky-default-my-backend-service-http";

            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_real_ip;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass http://sticky-default-my-backend-service-http;
        }

    }

Unfortunately I'm not this good with Go. So maybe someone else could determine the real issue and fix it.

[Gacko]

I'm not sure and I don't want to downgrade nginx now but is it possible that I need to set the path explicitly since #871? Or should path be set anyway?

[aledbf]

@Gacko you should set a path

[Gacko]

So the path is required even without sticky sessions? Okay. Then I guess this issue can stay closed. I was wondering since all my non-sticky ingresses were working even without the path.

[Gacko]

Here https://kubernetes.io/docs/concepts/services-networking/ingress/#name-based-virtual-hosting it's documented without a path.

[stp-ip]

@aledbf Can I assume this means the documentation is incorrect? If yes, I would open up an issue for the docs.

[Gacko]

... or could it be defaulted to path: /? proxy_pass can only be declared inside of a location block. So technically we always need a path.

Being able to omit it would be far better for those only interested in name based virtual hosting. They probably do not define path in their ingresses at the moment. Otherwise people need to always explicitly define path: / to make sure all features work as expected.

[shavo007]

thanks for that @Gacko . i didnt explicitly set a path and got the same issue as you. after setting path was fine.