Ingress in bare metal kubernetes environment does not work

[apannwitz]

NGINX Ingress controller version: latest (helm-chart-3.19.0 and NGINX: 0.43.0 (I tried several versions via helm and via command line))

Kubernetes version (use kubectl version): v1.19.6

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Debian 10 (Buster)
• Kernel (e.g. uname -a): inux k8s-con-master1 4.15.0-130-generic fix typo #134-Ubuntu SMP Tue Jan 5 20:46:26 UTC 2021 x86_64 x86_64 x86_64 GNU/Linu
• Install tools:
• Others:

What happened:

Docker container will not start. Logfile gives the following output:

I0110 19:50:26.062615 7 flags.go:206] "Watching for Ingress" class="nginx"
W0110 19:50:26.062671 7 flags.go:211] Ingresses with an empty class will also be processed by this Ingress controller
W0110 19:50:26.063161 7 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0110 19:50:26.063379 7 main.go:241] "Creating API client" host="https://10.96.0.1:443"

NGINX Ingress controller
Release: v0.43.0
Build: f3f6da1
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.6

I0110 19:50:26.074187 7 main.go:285] "Running in Kubernetes cluster" major="1" minor="19" git="v1.19.6" state="clean" commit="fbf646b339dc52336b55d8ec85c181981b86331a" platform="linux/amd64"
I0110 19:50:26.662634 7 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0110 19:50:26.664276 7 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
E0110 19:50:26.668102 7 main.go:134] Invalid IngressClass (Spec.Controller) value "nginx.org/ingress-controller". Should be "k8s.io/ingress-nginx"
F0110 19:50:26.668141 7 main.go:135] IngressClass with name nginx is not valid for ingress-nginx (invalid Spec.Controller)
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc00000e001, 0xc0001ac200, 0x81, 0x1e1)
k8s.io/klog/v2@v2.4.0/klog.go:1026 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x268b4e0, 0xc000000003, 0x0, 0x0, 0xc000411ce0, 0x25e101f, 0x7, 0x87, 0x0)
k8s.io/klog/v2@v2.4.0/klog.go:975 +0x19b
k8s.io/klog/v2.(*loggingT).printf(0x268b4e0, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1a60447, 0x52, 0xc0002652a0, 0x1, ...)
k8s.io/klog/v2@v2.4.0/klog.go:750 +0x191
k8s.io/klog/v2.Fatalf(...)
k8s.io/klog/v2@v2.4.0/klog.go:1502
main.main()
k8s.io/ingress-nginx/cmd/nginx/main.go:135 +0xf06
goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x268b4e0)
k8s.io/klog/v2@v2.4.0/klog.go:1169 +0x8b
created by k8s.io/klog/v2.init.0
k8s.io/klog/v2@v2.4.0/klog.go:417 +0xdf
goroutine 12 [IO wait]:
internal/poll.runtime_pollWait(0x7f0fdcdb3a58, 0x72, 0x1bfd620)
runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000442518, 0x72, 0x1bfd600, 0x25fb608, 0x0)
internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000442500, 0xc0000d8900, 0x8de, 0x8de, 0x0, 0x0, 0x0)
internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000442500, 0xc0000d8900, 0x8de, 0x8de, 0x203000, 0x74b2db, 0xc000246160)
net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0002b0000, 0xc0000d8900, 0x8de, 0x8de, 0x0, 0x0, 0x0)
net/net.go:182 +0x8e
crypto/tls.(*atLeastReader).Read(0xc0002a1660, 0xc0000d8900, 0x8de, 0x8de, 0x383, 0x844, 0xc0003d5710)
crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc000246280, 0x1bf95c0, 0xc0002a1660, 0x40b665, 0x1817a60, 0x1986740)
bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc000246000, 0x1bfb8a0, 0xc0002b0000, 0x5, 0xc0002b0000, 0x372)
crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000246000, 0x0, 0x0, 0xc0003d5d18)
crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(...)
crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc000246000, 0xc0000f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
crypto/tls/conn.go:1252 +0x15f
bufio.(*Reader).Read(0xc00007c8a0, 0xc00041e3b8, 0x9, 0x9, 0xc0003d5d18, 0x1abba00, 0x9551ab)
bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x1bf9420, 0xc00007c8a0, 0xc00041e3b8, 0x9, 0x9, 0x9, 0xc00004c070, 0x0, 0x1bf97a0)
io/io.go:314 +0x87
io.ReadFull(...)
io/io.go:333
golang.org/x/net/http2.readFrameHeader(0xc00041e3b8, 0x9, 0x9, 0x1bf9420, 0xc00007c8a0, 0x0, 0x0, 0xc0003d5dd0, 0x46cf65)
golang.org/x/net@v0.0.0-20201110031124-69a78807bb2b/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00041e380, 0xc000267740, 0x0, 0x0, 0x0)
golang.org/x/net@v0.0.0-20201110031124-69a78807bb2b/http2/frame.go:492 +0xa5
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0003d5fa8, 0x0, 0x0)
golang.org/x/net@v0.0.0-20201110031124-69a78807bb2b/http2/transport.go:1819 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000402a80)
golang.org/x/net@v0.0.0-20201110031124-69a78807bb2b/http2/transport.go:1741 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
golang.org/x/net@v0.0.0-20201110031124-69a78807bb2b/http2/transport.go:705 +0x6c5

What you expected to happen:

Docker container shall run...

helm install nginx-ingress-public ingress-nginx/ingress-nginx -n nginx-ingress --create-namespace
--namespace ingress-nginx --set controller.hostNetwork=true
--set controller.service.type=ClusterIP

Install the ingress controller

...same error messages when I try to install that way:

kubectl apply -f https://github.com/raw/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

/kind bug

[apannwitz]

BTW: Installing another helm chart "nginx-stable/nginx-ingress" does work, but helm chart "ingress-nginx/ingress-nginx" does not work.

[aledbf]

E0110 19:50:26.668102 7 main.go:134] Invalid IngressClass (Spec.Controller) value "nginx.org/ingress-controller". Should be "k8s.io/ingress-nginx"

It seems you created an invalid IngressClass.

The ingressClass should be something like this:

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: k8s.io/ingress-nginx

[apannwitz]

Hi.

I do not have any ingresses defined. We are talking about an "empty" kubernetes cluster. The first step I did was to install ingress with:

kubectl apply -f https://github.com/raw/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

or vi helm with e.g.:

helm install nginx-ingress-public ingress-nginx/ingress-nginx -n nginx-ingress --create-namespace
--namespace ingress-nginx --set controller.hostNetwork=true
--set controller.service.type=ClusterIP

Ingress will not start because of the error message above.

[aledbf]

@apannwitz please execute kubectl get ingressclass

[apannwitz]

NAME CONTROLLER PARAMETERS AGE
nginx nginx.org/ingress-controller 30d

OMG, Thank you very much. There seems to be an ingress that Rancher doesn't show me in GUI. Any idea how to get rid of it?

[aledbf]

OMG, Thank you very much. There seems to be an ingress that Rancher doesn't show me in GUI. Any idea how to get rid of it?

That should be like any other k8s deployment, just kubectl delete. Make sure you also delete the ingressClass

[apannwitz]

Thank you. I deleted it with "kubectl delete ingressclass nginx" and afterwards I had to do some cleanups. Now it works like a charm.