Example Service Entry load balancing issue and mTLS connection

[iandyh]

Describe the bug
This is not a bug for usage of admiral per se. I am following the docs: https://istio.io/latest/blog/2020/multi-cluster-mesh-automation/ to understand the idea behind Admiral but encountered the following issue:

1. After creating the service entry:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: se-test
  namespace: caas-sentinel
spec:
  endpoints:
  - address: sample-app.caas-sentinel.svc.cluster.local
    locality: jpe1/jpe1b
    ports:
      http: 80
  hosts:
  - productpage.global
  location: MESH_INTERNAL
  ports:
  - name: http
    number: 80
    protocol: http
  addresses:
  - 240.0.0.10
  resolution: DNS

the Envoy configuration does not look correct to me:

         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "sample-app.caas-sentinel.svc.cluster.local",
              "port_value": 80
             }
            }
           },
           "load_balancing_weight": 1
          }
         ],
         "load_balancing_weight": 1
        }

After I changed to using STATIC and actual pod IP, the configuration looks correct. sidecar proxy will do the direct pod load balancing. I am not sure whether this is a bug(probably by Istio) or by design. But it will be great if someone can help to confirm.

Second issue is, with the same service entry above, the mTLS connection to sample-app.caas-sentinel does not work. I got the upstream connect error or disconnect/reset before headers. reset reason: connection termination error.

Steps To Reproduce
Istio 1.6
Create above service entry
Turn on target remote service mTLS as shown here: https://istio.io/latest/docs/tasks/security/authentication/authn-policy/

Expected behavior
sidecar proxy should do the pod load balancing instead of calling the service FQDN directly.
mTLS should work with local service.

Thanks a lot for your help!

[aattuluri]

@iandyh
i) above is a known limitation in Istio, I just raised a feature request.
istio/istio#25898

For ii) did you make a http request to sample-app.caas-sentinel? Try using the full name to see if that works, sample-app.caas-sentinel.svc.cluster.local

[iandyh]

@aattuluri Thanks for the reply!

i) Will keep an eye at that issue.
ii) If I directly call the remote service with k8s FQDN, mTLS will work. But if I use the productpage.global it will fail. I guess SNI verification fails here.

[aattuluri]

@iandyh
You can configure subjectAltNames in your service entry if you are sure that its SAN verification thats failing

[iandyh]

@aattuluri After adding

  subjectAltNames:
  - sample-app.caas-sentinel.svc.cluster.local

It does not help...

[aattuluri]

@iandyh Can you add this to your deployment -> spec -> template -> annotations, make some requests and share the logs?

sidecar.istio.io/logger: debug

[aattuluri]

I am not sure if its a SAN verification failure, debug logging might help identity the actual issue.

[iandyh]

@aattuluri Looking at the logs, it only says 503 with UC flag(Upstream connection termination).

[2020-08-03T06:27:08.472Z] "GET / HTTP/1.1" 503 UC "-" "-" 0 95 1 - "-" "curl/7.70.0-DEV" "10845a32-7cfd-911d-b0c0-aab47eec9238" "productpage.global" "100.102.139.207:80" outbound|80||productpage.global 100.86.65.151:40152 100.99.126.31:80 100.86.65.151:38286 - default

[aattuluri]

This is not an admiral bug, closing this issue.