Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deploying DeathStarBenchmark/Social-Network (thrift protocol) #423

Open
yizhengx opened this issue May 27, 2024 · 4 comments
Open

Deploying DeathStarBenchmark/Social-Network (thrift protocol) #423

yizhengx opened this issue May 27, 2024 · 4 comments

Comments

@yizhengx
Copy link

yizhengx commented May 27, 2024 •
edited
Loading

Hi there, we're doing research on microservice and now working on deploying DeathStarBenchmark/Social-Network (using thrift RPC) on Aeraki. We just deploy Aeraki using make demo to install istio, aeraki and some examples. Then we deploy social-network benchmark by first labeling namespace then using helm-chart with port name as 'tcp-metaprotocol-thrift-xxx' for protocol discovery.

Example user-timeline-service definition:

yizhengx@node0:~/aeraki-benchmark$ k describe service user-timeline-service -n social-net 
Name:              user-timeline-service
Namespace:         social-net
Labels:            app.kubernetes.io/managed-by=Helm
Annotations:       meta.helm.sh/release-name: social-net
                   meta.helm.sh/release-namespace: social-net
Selector:          service=user-timeline-service
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.229.11
IPs:               10.96.229.11
Port:              tcp-metaprotocol-thrift-9090  9090/TCP
TargetPort:        9090/TCP
Endpoints:         192.168.235.163:9090
Session Affinity:  None
Events:            <none>

I checked the istioctl proxy-status which looks fine where listeners are using codec.thrift

yizhengx@node0:~/aeraki-benchmark/social-net/helm-chart/socialnetwork$ istioctl proxy-status user-timeline-service-59877c44b9-pkg5q.social-net
Clusters Match
--- Istiod Listeners
+++ Envoy Listeners
@@ -6166,73 +6166,73 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "accessLog": [
+                                                "access_log": [
                                                             {
                                                                         "name": "envoy.access_loggers.file",
-                                                                        "typedConfig": {
+                                                                        "typed_config": {
                                                                                     "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
-                                                                                    "logFormat": {
-                                                                                                "textFormatSource": {
-                                                                                                            "inlineString": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
+                                                                                    "log_format": {
+                                                                                                "text_format_source": {
+                                                                                                            "inline_string": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
                                                                                                         }
                                                                                             },
                                                                                     "path": "/dev/stdout"
                                                                                 }
                                                                     }
                                                         ],
-                                                "applicationProtocol": "thrift",
+                                                "application_protocol": "thrift",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.thrift"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "thrift-sample-server.meta-thrift.svc.cluster.local"
+                                                                                    "destination_service": "thrift-sample-server.meta-thrift.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "thrift-sample-server.meta-thrift.svc.cluster.local_9090"
-                                                        },
-                                                "statPrefix": "outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local",
+                                                            "route_config_name": "thrift-sample-server.meta-thrift.svc.cluster.local_9090"
+                                                        },
+                                                "stat_prefix": "outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local",
                                                 "tracing": {
-                                                            "clientSampling": {
+                                                            "client_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "overallSampling": {
+                                                            "overall_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "randomSampling": {
+                                                            "random_sampling": {
                                                                         "value": 100
                                                                     }
                                                         }
                                             }
                                     }
                                 }
                             ]
@@ -10792,73 +10792,73 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "accessLog": [
+                                                "access_log": [
                                                             {
                                                                         "name": "envoy.access_loggers.file",
-                                                                        "typedConfig": {
+                                                                        "typed_config": {
                                                                                     "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
-                                                                                    "logFormat": {
-                                                                                                "textFormatSource": {
-                                                                                                            "inlineString": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
+                                                                                    "log_format": {
+                                                                                                "text_format_source": {
+                                                                                                            "inline_string": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
                                                                                                         }
                                                                                             },
                                                                                     "path": "/dev/stdout"
                                                                                 }
                                                                     }
                                                         ],
-                                                "applicationProtocol": "dubbo",
+                                                "application_protocol": "dubbo",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.dubbo"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "org.apache.dubbo.samples.basic.api.demoservice"
+                                                                                    "destination_service": "org.apache.dubbo.samples.basic.api.demoservice"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "org.apache.dubbo.samples.basic.api.demoservice_20880"
-                                                        },
-                                                "statPrefix": "outbound|20880||org.apache.dubbo.samples.basic.api.demoservice",
+                                                            "route_config_name": "org.apache.dubbo.samples.basic.api.demoservice_20880"
+                                                        },
+                                                "stat_prefix": "outbound|20880||org.apache.dubbo.samples.basic.api.demoservice",
                                                 "tracing": {
-                                                            "clientSampling": {
+                                                            "client_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "overallSampling": {
+                                                            "overall_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "randomSampling": {
+                                                            "random_sampling": {
                                                                         "value": 100
                                                                     }
                                                         }
                                             }
                                     }
                                 }
                             ]
@@ -10912,73 +10912,73 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "accessLog": [
+                                                "access_log": [
                                                             {
                                                                         "name": "envoy.access_loggers.file",
-                                                                        "typedConfig": {
+                                                                        "typed_config": {
                                                                                     "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
-                                                                                    "logFormat": {
-                                                                                                "textFormatSource": {
-                                                                                                            "inlineString": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
+                                                                                    "log_format": {
+                                                                                                "text_format_source": {
+                                                                                                            "inline_string": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
                                                                                                         }
                                                                                             },
                                                                                     "path": "/dev/stdout"
                                                                                 }
                                                                     }
                                                         ],
-                                                "applicationProtocol": "dubbo",
+                                                "application_protocol": "dubbo",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.dubbo"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "org.apache.dubbo.samples.basic.api.secondservice"
+                                                                                    "destination_service": "org.apache.dubbo.samples.basic.api.secondservice"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "org.apache.dubbo.samples.basic.api.secondservice_20880"
-                                                        },
-                                                "statPrefix": "outbound|20880||org.apache.dubbo.samples.basic.api.secondservice",
+                                                            "route_config_name": "org.apache.dubbo.samples.basic.api.secondservice_20880"
+                                                        },
+                                                "stat_prefix": "outbound|20880||org.apache.dubbo.samples.basic.api.secondservice",
                                                 "tracing": {
-                                                            "clientSampling": {
+                                                            "client_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "overallSampling": {
+                                                            "overall_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "randomSampling": {
+                                                            "random_sampling": {
                                                                         "value": 100
                                                                     }
                                                         }
                                             }
                                     }
                                 }
                             ]

Routes Match (RDS last loaded at Mon, 27 May 2024 04:30:15 CDT)

However, the traces from the istio-proxy doesn't seem to recognize the protocol

2024-05-27T09:00:18.282725Z	info	FLAG: --concurrency="0"
2024-05-27T09:00:18.282755Z	info	FLAG: --domain="social-net.svc.cluster.local"
2024-05-27T09:00:18.282806Z	info	FLAG: --help="false"
2024-05-27T09:00:18.282813Z	info	FLAG: --log_as_json="false"
2024-05-27T09:00:18.282819Z	info	FLAG: --log_caller=""
2024-05-27T09:00:18.282829Z	info	FLAG: --log_output_level="default:debug"
2024-05-27T09:00:18.282834Z	info	FLAG: --log_rotate=""
2024-05-27T09:00:18.282838Z	info	FLAG: --log_rotate_max_age="30"
2024-05-27T09:00:18.282842Z	info	FLAG: --log_rotate_max_backups="1000"
2024-05-27T09:00:18.282847Z	info	FLAG: --log_rotate_max_size="104857600"
2024-05-27T09:00:18.282851Z	info	FLAG: --log_stacktrace_level="default:none"
2024-05-27T09:00:18.282859Z	info	FLAG: --log_target="[stdout]"
2024-05-27T09:00:18.282864Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2024-05-27T09:00:18.282868Z	info	FLAG: --outlierLogPath=""
2024-05-27T09:00:18.282873Z	info	FLAG: --profiling="true"
2024-05-27T09:00:18.282885Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2024-05-27T09:00:18.282891Z	info	FLAG: --proxyLogLevel="warning"
2024-05-27T09:00:18.282902Z	info	FLAG: --s2a_enable_appengine_dialer="false"
2024-05-27T09:00:18.282906Z	info	FLAG: --s2a_timeout="3s"
2024-05-27T09:00:18.282911Z	info	FLAG: --serviceCluster="istio-proxy"
2024-05-27T09:00:18.282916Z	info	FLAG: --stsPort="0"
2024-05-27T09:00:18.282920Z	info	FLAG: --templateFile=""
2024-05-27T09:00:18.282924Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2024-05-27T09:00:18.282929Z	info	FLAG: --vklog="0"
2024-05-27T09:00:18.282935Z	info	Version 1.18-dev-592868ecccb3270a3af17000fb41574a3cc0b888-dirty-Modified
2024-05-27T09:00:18.286288Z	info	Maximum file descriptors (ulimit -n): 1048576
2024-05-27T09:00:18.339234Z	info	Proxy role	ips=[192.168.235.163] type=sidecar id=user-timeline-service-59877c44b9-rxckr.social-net domain=social-net.svc.cluster.local
2024-05-27T09:00:18.339333Z	info	Apply proxy config from env {"tracing":{"zipkin":{"address":"zipkin.istio-system:9411"},"sampling":100},"proxyMetadata":{"ISTIO_META_DNS_CAPTURE":"true"},"proxyStatsMatcher":{"inclusionPrefixes":["thrift","dubbo","kafka","meta_protocol"],"inclusionRegexps":[".*dubbo.*",".*thrift.*",".*kafka.*",".*zookeeper.*",".*meta_protocol.*"]},"holdApplicationUntilProxyStarts":true}

2024-05-27T09:00:18.342271Z	info	cpu limit detected as 2, setting concurrency
2024-05-27T09:00:18.342984Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
holdApplicationUntilProxyStarts: true
proxyAdminPort: 15000
proxyMetadata:
  ISTIO_META_DNS_CAPTURE: "true"
proxyStatsMatcher:
  inclusionPrefixes:
  - thrift
  - dubbo
  - kafka
  - meta_protocol
  inclusionRegexps:
  - .*dubbo.*
  - .*thrift.*
  - .*kafka.*
  - .*zookeeper.*
  - .*meta_protocol.*
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  sampling: 100
  zipkin:
    address: zipkin.istio-system:9411

2024-05-27T09:00:18.343006Z	info	JWT policy is third-party-jwt
2024-05-27T09:00:18.343014Z	info	using credential fetcher of JWT type in cluster.local trust domain
2024-05-27T09:00:18.443799Z	debug	error in getting aws info for iam/info : Get "http://169.254.169.254/latest/meta-data/iam/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-05-27T09:00:18.443959Z	info	Opening status port 15020
2024-05-27T09:00:18.444108Z	info	dns	Starting local udp DNS server on 127.0.0.1:15053
2024-05-27T09:00:18.444348Z	info	dns	Starting local tcp DNS server on 127.0.0.1:15053
2024-05-27T09:00:18.444456Z	info	Workload SDS socket not found. Starting Istio SDS Server
2024-05-27T09:00:18.444476Z	info	CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2024-05-27T09:00:18.444508Z	info	Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2024-05-27T09:00:18.471294Z	info	ads	All caches have been synced up in 190.062096ms, marking server ready
2024-05-27T09:00:18.471872Z	info	xdsproxy	Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2024-05-27T09:00:18.471907Z	info	sds	Starting SDS grpc server
2024-05-27T09:00:18.472492Z	info	starting Http service at 127.0.0.1:15004
2024-05-27T09:00:18.474968Z	debug	Add file var/run/secrets/istio/root-cert.pem watcher
2024-05-27T09:00:18.475013Z	info	Pilot SAN: [istiod.istio-system.svc]
2024-05-27T09:00:18.476927Z	info	Starting proxy agent
2024-05-27T09:00:18.476967Z	info	starting
2024-05-27T09:00:18.477055Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n %g:%#	%v	thread=%t -l warning --component-log-level misc:error --config-yaml 
    {
       "static_resources":{
          "clusters":[
             {
                "name":"aeraki-xds",
                "type":"STRICT_DNS",
                "connect_timeout":"1s",
                "max_requests_per_connection":1,
                "circuit_breakers":{
                   "thresholds":[
                      {
                         "max_connections":100000,
                         "max_pending_requests":100000,
                         "max_requests":100000
                      },
                      {
                         "priority":"HIGH",
                         "max_connections":100000,
                         "max_pending_requests":100000,
                         "max_requests":100000
                      }
                   ]
                },
                "http2_protocol_options":{

                },
                "upstream_connection_options":{
                   "tcp_keepalive":{
                      "keepalive_time":300
                   }
                },
                "load_assignment":{
                   "cluster_name":"aeraki-xds",
                   "endpoints":[
                      {
                         "lb_endpoints":[
                            {
                               "endpoint":{
                                  "address":{
                                     "socket_address":{
                                        "address":"aeraki.istio-system",
                                        "port_value":15010
                                     }
                                  }
                               }
                            }
                         ]
                      }
                   ]
                }
             }
          ]
       }
    }
 --concurrency 2]
2024-05-27T09:00:18.627749Z	info	xdsproxy	connected to upstream XDS server: istiod.istio-system.svc:15012
2024-05-27T09:00:18.698564Z	info	cache	generated new workload certificate	latency=226.730508ms ttl=23h59m59.301455963s
2024-05-27T09:00:18.698639Z	info	cache	Root cert has changed, start rotating root cert
2024-05-27T09:00:18.698669Z	info	ads	XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2024-05-27T09:00:18.698758Z	info	cache	returned workload trust anchor from cache	ttl=23h59m59.301247339s
2024-05-27T09:00:18.767957Z	info	ads	ADS: new connection for node:user-timeline-service-59877c44b9-rxckr.social-net-2
2024-05-27T09:00:18.767992Z	info	ads	ADS: new connection for node:user-timeline-service-59877c44b9-rxckr.social-net-1
2024-05-27T09:00:18.768103Z	info	cache	returned workload certificate from cache	ttl=23h59m59.231911969s
2024-05-27T09:00:18.768173Z	info	cache	returned workload trust anchor from cache	ttl=23h59m59.231835145s
2024-05-27T09:00:18.768788Z	info	ads	SDS: PUSH request for node:user-timeline-service-59877c44b9-rxckr.social-net resources:1 size:1.1kB resource:ROOTCA
2024-05-27T09:00:18.768817Z	info	ads	SDS: PUSH request for node:user-timeline-service-59877c44b9-rxckr.social-net resources:1 size:4.0kB resource:default
2024-05-27T09:00:20.868514Z	info	Readiness succeeded in 2.596457797s
2024-05-27T09:00:20.869064Z	info	Envoy proxy is ready
[2024-05-27T09:00:21.092Z] - 0 - - "-" 0 0 0 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local - 10.101.81.9:27017 192.168.235.163:40336 -
[2024-05-27T09:00:21.594Z] - 0 - - "-" 0 0 0 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local - 10.101.81.9:27017 192.168.235.163:40342 -
[2024-05-27T09:00:22.393Z] - 0 - - "-" 0 0 0 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local - 10.101.81.9:27017 192.168.235.163:40358 -
[2024-05-27T09:00:22.895Z] - 0 - - "-" 0 0 0 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local - 10.101.81.9:27017 192.168.235.163:40360 -
[2024-05-27T09:00:23.692Z] - 0 - - "-" 0 0 0 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local - 10.101.81.9:27017 192.168.235.163:40366 -
[2024-05-27T09:00:24.195Z] - 0 - - "-" 0 0 0 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local - 10.101.81.9:27017 192.168.235.163:40368 -
[2024-05-27T09:00:24.992Z] - 0 - - "-" 0 0 0 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local - 10.101.81.9:27017 192.168.235.163:40376 -
[2024-05-27T09:00:25.493Z] - 0 - - "-" 0 0 0 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local - 10.101.81.9:27017 192.168.235.163:36414 -
[2024-05-27T09:00:26.292Z] - 0 - - "-" 0 0 0 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local - 10.101.81.9:27017 192.168.235.163:36418 -
[2024-05-27T09:00:26.793Z] - 0 - - "-" 0 0 7 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local 192.168.235.163:58566 10.101.81.9:27017 192.168.235.163:36434 -
[2024-05-27T09:00:27.592Z] - 0 - - "-" 0 0 9 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local 192.168.235.163:58570 10.101.81.9:27017 192.168.235.163:36436 -
[2024-05-27T09:00:28.103Z] - 0 - - "-" 0 0 6 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local 192.168.235.163:58572 10.101.81.9:27017 192.168.235.163:36452 -
[2024-05-27T09:00:28.891Z] - 0 - - "-" 0 0 9 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local 192.168.235.163:58584 10.101.81.9:27017 192.168.235.163:36468 -
[2024-05-27T09:00:29.402Z] - 0 - - "-" 0 0 8 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local 192.168.235.163:58596 10.101.81.9:27017 192.168.235.163:36476 -
[2024-05-27T09:00:30.190Z] - 0 - - "-" 0 0 8 "-" "-" outbound|27017||user-timeline-mongodb.social-net.svc.cluster.local 192.168.235.163:58612 10.101.81.9:27017 192.168.235.163:36482 -
[2024-05-27T09:05:22.439Z] - 0 - - "-" 908 871 10324 "-" "-" inbound|9090|| 127.0.0.6:60589 192.168.235.163:9090 192.168.235.177:43954 -
[2024-05-27T09:05:22.443Z] - 0 - - "-" 3466 1568 10608 "-" "-" inbound|9090|| 127.0.0.6:50781 192.168.235.163:9090 192.168.235.177:43960 -
[2024-05-27T09:05:22.439Z] - 0 - - "-" 3774 1650 10613 "-" "-" inbound|9090|| 127.0.0.6:54947 192.168.235.163:9090 192.168.235.177:43932 -
[2024-05-27T09:05:22.398Z] - 0 - - "-" 4076 1732 10657 "-" "-" inbound|9090|| 127.0.0.6:59917 192.168.235.163:9090 192.168.235.177:43898 -
[2024-05-27T09:05:22.439Z] - 0 - - "-" 3624 1609 10618 "-" "-" inbound|9090|| 127.0.0.6:41203 192.168.235.163:9090 192.168.235.177:43980 -
[2024-05-27T09:05:22.337Z] - 0 - - "-" 3772 1650 10731 "-" "-" inbound|9090|| 127.0.0.6:36113 192.168.235.163:9090 192.168.235.177:43850 -
[2024-05-27T09:05:22.344Z] - 0 - - "-" 3923 1691 10735 "-" "-" inbound|9090|| 127.0.0.6:59975 192.168.235.163:9090 192.168.235.177:43880 -
[2024-05-27T09:05:22.439Z] - 0 - - "-" 3921 1691 10640 "-" "-" inbound|9090|| 127.0.0.6:51513 192.168.235.163:9090 192.168.235.177:43964 -
[2024-05-27T09:05:22.443Z] - 0 - - "-" 4379 1814 10639 "-" "-" inbound|9090|| 127.0.0.6:55443 192.168.235.163:9090 192.168.235.177:43972 -
[2024-05-27T09:05:22.443Z] - 0 - - "-" 4376 1814 10644 "-" "-" inbound|9090|| 127.0.0.6:38975 192.168.235.163:9090 192.168.235.177:43982 -
[2024-05-27T09:05:22.379Z] - 0 - - "-" 34532 10014 13958 "-" "-" inbound|9090|| 127.0.0.6:39575 192.168.235.163:9090 192.168.235.177:43888 -
[2024-05-27T09:05:22.810Z] - 0 - - "-" 27294 8046 13837 "-" "-" inbound|9090|| 127.0.0.6:46237 192.168.235.163:9090 192.168.235.177:44012 -
[2024-05-27T09:05:22.439Z] - 0 - - "-" 40721 11695 14809 "-" "-" inbound|9090|| 127.0.0.6:33825 192.168.235.163:9090 192.168.235.177:43998 -
[2024-05-27T09:05:22.439Z] - 0 - - "-" 40708 11695 14816 "-" "-" inbound|9090|| 127.0.0.6:55789 192.168.235.163:9090 192.168.235.177:43948 -
[2024-05-27T09:05:22.337Z] - 0 - - "-" 53829 15262 14979 "-" "-" inbound|9090|| 127.0.0.6:39547 192.168.235.163:9090 192.168.235.177:43864 -
[2024-05-27T09:05:22.444Z] - 0 - - "-" 52337 14852 14877 "-" "-" inbound|9090|| 127.0.0.6:41849 192.168.235.163:9090 192.168.235.177:43996 -
[2024-05-27T09:05:22.439Z] - 0 - - "-" 28496 8374 14884 "-" "-" inbound|9090|| 127.0.0.6:36129 192.168.235.163:9090 192.168.235.177:43918 -
[2024-05-27T09:05:22.379Z] - 0 - - "-" 39817 11449 14947 "-" "-" inbound|9090|| 127.0.0.6:49263 192.168.235.163:9090 192.168.235.177:43894 -
[2024-05-27T09:05:22.439Z] - 0 - - "-" 22630 6775 14887 "-" "-" inbound|9090|| 127.0.0.6:55383 192.168.235.163:9090 192.168.235.177:44004 -
[2024-05-27T09:05:22.398Z] - 0 - - "-" 38762 11162 14938 "-" "-" inbound|9090|| 127.0.0.6:33723 192.168.235.163:9090 192.168.235.177:43904 -
[2024-05-27T09:05:22.439Z] - 0 - - "-" 51728 14688 14897 "-" "-" inbound|9090|| 127.0.0.6:45955 192.168.235.163:9090 192.168.235.177:43984 -

The expected traces from aeraki thrift example looks like

[2024-05-27T09:23:15.814Z] thrift 0  - "-" 0 0 1 "-" "125ed6e5-82d3-950d-933e-c26ae1cf3d47" inbound|9090|| - 192.168.235.162:9090 192.168.235.154:52654 -
[2024-05-27T09:23:20.817Z] thrift 0  - "-" 0 0 1 "-" "615528d2-2b7f-935d-b07e-cb9f1091786f" inbound|9090|| - 192.168.235.162:9090 192.168.235.154:52654 -
[2024-05-27T09:23:25.820Z] thrift 0  - "-" 0 0 1 "-" "ca05d554-13ba-9f5a-a1ba-6b9c3b729e29" inbound|9090|| - 192.168.235.162:9090 192.168.235.154:52654 -
[2024-05-27T09:23:30.823Z] thrift 0  - "-" 0 0 1 "-" "fd71cb81-500d-91f9-a9ab-9216bf0deb4d" inbound|9090|| - 192.168.235.162:9090 192.168.235.154:52654 -
[2024-05-27T09:23:35.826Z] thrift 0  - "-" 0 0 1 "-" "5144859b-c568-918f-9fca-093a46530974" inbound|9090|| - 192.168.235.162:9090 192.168.235.154:52654 -
[2024-05-27T09:23:40.830Z] thrift 0  - "-" 0 0 1 "-" "57f0ff96-64a9-9676-abe4-5fefd98bb708" inbound|9090|| - 192.168.235.162:9090 192.168.235.154:52654 -
[2024-05-27T09:23:45.833Z] thrift 0  - "-" 0 0 1 "-" "5897e401-b494-96b3-86cc-43720b810628" inbound|9090|| - 192.168.235.162:9090 192.168.235.154:52654 -
[2024-05-27T09:23:50.836Z] thrift 0  - "-" 0 0 1 "-" "95bd6724-6555-9d7d-b7bf-0d6901a5cd2d" inbound|9090|| - 192.168.235.162:9090 192.168.235.154:52654 -
[2024-05-27T09:23:55.839Z] thrift 0  - "-" 0 0 1 "-" "b4378221-b222-958f-91bc-affd09c32eef" inbound|9090|| - 192.168.235.162:9090 192.168.235.154:52654 -

Q1: Any suggestion to fix the problem? Or is there anybody using Aeraki built-in protocols to deploy their own applications?

Q2: Another point I don't understand: for mongodb service, I didn't change the port name to anything related to metaprotocol, but the 'istioctl proxy-status' shows the same output as the service like user-timeline-service I shown before.

yizhengx@node0:~/aeraki-benchmark/social-net/helm-chart/socialnetwork$ istioctl proxy-status user-timeline-mongodb-698869fb8f-x79fw.social-net
Clusters Match
--- Istiod Listeners
+++ Envoy Listeners
@@ -6166,73 +6166,73 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "accessLog": [
+                                                "access_log": [
                                                             {
                                                                         "name": "envoy.access_loggers.file",
-                                                                        "typedConfig": {
+                                                                        "typed_config": {
                                                                                     "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
-                                                                                    "logFormat": {
-                                                                                                "textFormatSource": {
-                                                                                                            "inlineString": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
+                                                                                    "log_format": {
+                                                                                                "text_format_source": {
+                                                                                                            "inline_string": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
                                                                                                         }
                                                                                             },
                                                                                     "path": "/dev/stdout"
                                                                                 }
                                                                     }
                                                         ],
-                                                "applicationProtocol": "thrift",
+                                                "application_protocol": "thrift",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.thrift"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "thrift-sample-server.meta-thrift.svc.cluster.local"
+                                                                                    "destination_service": "thrift-sample-server.meta-thrift.svc.cluster.local"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "thrift-sample-server.meta-thrift.svc.cluster.local_9090"
-                                                        },
-                                                "statPrefix": "outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local",
+                                                            "route_config_name": "thrift-sample-server.meta-thrift.svc.cluster.local_9090"
+                                                        },
+                                                "stat_prefix": "outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local",
                                                 "tracing": {
-                                                            "clientSampling": {
+                                                            "client_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "overallSampling": {
+                                                            "overall_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "randomSampling": {
+                                                            "random_sampling": {
                                                                         "value": 100
                                                                     }
                                                         }
                                             }
                                     }
                                 }
                             ]
@@ -10792,73 +10792,73 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "accessLog": [
+                                                "access_log": [
                                                             {
                                                                         "name": "envoy.access_loggers.file",
-                                                                        "typedConfig": {
+                                                                        "typed_config": {
                                                                                     "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
-                                                                                    "logFormat": {
-                                                                                                "textFormatSource": {
-                                                                                                            "inlineString": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
+                                                                                    "log_format": {
+                                                                                                "text_format_source": {
+                                                                                                            "inline_string": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
                                                                                                         }
                                                                                             },
                                                                                     "path": "/dev/stdout"
                                                                                 }
                                                                     }
                                                         ],
-                                                "applicationProtocol": "dubbo",
+                                                "application_protocol": "dubbo",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.dubbo"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "org.apache.dubbo.samples.basic.api.demoservice"
+                                                                                    "destination_service": "org.apache.dubbo.samples.basic.api.demoservice"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "org.apache.dubbo.samples.basic.api.demoservice_20880"
-                                                        },
-                                                "statPrefix": "outbound|20880||org.apache.dubbo.samples.basic.api.demoservice",
+                                                            "route_config_name": "org.apache.dubbo.samples.basic.api.demoservice_20880"
+                                                        },
+                                                "stat_prefix": "outbound|20880||org.apache.dubbo.samples.basic.api.demoservice",
                                                 "tracing": {
-                                                            "clientSampling": {
+                                                            "client_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "overallSampling": {
+                                                            "overall_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "randomSampling": {
+                                                            "random_sampling": {
                                                                         "value": 100
                                                                     }
                                                         }
                                             }
                                     }
                                 }
                             ]
@@ -10912,73 +10912,73 @@
                                 },
                                 {
                                     "name": "envoy.filters.network.meta_protocol_proxy",
                                     "typedConfig": {
                                         "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                                         "typeUrl": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
                                         "value": {
-                                                "accessLog": [
+                                                "access_log": [
                                                             {
                                                                         "name": "envoy.access_loggers.file",
-                                                                        "typedConfig": {
+                                                                        "typed_config": {
                                                                                     "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
-                                                                                    "logFormat": {
-                                                                                                "textFormatSource": {
-                                                                                                            "inlineString": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
+                                                                                    "log_format": {
+                                                                                                "text_format_source": {
+                                                                                                            "inline_string": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
                                                                                                         }
                                                                                             },
                                                                                     "path": "/dev/stdout"
                                                                                 }
                                                                     }
                                                         ],
-                                                "applicationProtocol": "dubbo",
+                                                "application_protocol": "dubbo",
                                                 "codec": {
                                                             "name": "aeraki.meta_protocol.codec.dubbo"
                                                         },
-                                                "metaProtocolFilters": [
+                                                "meta_protocol_filters": [
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.metadata_exchange"
                                                                     },
                                                             {
                                                                         "config": {
                                                                                     "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
-                                                                                    "destinationService": "org.apache.dubbo.samples.basic.api.secondservice"
+                                                                                    "destination_service": "org.apache.dubbo.samples.basic.api.secondservice"
                                                                                 },
                                                                         "name": "aeraki.meta_protocol.filters.istio_stats"
                                                                     },
                                                             {
                                                                         "name": "aeraki.meta_protocol.filters.router"
                                                                     }
                                                         ],
                                                 "rds": {
-                                                            "configSource": {
-                                                                        "apiConfigSource": {
-                                                                                    "apiType": "GRPC",
-                                                                                    "grpcServices": [
+                                                            "config_source": {
+                                                                        "api_config_source": {
+                                                                                    "api_type": "GRPC",
+                                                                                    "grpc_services": [
                                                                                                 {
-                                                                                                            "envoyGrpc": {
-                                                                                                                        "clusterName": "aeraki-xds"
+                                                                                                            "envoy_grpc": {
+                                                                                                                        "cluster_name": "aeraki-xds"
                                                                                                                     }
                                                                                                         }
                                                                                             ],
-                                                                                    "transportApiVersion": "V3"
+                                                                                    "transport_api_version": "V3"
                                                                                 },
-                                                                        "resourceApiVersion": "V3"
+                                                                        "resource_api_version": "V3"
                                                                     },
-                                                            "routeConfigName": "org.apache.dubbo.samples.basic.api.secondservice_20880"
-                                                        },
-                                                "statPrefix": "outbound|20880||org.apache.dubbo.samples.basic.api.secondservice",
+                                                            "route_config_name": "org.apache.dubbo.samples.basic.api.secondservice_20880"
+                                                        },
+                                                "stat_prefix": "outbound|20880||org.apache.dubbo.samples.basic.api.secondservice",
                                                 "tracing": {
-                                                            "clientSampling": {
+                                                            "client_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "overallSampling": {
+                                                            "overall_sampling": {
                                                                         "value": 100
                                                                     },
-                                                            "randomSampling": {
+                                                            "random_sampling": {
                                                                         "value": 100
                                                                     }
                                                         }
                                             }
                                     }
                                 }
                             ]

Routes Match (RDS last loaded at Mon, 27 May 2024 04:30:17 CDT)

The service definition for mongodb is the following:

Name:              user-timeline-mongodb
Namespace:         social-net
Labels:            app.kubernetes.io/managed-by=Helm
Annotations:       meta.helm.sh/release-name: social-net
                   meta.helm.sh/release-namespace: social-net
Selector:          service=user-timeline-mongodb
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.99.214.0
IPs:               10.99.214.0
Port:              27017  27017/TCP
TargetPort:        27017/TCP
Endpoints:         192.168.235.164:27017
Session Affinity:  None
Events:            <none>

Thanks! Looking forward to your support and help.
@zhaohuabing
Copy link
Member

@yizhengx I'm not certain if istioctl can recognize metaprotocol properly or not. Can you directly connect to the Envoy and dump its config through its admin port?

Besides, does the Aeraki Thrift demo work in your environment?

@yizhengx
Copy link
Author

@zhaohuabing Thanks for your reply!

does the Aeraki Thrift demo work in your environment?

Yes, that works well.

Can you directly connect to the Envoy and dump its config through its admin port?

Here's the config_dump/listeners for user-timeline-service in social-network benchmark
https://logpaste.com/pEfD8Qc4
It seems that it doesn't configure metaprotocol filter for inbound traffics to port 9090.

@yizhengx
Copy link
Author

yizhengx commented Jun 3, 2024 •
edited
Loading

Update: I can manually replace the tcp network filter with aeraki metaprotocol proxy for inbound traffic using the following envoy filter configuration:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: fix-metaprotocol-filter
  namespace: social-net # as defined in meshConfig resource.
spec:
  workloadSelector:
    labels:
      app: user-timeline-service
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_INBOUND # will match inbound listeners in all sidecars
      listener:
        portNumber: 9090
        filterChain:
          filter:
            name: "envoy.filters.network.tcp_proxy"
    patch:
      operation: REPLACE
      value:
        name: "envoy.filters.network.meta_protocol_proxy"
        typed_config: {
          "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
          "type_url": "type.googleapis.com/aeraki.meta_protocol_proxy.v1alpha.MetaProtocolProxy",
          "value": {
            "stat_prefix": "inbound|9090||",
            "application_protocol": "thrift",
            "route_config": {
            "name": "inbound|9090||",
            "routes": [
              {
              "route": {
                "cluster": "inbound|9090||"
              }
              }
            ]
            },
            "codec": {
              "name": "aeraki.meta_protocol.codec.thrift"
            },
            "meta_protocol_filters": [
            {
              "name": "aeraki.meta_protocol.filters.metadata_exchange"
            },
            {
              "name": "aeraki.meta_protocol.filters.istio_stats",
              "config": {
              "@type": "type.googleapis.com/aeraki.meta_protocol_proxy.filters.istio_stats.v1alpha.IstioStats",
              "destination_service": "user-timeline-service.social-net.svc.cluster.local"
              }
            },
            {
              "name": "aeraki.meta_protocol.filters.router"
            }
            ],
            "tracing": {
            "client_sampling": {
              "value": 100
            },
            "random_sampling": {
              "value": 100
            },
            "overall_sampling": {
              "value": 100
            }
            },
            "access_log": [
            {
              "name": "envoy.access_loggers.file",
              "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
              "path": "/dev/stdout",
              "log_format": {
                "text_format_source": {
                "inline_string": "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)% %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"
                }
              }
              }
            }
            ]
          }
        }

This serves as a workaround to recognize other layer-7 protocols, but it would be great for me to understand if I configure things wrong or any other reason that causes the root problems. Thanks for your help!

@zhaohuabing
Copy link
Member

@yizhengx Aeraki would automatically create an EnvoyFilter for inbound listeners. You can check if that EnvoyFilter has been created or not. If not, you can further check the Aerkai log to look for the root cause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zhaohuabing @yizhengx