Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refresh tries to connect to old NSE name and fails indefinitely #1357

Closed
yuraxdrumz opened this issue Sep 22, 2022 · 24 comments
Closed

Refresh tries to connect to old NSE name and fails indefinitely #1357

yuraxdrumz opened this issue Sep 22, 2022 · 24 comments
Assignees
@glazychev-art @NikitaSkrynnik
Labels
ASAP The issue that blocking SOW items or core use-cases bug Something isn't working
Milestone
v1.6.0

Comments

@yuraxdrumz
Copy link

Expected Behavior

NSC should be able to connect to an NSE according to the NS definition after NSE reschedule on k8s

Current Behavior

I have 3 NSC's running that try to connect to the old NSE even though it does not exist anymore after a k8s node running the NSE got terminated and the NSE got rescheduled to a different node.

Failure Information (for bugs)

The forwarder that tries to connect to the nse always fails with network service candidate for nse name XXX was not found.

It looks like when the node went down, healing should have worked when the nse got rescheduled on another node.

Looking at the discoverCandidatesServer code, I see that if the nse name already exists, we try to find it and continue, but in this case it does not exist anymore.

func (d *discoverCandidatesServer) Request(ctx context.Context, request *networkservice.NetworkServiceRequest) (*networkservice.Connection, error) {
	if clienturlctx.ClientURL(ctx) != nil {
		return next.Server(ctx).Request(ctx, request)
	}

	nseName := request.GetConnection().GetNetworkServiceEndpointName()
	if nseName != "" {
		nse, err := d.discoverNetworkServiceEndpoint(ctx, nseName)
		if err != nil {
			return nil, err
		}
		u, err := url.Parse(nse.Url)
		if err != nil {
			return nil, errors.WithStack(err)
		}
		return next.Server(ctx).Request(clienturlctx.WithClientURL(ctx, u), request)
	}

	ns, err := d.discoverNetworkService(ctx, request.GetConnection().GetNetworkService(), request.GetConnection().GetPayload())
	if err != nil {
		return nil, err
	}
	nses, err := d.discoverNetworkServiceEndpoints(ctx, ns, request.GetConnection().GetLabels())
	if err != nil {
		return nil, err
	}

	request.GetConnection().Payload = ns.Payload

	return next.Server(ctx).Request(WithCandidates(ctx, nses, ns), request.Clone())
}

Steps to Reproduce

  1. Create NSC
  2. Create NSE
  3. Restart NSE several times until problem is reproduced

Context

  • K8s v1.22 on EKS
  • forwarder vpp

Failure Logs
@denis-tingaikin denis-tingaikin added the bug Something isn't working label Sep 22, 2022
@denis-tingaikin denis-tingaikin added this to the v1.6.0 milestone Sep 22, 2022
@denis-tingaikin
Copy link
Member

denis-tingaikin commented Sep 22, 2022 •
edited
Loading

@yuraxdrumz Thanks for the report!

We'll have look.

@edwarnicke
Copy link
Member

@denis-tingaikin Any idea why healing isn't simply getting a DOWN event and trying for a different NSE instance?

@denis-tingaikin
Copy link
Member

My guesses are:

  1. reselect doesn't work properly
  2. heal client uses a custom healFunc and always thinks that dataPath is fine
  3. Uses custom client
  4. data path checker returns false positive response

To reduce the area we need to get more information

@yuraxdrumz

Questions:

  1. Do you have any logs?
  2. Could you describe NSC deployment?

@denis-tingaikin denis-tingaikin added the ASAP The issue that blocking SOW items or core use-cases label Sep 24, 2022
@glazychev-art
Copy link
Contributor

@yuraxdrumz
We tried to reproduce this issue several times - so far everything works without problems.

We would be very grateful for additional information:

  1. What version of NSM are you using?
  2. Do you use interdomain scenario?

@glazychev-art glazychev-art mentioned this issue Sep 26, 2022
Open
@yuraxdrumz
Copy link
Author

Hey,

Sorry for the late reply.

The issue attached regarding the healing is also an issue I am experiencing while using multiple NSC's.

Regarding the current one, it looks like I ran a kustomization.yaml with a slightly different NS service name, which caused new NSE's to be registered under the different NS name, while the NSC kept trying to connect to old NSE.

I will create a new cluster and try everything again just to be sure.

I am using NSM 1.5.0 without interdomain scenario.

If the problem persists, I will add logs here.

Thanks

@or-adar
Copy link

or-adar commented Sep 28, 2022 •
edited
Loading

Hi,
For this issue (having NSCs connect to "dead" NSEs), I could only reproduce for complex scenarios.

For the specified scenario of having only 1 broken component, like NSE, for instance,
it seems like the NSC does manage to connect to the correct NSE after a new Pod for the "broken" NSE gets rescheduled
(as long as there's 1 NSC, for multiple NSCs I experience the same issues described in #1358)

However, when having multiple broken components, my NSC attempts to request an already deleted Pod and it does not reference the new one.
I managed to reproduce this when deleting my NSMgr & my NSE pods (didn't have to delete all of them, only the ones my NSC referenced).
After that, my NSC emitted errors with the messgae network service endpoint <nse-pod-name> not found: all forwarders have failed.
where <nse-pod-name> is the previous pod of the NSE that got deleted, when it should have been the new scheduled pod.

If you need logs, let me know.

@glazychev-art
Copy link
Contributor

glazychev-art commented Sep 29, 2022 •
edited
Loading

Thanks guys!

@or-adar
Yes, it would be very helpful if you could attach the logs
Could you also clarify your NSM version?

@glazychev-art
Copy link
Contributor

So the current status:

@yuraxdrumz
Based on your description, it looks like you have a setup problem..
I didn't find the error you provided in the code (network service candidate...), but the most similar is network service endpoint candidates not found - https://github.com/networkservicemesh/sdk/blob/main/pkg/networkservice/common/discover/server.go#L152
So if you:

  1. NSC (requires ServiceA) ---------> NSE-1 (provides ServiceA)
  2. Replace ServiceA with ServiceB for the NSE-1 (if I get you right Refresh tries to connect to old NSE name and fails indefinitely #1357 (comment))
  3. NSE-1 is recreated and becomes NSE-2 (provides ServiceB)
  4. Healing closes current connection
  5. NSC (requires ServiceA) ---xxx--- NSE-1 (provides ServiceB).
    Because you also need to change the required service from the NSC.

In any case, it will become more clear only with the logs :)

@glazychev-art
Copy link
Contributor

@or-adar

Your error is slightly different, but I think I was able to reproduce it under load.

The reason is that under load, refreshes can return an error (for example, the grpc Dial time increases).
In this case, no healing is started on the way back (because of connection error), so no one monitors the connection.

The next refresh will happen later (usually in a couple of minutes). And if during these 2 minutes the NSE dies, the client will not know about it, and will give an error on requests: network service endpoint <nse-pod-name> not found.

@glazychev-art
Copy link
Contributor

@edwarnicke

So we have several possible solutions:

  1. Add instant retries to the refresh in case of a fail - wrap this in a loop.
    But we need to add a threshold after which we will start making requests with reselect (removes NetworkServiceEndpointName from the request)

  2. Add retry chain element after begin (it currently wraps the entire client)
    But in case of an error during the refresh, the call stack will be complicated in my opinion:

  • Start refresh: refresh (goroutine) -> eventfactory -> retry -> refresh -> ...
  • Request returns an error: retry handles it on the way back, and this time the request is sent from the retry chain element (and this is possible several times)
    We will also need to add a threshold
  1. Don't close previous eventLoop before request in healing.
    Perhaps we need to implement some kind of pause before Request call.
    If the request returns an error, we resume the paused eventLoop, else - close it and start a new one.
    In this case, if something happens to DataPlane, we will know about it (even if the refreshes will fall).

Thoughts?

@or-adar
Copy link

or-adar commented Oct 2, 2022

@glazychev-art

Oct  2 09:52:57.014 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (2)  ⎆ sdk/pkg/networkservice/common/refresh/refreshClient.Request()
Oct  2 09:52:57.015 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (3)   ⎆ sdk/pkg/networkservice/common/clienturl/clientURLClient.Request()
Oct  2 09:52:57.015 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (3.1)     passed clientURL: unix:///var/lib/networkservicemesh/nsm.io.sock
Oct  2 09:52:57.015 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (4)    ⎆ sdk/pkg/networkservice/common/clientconn/clientConnClient.Request()
Oct  2 09:52:57.015 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (5)     ⎆ sdk/pkg/networkservice/common/heal/healClient.Request()
Oct  2 09:52:57.015 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [heal:eventLoop] [type:networkService] (5.1)       received eventIn: <nil>, err: rpc error: code = Canceled desc = context canceled
Oct  2 09:52:57.015 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (6)      ⎆ sdk/pkg/networkservice/common/dial/dialClient.Request()
Oct  2 09:52:57.016 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7)       ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
Oct  2 09:52:57.016 [WARN] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.1)         The label podName was already assigned to eu-west-1-4cc2a6ad8cdc4f-0. Skipping.
Oct  2 09:52:57.016 [WARN] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.2)         Environment variable CLUSTER_NAME is not set. Skipping.
Oct  2 09:52:57.016 [WARN] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.3)         Environment variable NODE_NAME is not set. Skipping.
Oct  2 09:52:57.016 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (8)        ⎆ sdk/pkg/networkservice/common/upstreamrefresh/upstreamRefreshClient.Request()
Oct  2 09:52:57.016 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (9)         ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
Oct  2 09:52:57.018 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
Oct  2 09:52:57.018 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (11)           ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
Oct  2 09:52:57.018 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (12)            ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
Oct  2 09:52:57.019 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (12.1)              request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"file:///proc/thread-self/ns/net"}}}}
Oct  2 09:52:57.019 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (13)             ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
Oct  2 09:52:57.019 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (13.1)               request-diff={"connection":{"mechanism":{"parameters":{"inodeURL":"inode://4/4026532932"}}},"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026532932"}}}}
Oct  2 09:52:57.019 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (14)              ⎆ sdk/pkg/networkservice/connectioncontext/dnscontext/dnsContextClient.Request()
Oct  2 09:52:57.020 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (15)               ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
Oct  2 09:52:57.020 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [ExcludedPrefixesClient:Request] [type:networkService] (15.1)
      Adding new excluded IPs to the request: [10.12.1.1/32 10.12.1.0/32 172.20.0.1/32 172.20.0.10/32 172.20.2.59/32 172.20.13.247/32 172.20.32.62/32 172.20.32.125/32 172.20.34.216/32 172.20.44.123/32 172.20.48.63/32 172.20.66.157/32 172.20.86.34/32 172.20.88.185/32 172.20.90.5/32 172.20.98.91/32 172.20.157.76/32 172.20.180.165/32 172.20.192.105/32 172.20.194.58/32 172.20.194.82/32 172.20.205.246/32 172.20.210.78/32 172.20.242.74/32 172.20.244.146/32 172.20.246.26/32 172.20.76.247/32]
Oct  2 09:52:57.020 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [ExcludedPrefixesClient:Request] [type:networkService] (15.2)
      Excluded prefixes from request - [172.20.0.1/32 172.20.0.10/32 172.20.2.59/32 172.20.13.247/32 172.20.32.62/32 172.20.32.125/32 172.20.34.216/32 172.20.44.123/32 172.20.48.63/32 172.20.66.157/32 172.20.86.34/32 172.20.88.185/32 172.20.90.5/32 172.20.98.91/32 172.20.157.76/32 172.20.180.165/32 172.20.192.105/32 172.20.194.58/32 172.20.194.82/32 172.20.205.246/32 172.20.210.78/32 172.20.242.74/32 172.20.244.146/32 172.20.246.26/32 172.20.76.247/32]
Oct  2 09:52:57.020 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (16)                ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
Oct  2 09:52:57.020 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (17)                 ⎆ sdk/pkg/networkservice/common/trimpath/trimpathClient.Request()
Oct  2 09:52:57.020 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (18)                  ⎆ sdk/pkg/networkservice/common/connect/connectClient.Request()
Oct  2 09:52:57.211 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (18.1)                    request-response={"id":"eu-west-1-4cc2a6ad8cdc4f-0-0","network_service":"nsm-inet","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026532932","name":"nsm-2"}},"context":{"ip_context":{"src_ip_addrs":["10.12.1.1/32"],"dst_ip_addrs":["10.12.1.0/32"],"src_routes":[{"prefix":"10.12.1.0/32"}],"dst_routes":[{"prefix":"10.12.1.1/32"}],"excluded_prefixes":["172.20.0.1/32","172.20.0.10/32","172.20.2.59/32","172.20.13.247/32","172.20.32.62/32","172.20.32.125/32","172.20.34.216/32","172.20.44.123/32","172.20.48.63/32","172.20.66.157/32","172.20.86.34/32","172.20.88.185/32","172.20.90.5/32","172.20.98.91/32","172.20.157.76/32","172.20.180.165/32","172.20.192.105/32","172.20.194.58/32","172.20.194.82/32","172.20.205.246/32","172.20.210.78/32","172.20.242.74/32","172.20.244.146/32","172.20.246.26/32","172.20.76.247/32"],"policies":[{"proto":"6"},{"proto":"1"},{"proto":"17"}]},"dns_context":{"configs":[{"dns_server_ips":["172.20.0.10"],"search_domains":["4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local","svc.cluster.local","cluster.local","eu-west-1.compute.internal"]},{"dns_server_ips":["172.20.0.10"],"search_domains":["4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local","svc.cluster.local","cluster.local","eu-west-1.compute.internal"]},{"dns_server_ips":["172.20.0.10"],"search_domains":["4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local","svc.cluster.local","cluster.local","eu-west-1.compute.internal"]}]},"MTU":8947},"labels":{"nodeName":"ip-10-111-100-207.eu-west-1.compute.internal","podName":"eu-west-1-4cc2a6ad8cdc4f-0","subnet":"100.124.0.0/24"},"path":{"path_segments":[{"name":"eu-west-1-4cc2a6ad8cdc4f-0","id":"eu-west-1-4cc2a6ad8cdc4f-0-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy80Y2MyYTZhZC04Y2RjLTRmMmQtODZkMy1jMDJiNzMxOTBmMGQvcG9kL2V1LXdlc3QtMS00Y2MyYTZhZDhjZGM0Zi0wLzkzM2ExZjczLTYwMDQtNGVmNi1iMzc1LWRjMGRjY2Q3MTAyNCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci04aHpreC8xOTRmYzFkOS05YTBhLTRmZGYtOWRiYS1mZTI0MjI5NGQ4NGMiXSwiZXhwIjoxNjY0NzA0OTc3fQ.Czx66_sNuJ6ucI_u12a07kG0-Me97ozlEwHazS-HMuhrsNLGwZsMcdBaNdIaafqb_E0D7yi8AI758QeQEEEHDw","expires":{"seconds":1664704977,"nanos":20862120}},{"name":"nsmgr-8hzkx","id":"82f870d8-ad8f-40bb-93b3-820348581331","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci04aHpreC8xOTRmYzFkOS05YTBhLTRmZGYtOWRiYS1mZTI0MjI5NGQ4NGMiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC12OHpybC9mNjUzNDZkZi1lNGJlLTQ1MDgtOWUwYy0zM2M4YzJlNzkzYjgiXSwiZXhwIjoxNjY0NzA0OTc3fQ.pCAczepbvdRPVRMc2qT--deGsjUexCJImRpQEHRhkV06sZcg7MpS21dfjH1CEoAFbwrRoL4uj7Z7QfEGBMmyJw","expires":{"seconds":1664704977,"nanos":42356498}},{"name":"forwarder-vpp-v8zrl","id":"2e946c2d-d6cc-48fa-aaea-7eeb2461b659","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLXY4enJsL2Y2NTM0NmRmLWU0YmUtNDUwOC05ZTBjLTMzYzhjMmU3OTNiOCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci00OTZqNi8xM2Y5NWYxNy1kYTVhLTQ3ZmEtOTc2NC1jYmQwNTAzZTBlNDIiXSwiZXhwIjoxNjY0NzA0OTc3fQ.tgdabUpzntDud3JjGzK0hGKxbzNJcVpqy9cdbwERvieREyUBroF8IEh9LzLS_NB_lPcJczfW-cmI8Og9tiWNig","expires":{"seconds":1664704977,"nanos":77344807},"metrics":{"client_drops":"0","client_rx_bytes":"3754836","client_rx_packets":"67170","client_tx_bytes":"6185420","client_tx_packets":"67288","server_drops":"0","server_rx_bytes":"3763052","server_rx_packets":"67288","server_tx_bytes":"3754836","server_tx_packets":"67170"}},{"name":"nsmgr-496j6","id":"8886fac2-cca2-4312-8dfc-2d33ff3c0118","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci00OTZqNi8xM2Y5NWYxNy1kYTVhLTQ3ZmEtOTc2NC1jYmQwNTAzZTBlNDIiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC02a3R4cy8zODhmOTIyMy04YjNhLTQzMzMtOGQ4Ny1mZDFjYzEwOTdhODIiXSwiZXhwIjoxNjY0NzA0OTc3fQ.hmK_cVgkKDVX5ZGoO3eNX58AHrtivnBBz243fLc9d2DjOEH147Si2uw5dEMO2Oe8gX7cNIy55821LpQFBK-rlg","expires":{"seconds":1664704977,"nanos":99708387}},{"name":"forwarder-vpp-6ktxs","id":"d4666cff-a00b-472d-b2fb-19ff76c3aa38","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLTZrdHhzLzM4OGY5MjIzLThiM2EtNDMzMy04ZDg3LWZkMWNjMTA5N2E4MiIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9pbmV0L3BvZC9uc2UtaW5ldC01NWY0NDRiNTk4LWd3NW12LzFjZTQ3NGQ2LWIwNWUtNDRkMC1iMjFhLTc5M2M0MmM2OGQ3OCJdLCJleHAiOjE2NjQ3MDQ5Nzd9.9xbUFfH5wjoQustLx9iajvKhhaPCiRXJdcRWveYmHE1z9gvXvBJriVCAlso0S2dwCy3AhiLd6m-Iy6cFfcQRKg","expires":{"seconds":1664704977,"nanos":142908776},"metrics":{"client_drops":"0","client_rx_bytes":"3754836","client_rx_packets":"67170","client_tx_bytes":"3762562","client_tx_packets":"67279","server_drops":"0","server_rx_bytes":"3762562","server_rx_packets":"67279","server_tx_bytes":"6172956","server_tx_packets":"67170"}},{"name":"nse-inet-55f444b598-gw5mv","id":"1e670852-b8f7-48a8-8997-f80ac23d44cb","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9pbmV0L3BvZC9uc2UtaW5ldC01NWY0NDRiNTk4LWd3NW12LzFjZTQ3NGQ2LWIwNWUtNDRkMC1iMjFhLTc5M2M0MmM2OGQ3OCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLTZrdHhzLzM4OGY5MjIzLThiM2EtNDMzMy04ZDg3LWZkMWNjMTA5N2E4MiJdLCJleHAiOjE2NjQ3MDQ5Nzd9.KhEsrTvsg-CUOrekXPzLlZctXj448W-RvqVp-LOP2NFN1I4xoSv7uFVHZ7De_5XaCFhrEy5FIjRgM8fuqgOEmQ","expires":{"seconds":1664704977,"nanos":144532796}}]},"network_service_endpoint_name":"nse-inet-55f444b598-gw5mv","payload":"ETHERNET"}
Oct  2 09:52:57.214 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [ExcludedPrefixesClient:Request] [type:networkService] (15.3)
      Request excluded IPs - srcIPs: [10.12.1.1/32], dstIPs: [10.12.1.0/32], excluded prefixes: [172.20.0.1/32 172.20.0.10/32 172.20.2.59/32 172.20.13.247/32 172.20.32.62/32 172.20.32.125/32 172.20.34.216/32 172.20.44.123/32 172.20.48.63/32 172.20.66.157/32 172.20.86.34/32 172.20.88.185/32 172.20.90.5/32 172.20.98.91/32 172.20.157.76/32 172.20.180.165/32 172.20.192.105/32 172.20.194.58/32 172.20.194.82/32 172.20.205.246/32 172.20.210.78/32 172.20.242.74/32 172.20.244.146/32 172.20.246.26/32 172.20.76.247/32]
Oct  2 09:52:57.214 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [ExcludedPrefixesClient:Request] [type:networkService] (15.4)
      Added excluded prefixes: [10.12.1.1/32 10.12.1.0/32 172.20.0.1/32 172.20.0.10/32 172.20.2.59/32 172.20.13.247/32 172.20.32.62/32 172.20.32.125/32 172.20.34.216/32 172.20.44.123/32 172.20.48.63/32 172.20.66.157/32 172.20.86.34/32 172.20.88.185/32 172.20.90.5/32 172.20.98.91/32 172.20.157.76/32 172.20.180.165/32 172.20.192.105/32 172.20.194.58/32 172.20.194.82/32 172.20.205.246/32 172.20.210.78/32 172.20.242.74/32 172.20.244.146/32 172.20.246.26/32 172.20.76.247/32]
Oct  2 09:52:57.215 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (13.2)               request-response-diff={"mechanism":{"parameters":{"inodeURL":"file:///proc/thread-self/ns/net"}}}
Oct  2 09:52:57.218 [INFO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (2.1)    expiration after 9m59.802749163s at 2022-10-02 10:02:57.02086212 +0000 UTC
Oct  2 09:54:32.629 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [heal:eventLoop] [type:networkService] (5.1)       received eventIn: type:UPDATE connections:{key:"82f870d8-ad8f-40bb-93b3-820348581331" value:{id:"82f870d8-ad8f-40bb-93b3-820348581331" network_service:"nsm-inet" mechanism:{cls:"LOCAL" type:"KERNEL" parameters:{key:"inodeURL" value:"inode://4/4026532932"} parameters:{key:"name" value:"nsm-2"}} context:{ip_context:{src_ip_addrs:"10.12.1.1/32" dst_ip_addrs:"10.12.1.0/32" src_routes:{prefix:"10.12.1.0/32"} dst_routes:{prefix:"10.12.1.1/32"} excluded_prefixes:"172.20.0.1/32" excluded_prefixes:"172.20.0.10/32" excluded_prefixes:"172.20.2.59/32" excluded_prefixes:"172.20.13.247/32" excluded_prefixes:"172.20.32.62/32" excluded_prefixes:"172.20.32.125/32" excluded_prefixes:"172.20.34.216/32" excluded_prefixes:"172.20.44.123/32" excluded_prefixes:"172.20.48.63/32" excluded_prefixes:"172.20.66.157/32" excluded_prefixes:"172.20.86.34/32" excluded_prefixes:"172.20.88.185/32" excluded_prefixes:"172.20.90.5/32" excluded_prefixes:"172.20.98.91/32" excluded_prefixes:"172.20.157.76/32" excluded_prefixes:"172.20.180.165/32" excluded_prefixes:"172.20.192.105/32" excluded_prefixes:"172.20.194.58/32" excluded_prefixes:"172.20.194.82/32" excluded_prefixes:"172.20.205.246/32" excluded_prefixes:"172.20.210.78/32" excluded_prefixes:"172.20.242.74/32" excluded_prefixes:"172.20.244.146/32" excluded_prefixes:"172.20.246.26/32" excluded_prefixes:"172.20.76.247/32" policies:{proto:"6"} policies:{proto:"1"} policies:{proto:"17"}} dns_context:{configs:{dns_server_ips:"172.20.0.10" search_domains:"4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local" search_domains:"svc.cluster.local" search_domains:"cluster.local" search_domains:"eu-west-1.compute.internal"} configs:{dns_server_ips:"172.20.0.10" search_domains:"4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local" search_domains:"svc.cluster.local" search_domains:"cluster.local" search_domains:"eu-west-1.compute.internal"} configs:{dns_server_ips:"172.20.0.10" search_domains:"4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local" search_domains:"svc.cluster.local" search_domains:"cluster.local" search_domains:"eu-west-1.compute.internal"}} MTU:8947} labels:{key:"nodeName" value:"ip-10-111-100-207.eu-west-1.compute.internal"} labels:{key:"podName" value:"eu-west-1-4cc2a6ad8cdc4f-0"} labels:{key:"subnet" value:"100.124.0.0/24"} path:{index:1 path_segments:{name:"eu-west-1-4cc2a6ad8cdc4f-0" id:"eu-west-1-4cc2a6ad8cdc4f-0-0" token:"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy80Y2MyYTZhZC04Y2RjLTRmMmQtODZkMy1jMDJiNzMxOTBmMGQvcG9kL2V1LXdlc3QtMS00Y2MyYTZhZDhjZGM0Zi0wLzkzM2ExZjczLTYwMDQtNGVmNi1iMzc1LWRjMGRjY2Q3MTAyNCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci04aHpreC8xOTRmYzFkOS05YTBhLTRmZGYtOWRiYS1mZTI0MjI5NGQ4NGMiXSwiZXhwIjoxNjY0NzA0OTc3fQ.Czx66_sNuJ6ucI_u12a07kG0-Me97ozlEwHazS-HMuhrsNLGwZsMcdBaNdIaafqb_E0D7yi8AI758QeQEEEHDw" expires:{seconds:1664704977 nanos:20862120}} path_segments:{name:"nsmgr-8hzkx" id:"82f870d8-ad8f-40bb-93b3-820348581331" token:"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci04aHpreC8xOTRmYzFkOS05YTBhLTRmZGYtOWRiYS1mZTI0MjI5NGQ4NGMiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC12OHpybC9mNjUzNDZkZi1lNGJlLTQ1MDgtOWUwYy0zM2M4YzJlNzkzYjgiXSwiZXhwIjoxNjY0NzA0OTc3fQ.pCAczepbvdRPVRMc2qT--deGsjUexCJImRpQEHRhkV06sZcg7MpS21dfjH1CEoAFbwrRoL4uj7Z7QfEGBMmyJw" expires:{seconds:1664704977 nanos:42356498}} path_segments:{name:"forwarder-vpp-v8zrl" id:"2e946c2d-d6cc-48fa-aaea-7eeb2461b659" token:"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLXY4enJsL2Y2NTM0NmRmLWU0YmUtNDUwOC05ZTBjLTMzYzhjMmU3OTNiOCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci00OTZqNi8xM2Y5NWYxNy1kYTVhLTQ3ZmEtOTc2NC1jYmQwNTAzZTBlNDIiXSwiZXhwIjoxNjY0NzA0OTc3fQ.tgdabUpzntDud3JjGzK0hGKxbzNJcVpqy9cdbwERvieREyUBroF8IEh9LzLS_NB_lPcJczfW-cmI8Og9tiWNig" expires:{seconds:1664704977 nanos:77344807} metrics:{key:"client_drops" value:"0"} metrics:{key:"client_rx_bytes" value:"3753782"} metrics:{key:"client_rx_packets" value:"67151"} metrics:{key:"client_tx_bytes" value:"6183682"} metrics:{key:"client_tx_packets" value:"67269"} metrics:{key:"server_drops" value:"0"} metrics:{key:"server_rx_bytes" value:"3761998"} metrics:{key:"server_rx_packets" value:"67269"} metrics:{key:"server_tx_bytes" value:"3753782"} metrics:{key:"server_tx_packets" value:"67151"}} path_segments:{name:"nsmgr-496j6" id:"8886fac2-cca2-4312-8dfc-2d33ff3c0118" token:"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci00OTZqNi8xM2Y5NWYxNy1kYTVhLTQ3ZmEtOTc2NC1jYmQwNTAzZTBlNDIiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC02a3R4cy8zODhmOTIyMy04YjNhLTQzMzMtOGQ4Ny1mZDFjYzEwOTdhODIiXSwiZXhwIjoxNjY0NzA0OTc3fQ.hmK_cVgkKDVX5ZGoO3eNX58AHrtivnBBz243fLc9d2DjOEH147Si2uw5dEMO2Oe8gX7cNIy55821LpQFBK-rlg" expires:{seconds:1664704977 nanos:99708387}} path_segments:{name:"forwarder-vpp-6ktxs" id:"d4666cff-a00b-472d-b2fb-19ff76c3aa38" token:"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLTZrdHhzLzM4OGY5MjIzLThiM2EtNDMzMy04ZDg3LWZkMWNjMTA5N2E4MiIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9pbmV0L3BvZC9uc2UtaW5ldC01NWY0NDRiNTk4LWd3NW12LzFjZTQ3NGQ2LWIwNWUtNDRkMC1iMjFhLTc5M2M0MmM2OGQ3OCJdLCJleHAiOjE2NjQ3MDQ5Nzd9.9xbUFfH5wjoQustLx9iajvKhhaPCiRXJdcRWveYmHE1z9gvXvBJriVCAlso0S2dwCy3AhiLd6m-Iy6cFfcQRKg" expires:{seconds:1664704977 nanos:142908776} metrics:{key:"client_drops" value:"0"} metrics:{key:"client_rx_bytes" value:"3754836"} metrics:{key:"client_rx_packets" value:"67170"} metrics:{key:"client_tx_bytes" value:"3762562"} metrics:{key:"client_tx_packets" value:"67279"} metrics:{key:"server_drops" value:"0"} metrics:{key:"server_rx_bytes" value:"3762562"} metrics:{key:"server_rx_packets" value:"67279"} metrics:{key:"server_tx_bytes" value:"6172956"} metrics:{key:"server_tx_packets" value:"67170"}} path_segments:{name:"nse-inet-55f444b598-gw5mv" id:"1e670852-b8f7-48a8-8997-f80ac23d44cb" token:"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9pbmV0L3BvZC9uc2UtaW5ldC01NWY0NDRiNTk4LWd3NW12LzFjZTQ3NGQ2LWIwNWUtNDRkMC1iMjFhLTc5M2M0MmM2OGQ3OCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLTZrdHhzLzM4OGY5MjIzLThiM2EtNDMzMy04ZDg3LWZkMWNjMTA5N2E4MiJdLCJleHAiOjE2NjQ3MDQ5Nzd9.KhEsrTvsg-CUOrekXPzLlZctXj448W-RvqVp-LOP2NFN1I4xoSv7uFVHZ7De_5XaCFhrEy5FIjRgM8fuqgOEmQ" expires:{seconds:1664704977 nanos:144532796}}} network_service_endpoint_name:"nse-inet-55f444b598-gw5mv" payload:"ETHERNET" state:DOWN}}, err: <nil>
Oct  2 09:54:32.629 [WARN] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [heal:eventLoop] [type:networkService] (5.2)       Control plane is downOct  2 09:54:32.629 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
Oct  2 09:54:32.630 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (1.1)   request={"connection":{"id":"eu-west-1-4cc2a6ad8cdc4f-0-0","network_service":"nsm-inet","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"file:///proc/thread-self/ns/net","name":"nsm-2"}},"context":{"ip_context":{"src_ip_addrs":["10.12.1.1/32"],"dst_ip_addrs":["10.12.1.0/32"],"src_routes":[{"prefix":"10.12.1.0/32"}],"dst_routes":[{"prefix":"10.12.1.1/32"}],"excluded_prefixes":["172.20.0.1/32","172.20.0.10/32","172.20.2.59/32","172.20.13.247/32","172.20.32.62/32","172.20.32.125/32","172.20.34.216/32","172.20.44.123/32","172.20.48.63/32","172.20.66.157/32","172.20.86.34/32","172.20.88.185/32","172.20.90.5/32","172.20.98.91/32","172.20.157.76/32","172.20.180.165/32","172.20.192.105/32","172.20.194.58/32","172.20.194.82/32","172.20.205.246/32","172.20.210.78/32","172.20.242.74/32","172.20.244.146/32","172.20.246.26/32","172.20.76.247/32"],"policies":[{"proto":"6"},{"proto":"1"},{"proto":"17"}]},"dns_context":{"configs":[{"dns_server_ips":["172.20.0.10"],"search_domains":["4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local","svc.cluster.local","cluster.local","eu-west-1.compute.internal"]},{"dns_server_ips":["172.20.0.10"],"search_domains":["4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local","svc.cluster.local","cluster.local","eu-west-1.compute.internal"]},{"dns_server_ips":["172.20.0.10"],"search_domains":["4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local","svc.cluster.local","cluster.local","eu-west-1.compute.internal"]}]},"MTU":8947},"labels":{"nodeName":"ip-10-111-100-207.eu-west-1.compute.internal","podName":"eu-west-1-4cc2a6ad8cdc4f-0","subnet":"100.124.0.0/24"},"path":{"path_segments":[{"name":"eu-west-1-4cc2a6ad8cdc4f-0","id":"eu-west-1-4cc2a6ad8cdc4f-0-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy80Y2MyYTZhZC04Y2RjLTRmMmQtODZkMy1jMDJiNzMxOTBmMGQvcG9kL2V1LXdlc3QtMS00Y2MyYTZhZDhjZGM0Zi0wLzkzM2ExZjczLTYwMDQtNGVmNi1iMzc1LWRjMGRjY2Q3MTAyNCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci04aHpreC8xOTRmYzFkOS05YTBhLTRmZGYtOWRiYS1mZTI0MjI5NGQ4NGMiXSwiZXhwIjoxNjY0NzA0OTc3fQ.Czx66_sNuJ6ucI_u12a07kG0-Me97ozlEwHazS-HMuhrsNLGwZsMcdBaNdIaafqb_E0D7yi8AI758QeQEEEHDw","expires":{"seconds":1664704977,"nanos":20862120}},{"name":"nsmgr-8hzkx","id":"82f870d8-ad8f-40bb-93b3-820348581331","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci04aHpreC8xOTRmYzFkOS05YTBhLTRmZGYtOWRiYS1mZTI0MjI5NGQ4NGMiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC12OHpybC9mNjUzNDZkZi1lNGJlLTQ1MDgtOWUwYy0zM2M4YzJlNzkzYjgiXSwiZXhwIjoxNjY0NzA0OTc3fQ.pCAczepbvdRPVRMc2qT--deGsjUexCJImRpQEHRhkV06sZcg7MpS21dfjH1CEoAFbwrRoL4uj7Z7QfEGBMmyJw","expires":{"seconds":1664704977,"nanos":42356498}},{"name":"forwarder-vpp-v8zrl","id":"2e946c2d-d6cc-48fa-aaea-7eeb2461b659","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLXY4enJsL2Y2NTM0NmRmLWU0YmUtNDUwOC05ZTBjLTMzYzhjMmU3OTNiOCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci00OTZqNi8xM2Y5NWYxNy1kYTVhLTQ3ZmEtOTc2NC1jYmQwNTAzZTBlNDIiXSwiZXhwIjoxNjY0NzA0OTc3fQ.tgdabUpzntDud3JjGzK0hGKxbzNJcVpqy9cdbwERvieREyUBroF8IEh9LzLS_NB_lPcJczfW-cmI8Og9tiWNig","expires":{"seconds":1664704977,"nanos":77344807},"metrics":{"client_drops":"0","client_rx_bytes":"3754836","client_rx_packets":"67170","client_tx_bytes":"6185420","client_tx_packets":"67288","server_drops":"0","server_rx_bytes":"3763052","server_rx_packets":"67288","server_tx_bytes":"3754836","server_tx_packets":"67170"}},{"name":"nsmgr-496j6","id":"8886fac2-cca2-4312-8dfc-2d33ff3c0118","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci00OTZqNi8xM2Y5NWYxNy1kYTVhLTQ3ZmEtOTc2NC1jYmQwNTAzZTBlNDIiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC02a3R4cy8zODhmOTIyMy04YjNhLTQzMzMtOGQ4Ny1mZDFjYzEwOTdhODIiXSwiZXhwIjoxNjY0NzA0OTc3fQ.hmK_cVgkKDVX5ZGoO3eNX58AHrtivnBBz243fLc9d2DjOEH147Si2uw5dEMO2Oe8gX7cNIy55821LpQFBK-rlg","expires":{"seconds":1664704977,"nanos":99708387}},{"name":"forwarder-vpp-6ktxs","id":"d4666cff-a00b-472d-b2fb-19ff76c3aa38","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLTZrdHhzLzM4OGY5MjIzLThiM2EtNDMzMy04ZDg3LWZkMWNjMTA5N2E4MiIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9pbmV0L3BvZC9uc2UtaW5ldC01NWY0NDRiNTk4LWd3NW12LzFjZTQ3NGQ2LWIwNWUtNDRkMC1iMjFhLTc5M2M0MmM2OGQ3OCJdLCJleHAiOjE2NjQ3MDQ5Nzd9.9xbUFfH5wjoQustLx9iajvKhhaPCiRXJdcRWveYmHE1z9gvXvBJriVCAlso0S2dwCy3AhiLd6m-Iy6cFfcQRKg","expires":{"seconds":1664704977,"nanos":142908776},"metrics":{"client_drops":"0","client_rx_bytes":"3754836","client_rx_packets":"67170","client_tx_bytes":"3762562","client_tx_packets":"67279","server_drops":"0","server_rx_bytes":"3762562","server_rx_packets":"67279","server_tx_bytes":"6172956","server_tx_packets":"67170"}},{"name":"nse-inet-55f444b598-gw5mv","id":"1e670852-b8f7-48a8-8997-f80ac23d44cb","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9pbmV0L3BvZC9uc2UtaW5ldC01NWY0NDRiNTk4LWd3NW12LzFjZTQ3NGQ2LWIwNWUtNDRkMC1iMjFhLTc5M2M0MmM2OGQ3OCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLTZrdHhzLzM4OGY5MjIzLThiM2EtNDMzMy04ZDg3LWZkMWNjMTA5N2E4MiJdLCJleHAiOjE2NjQ3MDQ5Nzd9.KhEsrTvsg-CUOrekXPzLlZctXj448W-RvqVp-LOP2NFN1I4xoSv7uFVHZ7De_5XaCFhrEy5FIjRgM8fuqgOEmQ","expires":{"seconds":1664704977,"nanos":144532796}}]},"network_service_endpoint_name":"nse-inet-55f444b598-gw5mv","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026532932","name":"nsm-2"}}]}
Oct  2 09:54:32.630 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (2)  ⎆ sdk/pkg/networkservice/common/refresh/refreshClient.Request()
Oct  2 09:54:32.630 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (3)   ⎆ sdk/pkg/networkservice/common/clienturl/clientURLClient.Request()
Oct  2 09:54:32.630 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (3.1)     passed clientURL: unix:///var/lib/networkservicemesh/nsm.io.sock
Oct  2 09:54:32.630 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (4)    ⎆ sdk/pkg/networkservice/common/clientconn/clientConnClient.Request()
Oct  2 09:54:32.631 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (5)     ⎆ sdk/pkg/networkservice/common/heal/healClient.Request()
Oct  2 09:54:32.631 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (6)      ⎆ sdk/pkg/networkservice/common/dial/dialClient.Request()
Oct  2 09:54:32.631 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7)       ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
Oct  2 09:54:32.631 [WARN] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.1)         Environment variable CLUSTER_NAME is not set. Skipping.
Oct  2 09:54:32.631 [WARN] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.2)         Environment variable NODE_NAME is not set. Skipping.
Oct  2 09:54:32.631 [WARN] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.3)         The label podName was already assigned to eu-west-1-4cc2a6ad8cdc4f-0. Skipping.
Oct  2 09:54:32.632 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (8)        ⎆ sdk/pkg/networkservice/common/upstreamrefresh/upstreamRefreshClient.Request()
Oct  2 09:54:32.632 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (9)         ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
Oct  2 09:54:32.632 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
Oct  2 09:54:32.632 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (11)           ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
Oct  2 09:54:32.632 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (12)            ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
Oct  2 09:54:32.633 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (12.1)              request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"file:///proc/thread-self/ns/net"}}}}
Oct  2 09:54:32.633 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (13)             ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
Oct  2 09:54:32.633 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (13.1)               request-diff={"connection":{"mechanism":{"parameters":{"inodeURL":"inode://4/4026532932"}}},"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026532932"}}}}
Oct  2 09:54:32.633 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (14)              ⎆ sdk/pkg/networkservice/connectioncontext/dnscontext/dnsContextClient.Request()
Oct  2 09:54:32.634 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (15)               ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
Oct  2 09:54:32.634 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [ExcludedPrefixesClient:Request] [type:networkService] (15.1)
      Adding new excluded IPs to the request: [10.12.1.1/32 10.12.1.0/32 172.20.0.1/32 172.20.0.10/32 172.20.2.59/32 172.20.13.247/32 172.20.32.62/32 172.20.32.125/32 172.20.34.216/32 172.20.44.123/32 172.20.48.63/32 172.20.66.157/32 172.20.86.34/32 172.20.88.185/32 172.20.90.5/32 172.20.98.91/32 172.20.157.76/32 172.20.180.165/32 172.20.192.105/32 172.20.194.58/32 172.20.194.82/32 172.20.205.246/32 172.20.210.78/32 172.20.242.74/32 172.20.244.146/32 172.20.246.26/32 172.20.76.247/32]
Oct  2 09:54:32.634 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [ExcludedPrefixesClient:Request] [type:networkService] (15.2)
      Excluded prefixes from request - [172.20.0.1/32 172.20.0.10/32 172.20.2.59/32 172.20.13.247/32 172.20.32.62/32 172.20.32.125/32 172.20.34.216/32 172.20.44.123/32 172.20.48.63/32 172.20.66.157/32 172.20.86.34/32 172.20.88.185/32 172.20.90.5/32 172.20.98.91/32 172.20.157.76/32 172.20.180.165/32 172.20.192.105/32 172.20.194.58/32 172.20.194.82/32 172.20.205.246/32 172.20.210.78/32 172.20.242.74/32 172.20.244.146/32 172.20.246.26/32 172.20.76.247/32]
Oct  2 09:54:32.634 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (16)                ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
Oct  2 09:54:32.634 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (17)                 ⎆ sdk/pkg/networkservice/common/trimpath/trimpathClient.Request()
Oct  2 09:54:32.634 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (18)                  ⎆ sdk/pkg/networkservice/common/connect/connectClient.Request()
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (18.1)                    rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF;  Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.logError;            /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/common.go:206;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;            /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:57;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;       /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60;     github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:81;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request/go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60;     github.com/networkservicemesh/sdk/pkg/networkservice/common/trimpath.(*trimpathClient).Request;              /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/common/trimpath/client.go:38;   github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:55;   github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60;github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:81;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize.(*authorizeClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/common/authorize/client.go:68;  github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:55;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request/go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60;     github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request;  /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:81;    github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/excludedprefixes.(*excludedPrefixesClient).Request;              /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/common/excludedprefixes/client.go:113;  github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;            /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:55;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:81;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;       /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60;     github.com/networkservicemesh/sdk/pkg/networkservice/connectioncontext/dnscontext.(*dnsContextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/connectioncontext/dnscontext/client.go:82;      github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:55;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request;          /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:81;    github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/common/mechanisms/sendfd/client.go:55;  github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;            /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:55;
Oct  2 09:54:35.772 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (17.1)                   request-response=null
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (17.2)                   Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (16.1)                  Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (15.3)                 Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (14.1)                Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (13.2)               Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (12.2)              Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (11.1)             Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (10.1)            Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (9.1)           Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (8.1)          Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.772 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.4)         Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.775 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (6.1)        Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.775 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (5.1)       Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.775 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (4.1)      Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.775 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (3.2)     Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.775 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (2.1)    Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.775 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (1.2)   Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unavailable desc = error reading from server: EOF
Oct  2 09:54:35.775 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
Oct  2 09:54:35.775 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (1.1)   request={"connection":{"id":"eu-west-1-4cc2a6ad8cdc4f-0-0","network_service":"nsm-inet","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"file:///proc/thread-self/ns/net","name":"nsm-2"}},"context":{"ip_context":{"src_ip_addrs":["10.12.1.1/32"],"dst_ip_addrs":["10.12.1.0/32"],"src_routes":[{"prefix":"10.12.1.0/32"}],"dst_routes":[{"prefix":"10.12.1.1/32"}],"excluded_prefixes":["172.20.0.1/32","172.20.0.10/32","172.20.2.59/32","172.20.13.247/32","172.20.32.62/32","172.20.32.125/32","172.20.34.216/32","172.20.44.123/32","172.20.48.63/32","172.20.66.157/32","172.20.86.34/32","172.20.88.185/32","172.20.90.5/32","172.20.98.91/32","172.20.157.76/32","172.20.180.165/32","172.20.192.105/32","172.20.194.58/32","172.20.194.82/32","172.20.205.246/32","172.20.210.78/32","172.20.242.74/32","172.20.244.146/32","172.20.246.26/32","172.20.76.247/32"],"policies":[{"proto":"6"},{"proto":"1"},{"proto":"17"}]},"dns_context":{"configs":[{"dns_server_ips":["172.20.0.10"],"search_domains":["4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local","svc.cluster.local","cluster.local","eu-west-1.compute.internal"]},{"dns_server_ips":["172.20.0.10"],"search_domains":["4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local","svc.cluster.local","cluster.local","eu-west-1.compute.internal"]},{"dns_server_ips":["172.20.0.10"],"search_domains":["4cc2a6ad-8cdc-4f2d-86d3-c02b73190f0d.svc.cluster.local","svc.cluster.local","cluster.local","eu-west-1.compute.internal"]}]},"MTU":8947},"labels":{"nodeName":"ip-10-111-100-207.eu-west-1.compute.internal","podName":"eu-west-1-4cc2a6ad8cdc4f-0","subnet":"100.124.0.0/24"},"path":{"path_segments":[{"name":"eu-west-1-4cc2a6ad8cdc4f-0","id":"eu-west-1-4cc2a6ad8cdc4f-0-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy80Y2MyYTZhZC04Y2RjLTRmMmQtODZkMy1jMDJiNzMxOTBmMGQvcG9kL2V1LXdlc3QtMS00Y2MyYTZhZDhjZGM0Zi0wLzkzM2ExZjczLTYwMDQtNGVmNi1iMzc1LWRjMGRjY2Q3MTAyNCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci04aHpreC8xOTRmYzFkOS05YTBhLTRmZGYtOWRiYS1mZTI0MjI5NGQ4NGMiXSwiZXhwIjoxNjY0NzA0OTc3fQ.Czx66_sNuJ6ucI_u12a07kG0-Me97ozlEwHazS-HMuhrsNLGwZsMcdBaNdIaafqb_E0D7yi8AI758QeQEEEHDw","expires":{"seconds":1664704977,"nanos":20862120}},{"name":"nsmgr-8hzkx","id":"82f870d8-ad8f-40bb-93b3-820348581331","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci04aHpreC8xOTRmYzFkOS05YTBhLTRmZGYtOWRiYS1mZTI0MjI5NGQ4NGMiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC12OHpybC9mNjUzNDZkZi1lNGJlLTQ1MDgtOWUwYy0zM2M4YzJlNzkzYjgiXSwiZXhwIjoxNjY0NzA0OTc3fQ.pCAczepbvdRPVRMc2qT--deGsjUexCJImRpQEHRhkV06sZcg7MpS21dfjH1CEoAFbwrRoL4uj7Z7QfEGBMmyJw","expires":{"seconds":1664704977,"nanos":42356498}},{"name":"forwarder-vpp-v8zrl","id":"2e946c2d-d6cc-48fa-aaea-7eeb2461b659","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLXY4enJsL2Y2NTM0NmRmLWU0YmUtNDUwOC05ZTBjLTMzYzhjMmU3OTNiOCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci00OTZqNi8xM2Y5NWYxNy1kYTVhLTQ3ZmEtOTc2NC1jYmQwNTAzZTBlNDIiXSwiZXhwIjoxNjY0NzA0OTc3fQ.tgdabUpzntDud3JjGzK0hGKxbzNJcVpqy9cdbwERvieREyUBroF8IEh9LzLS_NB_lPcJczfW-cmI8Og9tiWNig","expires":{"seconds":1664704977,"nanos":77344807},"metrics":{"client_drops":"0","client_rx_bytes":"3754836","client_rx_packets":"67170","client_tx_bytes":"6185420","client_tx_packets":"67288","server_drops":"0","server_rx_bytes":"3763052","server_rx_packets":"67288","server_tx_bytes":"3754836","server_tx_packets":"67170"}},{"name":"nsmgr-496j6","id":"8886fac2-cca2-4312-8dfc-2d33ff3c0118","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9uc21nci00OTZqNi8xM2Y5NWYxNy1kYTVhLTQ3ZmEtOTc2NC1jYmQwNTAzZTBlNDIiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC02a3R4cy8zODhmOTIyMy04YjNhLTQzMzMtOGQ4Ny1mZDFjYzEwOTdhODIiXSwiZXhwIjoxNjY0NzA0OTc3fQ.hmK_cVgkKDVX5ZGoO3eNX58AHrtivnBBz243fLc9d2DjOEH147Si2uw5dEMO2Oe8gX7cNIy55821LpQFBK-rlg","expires":{"seconds":1664704977,"nanos":99708387}},{"name":"forwarder-vpp-6ktxs","id":"d4666cff-a00b-472d-b2fb-19ff76c3aa38","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLTZrdHhzLzM4OGY5MjIzLThiM2EtNDMzMy04ZDg3LWZkMWNjMTA5N2E4MiIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9pbmV0L3BvZC9uc2UtaW5ldC01NWY0NDRiNTk4LWd3NW12LzFjZTQ3NGQ2LWIwNWUtNDRkMC1iMjFhLTc5M2M0MmM2OGQ3OCJdLCJleHAiOjE2NjQ3MDQ5Nzd9.9xbUFfH5wjoQustLx9iajvKhhaPCiRXJdcRWveYmHE1z9gvXvBJriVCAlso0S2dwCy3AhiLd6m-Iy6cFfcQRKg","expires":{"seconds":1664704977,"nanos":142908776},"metrics":{"client_drops":"0","client_rx_bytes":"3754836","client_rx_packets":"67170","client_tx_bytes":"3762562","client_tx_packets":"67279","server_drops":"0","server_rx_bytes":"3762562","server_rx_packets":"67279","server_tx_bytes":"6172956","server_tx_packets":"67170"}},{"name":"nse-inet-55f444b598-gw5mv","id":"1e670852-b8f7-48a8-8997-f80ac23d44cb","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9pbmV0L3BvZC9uc2UtaW5ldC01NWY0NDRiNTk4LWd3NW12LzFjZTQ3NGQ2LWIwNWUtNDRkMC1iMjFhLTc5M2M0MmM2OGQ3OCIsImF1ZCI6WyJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3BvZC9mb3J3YXJkZXItdnBwLTZrdHhzLzM4OGY5MjIzLThiM2EtNDMzMy04ZDg3LWZkMWNjMTA5N2E4MiJdLCJleHAiOjE2NjQ3MDQ5Nzd9.KhEsrTvsg-CUOrekXPzLlZctXj448W-RvqVp-LOP2NFN1I4xoSv7uFVHZ7De_5XaCFhrEy5FIjRgM8fuqgOEmQ","expires":{"seconds":1664704977,"nanos":144532796}}]},"network_service_endpoint_name":"nse-inet-55f444b598-gw5mv","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026532932","name":"nsm-2"}}]}
Oct  2 09:54:35.775 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (2)  ⎆ sdk/pkg/networkservice/common/refresh/refreshClient.Request()
Oct  2 09:54:35.776 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (3)   ⎆ sdk/pkg/networkservice/common/clienturl/clientURLClient.Request()
Oct  2 09:54:35.776 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (3.1)     passed clientURL: unix:///var/lib/networkservicemesh/nsm.io.sock
Oct  2 09:54:35.776 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (4)    ⎆ sdk/pkg/networkservice/common/clientconn/clientConnClient.Request()
Oct  2 09:54:35.776 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (5)     ⎆ sdk/pkg/networkservice/common/heal/healClient.Request()
Oct  2 09:54:35.776 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (6)      ⎆ sdk/pkg/networkservice/common/dial/dialClient.Request()
Oct  2 09:54:35.776 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7)       ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
Oct  2 09:54:35.777 [WARN] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.1)         Environment variable NODE_NAME is not set. Skipping.
Oct  2 09:54:35.777 [WARN] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.2)         The label podName was already assigned to eu-west-1-4cc2a6ad8cdc4f-0. Skipping.
Oct  2 09:54:35.777 [WARN] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.3)         Environment variable CLUSTER_NAME is not set. Skipping.
Oct  2 09:54:35.777 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (8)        ⎆ sdk/pkg/networkservice/common/upstreamrefresh/upstreamRefreshClient.Request()
Oct  2 09:54:35.777 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (9)         ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
Oct  2 09:54:35.777 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (10)          ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
Oct  2 09:54:35.777 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (11)           ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
Oct  2 09:54:35.777 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (12)            ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
Oct  2 09:54:35.778 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (12.1)              request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"file:///proc/thread-self/ns/net"}}}}
Oct  2 09:54:35.778 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (13)             ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
Oct  2 09:54:35.778 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (13.1)               request-diff={"connection":{"mechanism":{"parameters":{"inodeURL":"inode://4/4026532932"}}},"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026532932"}}}}
Oct  2 09:54:35.778 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (14)              ⎆ sdk/pkg/networkservice/connectioncontext/dnscontext/dnsContextClient.Request()
Oct  2 09:54:35.779 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (15)               ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
Oct  2 09:54:35.779 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [ExcludedPrefixesClient:Request] [type:networkService] (15.1)
      Adding new excluded IPs to the request: [10.12.1.1/32 10.12.1.0/32 172.20.0.1/32 172.20.0.10/32 172.20.2.59/32 172.20.13.247/32 172.20.32.62/32 172.20.32.125/32 172.20.34.216/32 172.20.44.123/32 172.20.48.63/32 172.20.66.157/32 172.20.86.34/32 172.20.88.185/32 172.20.90.5/32 172.20.98.91/32 172.20.157.76/32 172.20.180.165/32 172.20.192.105/32 172.20.194.58/32 172.20.194.82/32 172.20.205.246/32 172.20.210.78/32 172.20.242.74/32 172.20.244.146/32 172.20.246.26/32 172.20.76.247/32]
Oct  2 09:54:35.779 [DEBU] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [ExcludedPrefixesClient:Request] [type:networkService] (15.2)
      Excluded prefixes from request - [172.20.0.1/32 172.20.0.10/32 172.20.2.59/32 172.20.13.247/32 172.20.32.62/32 172.20.32.125/32 172.20.34.216/32 172.20.44.123/32 172.20.48.63/32 172.20.66.157/32 172.20.86.34/32 172.20.88.185/32 172.20.90.5/32 172.20.98.91/32 172.20.157.76/32 172.20.180.165/32 172.20.192.105/32 172.20.194.58/32 172.20.194.82/32 172.20.205.246/32 172.20.210.78/32 172.20.242.74/32 172.20.244.146/32 172.20.246.26/32 172.20.76.247/32]
Oct  2 09:54:35.779 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (16)                ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
Oct  2 09:54:35.779 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (17)                 ⎆ sdk/pkg/networkservice/common/trimpath/trimpathClient.Request()
Oct  2 09:54:35.779 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (18)                  ⎆ sdk/pkg/networkservice/common/connect/connectClient.Request()
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (18.1)                    rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed;      Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request;        github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.logError;            /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/common.go:206;       github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;/go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:57;    github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;       /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60;     github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:81;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request/go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60;     github.com/networkservicemesh/sdk/pkg/networkservice/common/trimpath.(*trimpathClient).Request;              /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/common/trimpath/client.go:38;   github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:55;   github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60;github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:81;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize.(*authorizeClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/common/authorize/client.go:68;  github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:55;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request/go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60;     github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request;  /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:81;    github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/excludedprefixes.(*excludedPrefixesClient).Request;              /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/common/excludedprefixes/client.go:113;  github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;            /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:55;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:81;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;       /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60;     github.com/networkservicemesh/sdk/pkg/networkservice/connectioncontext/dnscontext.(*dnsContextClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/connectioncontext/dnscontext/client.go:82;      github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;         /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:55;        github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;           /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request;          /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:81;    github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request;                /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request;               /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/common/mechanisms/sendfd/client.go:55;  github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request;            /go/pkg/mod/github.com/networkservicemesh/sdk@v1.5.0-rc.2/pkg/networkservice/core/trace/client.go:55;
Oct  2 09:54:46.206 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (17.1)                   request-response=null
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (17.2)                   Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (16.1)                  Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (15.3)                 Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (14.1)                Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (13.2)               Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (12.2)              Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (11.1)             Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (10.1)            Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (9.1)           Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (8.1)          Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (7.4)         Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (6.1)        Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (5.1)       Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (4.1)      Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (3.2)     Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (2.1)    Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [ERRO] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (1.2)   Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-v8zrl --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-6ktxs --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request: network service endpoint nse-inet-55f444b598-gw5mv not found: all forwarders have failed: all forwarders have failed
Oct  2 09:54:46.206 [TRAC] [id:eu-west-1-4cc2a6ad8cdc4f-0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()

Around 09:54 I deleted nsmgr-496j6 & nse-inet-55f444b598-gw5m pods to receive this error.
nse-inet-55f444b598-gw5m is the only pod in the replica for this nse.
As a result, the ReplicaSet rescheduled a new pod nse-inet-55f444b598-fkl4g, but my nsc still references nse-inet-55f444b598-gw5m (which was deleted)

Also, sorry for the confusion but @yuraxdrumz and I work together on the same setup,
we thought that the error was caused only by the NSE being rescheduled.
but it seems that it happened due to multiple components being deleted and rescheduled (NSE & NSMgr, for instance).

@glazychev-art
Copy link
Contributor

@or-adar
Thanks for the logs!

It is very interesting. A few questions:

  1. Do you still have multiple clients in this setup?
  2. Do you have any additional envs for the NSC? (like NSM_LIVENESSCHECK...)
  3. What version of forwarder-vpp are you using?

@or-adar
Copy link

or-adar commented Oct 3, 2022

@glazychev-art

  1. I managed to reproduce it for both multiple & single NSC scenarios. For the scenario shown in the logs, I had only one NSC.
  2. For the NSC I use the following envs: NSM_LOG_LEVEL, NSM_REQUEST_TIMEOUT, NSM_LIVENESSCHECKTIMEOUT, NSM_LIVENESSCHECKINTERVAL, SPIFFE_ENDPOINT_SOCKET, NSM_NAME, POD_NAME, NSM_NETWORK_SERVICES
  3. We use a custom fork of forwarder-vpp. it should represent v1.3.0

@denis-tingaikin
Copy link
Member

@or-adar

We use a custom fork of forwarder-vpp. it should represent v1.3.0

Interesting... Is the fork public? Is it possible to share? It'd be nice to use for reproducing

@denis-tingaikin
Copy link
Member

Also values forNSM_LIVENESSCHECKTIMEOUT, NSM_LIVENESSCHECKINTERVAL would be nice to get :)

@yuraxdrumz
Copy link
Author

Hey @denis-tingaikin @glazychev-art

We use a a slightly modified forwarder-vpp v1.5.0, that uses this PR I opened, not v1.3.0 like @or-adar mentioned.

Regarding liveness

NSM_REQUEST_TIMEOUT=15s

NSM_LIVENESSCHECKINTERVAL=10s

NSM_LIVENESSCHECKTIMEOUT=40s

@or-adar
Copy link

or-adar commented Oct 3, 2022

@denis-tingaikin @glazychev-art
To reproduce it I used the following values for the requested envs:

NSM_REQUEST_TIMEOUT=60s
NSM_LIVENESSCHECKINTERVAL=10s
NSM_LIVENESSCHECKTIMEOUT=40s

@denis-tingaikin
Copy link
Member

@yuraxdrumz , @or-adar Thanks!

Question: Why you don't use default values? Is something going wrong?

@glazychev-art
Copy link
Contributor

glazychev-art commented Oct 3, 2022 •
edited
Loading

@yuraxdrumz @or-adar
NSM_LIVENESSCHECKTIMEOUT means that one session of DataPath validation will take 40 seconds. Only after this time a decision is made - DataPath is fine or not.
NSM_LIVENESSCHECKINTERVAL - this is the interval between these sessions.
In other words, the selection of a new endpoint will take place in about 40-80 seconds.
80 seconds may be due to the fact that in the first session one of the ping packets was successful after the death of the Nsmgr, but before the death of the NSE. Until then, there will be attempts to re-connect to the old one.

Please, check that after 80 seconds everything will be fine. But I would recommend using the default values ​​(or if they are not enough, slightly increase by 1-5 seconds)

@yuraxdrumz
Copy link
Author

@glazychev-art Thanks for the information!

We ran some various commands manually during development and the pinger failed so fast, we could not check anything, so we increased the limits.

If I understand, we are hitting a scenario in between deaths of NSMGR and NSE and it should resolve itself after 2 X NSM_LIVENESSCHECKTIMEOUT, but from what I saw, the state of network service endpoint candidates not found never resolves. We left it for days to see what happens and nothing changes. @or-adar correct me if I am wrong

Decreasing the values only narrows the window for this error to occur, but it does not resolve it completely, no?

@glazychev-art
Copy link
Contributor

@yuraxdrumz
Yes, but if you see network service endpoint candidates not found - it really can be indefinitely.
Possible cause of this error - #1357 (comment)
If there is a service name mismatch (NSC and NSE), that is exactly what will happen.

But on the last logs we see another error - network service endpoint {NAME} not found: all forwarders have failed. This error may be NSM_LIVENESSCHECKTIMEOUT related (you are right, it should be resolved after 2 X NSM_LIVENESSCHECKTIMEOUT).

@or-adar
Copy link

or-adar commented Oct 3, 2022 •
edited
Loading

@denis-tingaikin @glazychev-art
Sorry for the confusion, looks like I had other NSCs running without noticing when trying to reproduce this issue,
so seems like this error, as you mentioned is related to #1358

@glazychev-art glazychev-art mentioned this issue Oct 11, 2022
Merged
9 tasks
@glazychev-art
Copy link
Contributor

We seem to have found a way to reproduce this issue.
Logs from AWS cluster:
TestRunHealSuite.zip

Steps to reproduce:

  1. Deploy any NSM remote case (for example kernel2vxlan2kernel)
  2. Delete remote Nsmgr pod
  3. Wait for a new Nsmgr
  4. Delete NSE pod

Attached the fix above

@glazychev-art glazychev-art mentioned this issue Oct 13, 2022
Merged
9 tasks
@denis-tingaikin
Copy link
Member

Should be fixed.

Feel free to reopen if problem is still reproducible :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@glazychev-art glazychev-art

@NikitaSkrynnik NikitaSkrynnik

Labels
ASAP The issue that blocking SOW items or core use-cases bug Something isn't working
Projects
None yet
Milestone
v1.6.0
Development

No branches or pull requests
6 participants
@edwarnicke @yuraxdrumz @glazychev-art @or-adar @denis-tingaikin @NikitaSkrynnik