antrea CNI not setting default gateway correctly

[jayunit100]

Antrea CNI works be using the node-ip address to find a device, and then tries to use that specific Device to pick a gateway. That default gateway is the uplink for OVS.

However, it looks like it wants that device to have a destinationPrefix of 0.0.0.0/0 , as a way to find the default gateway by then extracting the NextHop value.

So the question is... can we hardcode that NextHop value in cases like, this repos, where the node ip doesnt come up w/ a device that is directly connected to a default gateway?

update this is fixed now:) antrea-io/antrea@da17b5d

• agent_windows.go: nodeConfig.NodeIpAddr.Ip → GetIPNetDeviceFromIP(..) → adaptor .index → GetDefaultGatewayByInterface(...) → default gateway string
• Why is the GetDefaultGateway call failing ?
  • because the node ip address device doesn't have a destination prefix for the default gateway... 0.0.0.0/0 !
    • what device DOES have a 0.0.0.0 destination prefix ???
      • Can we hack or hardcode antrea to consume THAT device to be the thing as the uplink interface ?

[jayunit100]

working cluster from another env on vsphere, we can see that Get-NetRoute has a 0.0.0.0/0 destination prefix for the 13 if... that 13 if is queried (see bottom of this paste), using Get-NetAdapter

PS C:\Users\capv> Get-NetRoute                                                                                                                                                                                                                                                                                              
13      10.161.191.255/32                              0.0.0.0                                          256 15       ActiveStore                                                                                                                                                                                            
13      10.161.170.22/32                               0.0.0.0                                          256 15       ActiveStore                                                                                                                                                                                            
13      10.161.160.0/19                                0.0.0.0                                          256 15       ActiveStore                                                                                                                                                                                            
13      0.0.0.0/0                                      10.161.191.253                                     0 15       ActiveStore                                                                                                                                                                                            
13      ::/0                                           fe80::2613:191:253                               256 15       ActiveStore                                                                                                                                                                                                                              

the ipconfig output

PS C:\Users\capv> ipconfig                                                                                                                                                                                                                                                                               
Windows IP Configuration                                                                                                                                                                                                                                                                                 
Ethernet adapter vEthernet (KubeProxyInternalSwitch):                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                            
   Connection-specific DNS Suffix  . :                                                                                                                                                                                                                                                                                      
   Link-local IPv6 Address . . . . . : fe80::7542:3120:8fbd:fd36%8                                                                                                                                                                                                                                                          
   Autoconfiguration IPv4 Address. . : 169.254.253.54                                                                                                                                                                                                                                                                       
   Subnet Mask . . . . . . . . . . . : 255.255.0.0                                                                                                                                                                                                                                                                          
   Default Gateway . . . . . . . . . :                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                            
Ethernet adapter vEthernet (HNS Internal NIC):                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                            
   Connection-specific DNS Suffix  . :                                                                                                                                                                                                                                                                                      
   Link-local IPv6 Address . . . . . : fe80::3d68:4a12:a302:4084%4                                                                                                                                                                                                                                                          
   IPv4 Address. . . . . . . . . . . : 100.64.0.1                                                                                                                                                                                                                                                                           
   Subnet Mask . . . . . . . . . . . : 255.0.0.0                                                                                                                                                                                                                                                                            
   IPv4 Address. . . . . . . . . . . : 100.64.0.10                                                                                                                                                                                                                                                                          
   Subnet Mask . . . . . . . . . . . : 255.0.0.0                                                                                                                                                                                                                                                                            
   IPv4 Address. . . . . . . . . . . : 100.66.250.99                                                                                                                                                                                                                                                                        
   Subnet Mask . . . . . . . . . . . : 255.0.0.0                                                                                                                                                                                                                                                                            
   IPv4 Address. . . . . . . . . . . : 100.68.62.30                                                                                                                                                                                                                                                                         
   Subnet Mask . . . . . . . . . . . : 255.0.0.0                                                                                                                                                                                                                                                                            
   IPv4 Address. . . . . . . . . . . : 100.68.198.134                                                                                                                                                                                                                                                                       
   Subnet Mask . . . . . . . . . . . : 255.0.0.0                                                                                                                                                                                                                                                                            
   IPv4 Address. . . . . . . . . . . : 100.70.143.158                                                                                                                                                                                                                                                                       
   Subnet Mask . . . . . . . . . . . : 255.0.0.0                                                                                                                                                                                                                                                                            
   Default Gateway . . . . . . . . . :                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                            
Ethernet adapter br-int:                                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                            
   Connection-specific DNS Suffix  . : eng.vmware.com                                                                                                                                                                                                                                                                       
   IPv6 Address. . . . . . . . . . . : fd01:0:101:2613:0:a:0:a2b                                                                                                                                                                                                                                                            
   IPv6 Address. . . . . . . . . . . : fd01:0:101:2613:2c28:2de8:3280:b7a5                                                                                                                                                                                                                                                  
   Link-local IPv6 Address . . . . . : fe80::2c28:2de8:3280:b7a5%13                                                                                                                                                                                                                                                         
   IPv4 Address. . . . . . . . . . . : 10.161.170.22                                                                                                                                                                                                                                                                        
   Subnet Mask . . . . . . . . . . . : 255.255.224.0                                                                                                                                                                                                                                                                        
   Default Gateway . . . . . . . . . : fe80::2613:191:253%13                                                                                                                                                                                                                                                                
                                       10.161.191.253                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                            
Ethernet adapter antrea-gw0:                                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                            
   Connection-specific DNS Suffix  . :                                                                                                                                                                                                                                                                                      
   IPv6 Address. . . . . . . . . . . : fd01:0:101:2613:61de:7d5a:1724:fee3                                                                                                                                                                                                                                                  
   Link-local IPv6 Address . . . . . : fe80::61de:7d5a:1724:fee3%25                                                                                                                                                                                                                                                         
   IPv4 Address. . . . . . . . . . . : 100.96.3.1                                                                                                                                                                                                                                                                           
   Subnet Mask . . . . . . . . . . . : 255.255.255.0                                                                                                                                                                                                                                                                        
   Default Gateway . . . . . . . . . :                                                                                                                                                                                                                                                                                      

the network adapter 13... is the one that is correct

PS C:\Users\capv> Get-NetAdapter                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                            
Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed                                                                                                                                                                                                     
----                      --------------------                    ------- ------       ----------             ---------                                                                                                                                                                                                     
vEthernet (KubeProxyIn... Hyper-V Virtual Ethernet Adapter              8 Up           00-15-5D-BE-30-00        10 Gbps                                                                                                                                                                                                     
antrea-gw0                Hyper-V Virtual Ethernet Adapter #4          25 Up           00-15-5D-BE-30-03        10 Gbps                                                                                                                                                                                                     
br-int                    Hyper-V Virtual Ethernet Adapter #3          13 Up           00-50-56-AF-DF-0B        10 Gbps                                                                                                                                                                                                     
vEthernet (HNS Interna... Hyper-V Virtual Ethernet Adapter #2           4 Up           00-15-5D-BE-30-01        10 Gbps                                                                                                                                                                                                     
Ethernet0                 vmxnet3 Ethernet Adapter                      2 Up           00-50-56-AF-DF-0B        10 Gbps                                                                                                                                                                                                     

[jayunit100]

xref antrea-io/antrea#2344
ive experimented with a couple of other options/workarounds for us...

antrea

• public_ip seems to fail (somehow in this case, windows doesnt setup the right routes at all
• private_ip which is what we do now seems to mostly work, but doesnt have a default gateway, so antrea gets mad

calico

• we havent tried calico yet but,
• calico might allows us by https://docs.projectcalico.org/reference/felix/configuration to configure the default interface to use

[jayunit100]

@wenyingd can you suggest us a fix here ? Would like to keep using antrea as the default CNI if possible. But Virtualbox wants to make 2 NICs and my initial experiments to use public_network seem to fail.

[JocelynBerrendonner]

@Keith-Mange, @daschott, @sbangari, the above topic was brought up by @jayunit100 during the SIG Windows meeting today. Can I let you look into it, please?

[jayunit100]

as one workaround, ive attempted to use public_networks, however that didnt seem to result in the same IP space for windows and linux. Somehow the windows VMs dont adopt the same 10.... subnet ips. https://serverfault.com/questions/1068830/how-can-i-use-vagrant-to-give-windows-and-linux-servers-in-a-vagrantvirtualbox

[jayunit100]

Reproducing the issue:

• set cni=antrea
• make all
• verify that antrea-agent fails
• try to swap antrea agent out with Decouple kubelet node-ip from the Antrea default Gateway uplink antrea-io/antrea#2344
  • make windows to make antrea-agent.exe
  • mount the new antrea-agent.exe into the windows VM

[wenyingd]

You can follow these steps to test antrea patch: antrea-io/antrea#2344

1. Compile antrea-agent.exe under antrea package. If you run Antrea Agent in daemonset (requiring docker is working as runtime on Windows Node), run make build-windows to generate Windows docker image; If you run it as a Windows process, run make windows-bin
2. Ensure the Windows network adpater which is expected to work as the transport interface for Pods is using the same name on all Windows Nodes
3. Modify Antrea Agent configuration item transportInterface with the name of the network adapter in step2.
4. Run Antrea Agent

BTW, I always compile antrea-agent.exe on Windows host successfully, didn't try it on other kinds of OSs.

[jayunit100]

ok thanks @wenyingd , will test it out - do you have a binary we can use ?

[jayunit100]

potentially fixed by #90 ... will try to close this once its automated with the right NIC input

[knabben]

Having this issue when provisioning Antrea steps on master

Creating network adapter: vEthernet (HNS Internal NIC) for kube-proxy
8/31/2021 11:41 AM Installation log location: C:\k\antrea\install_ovs.log 

8/31/2021 11:41 AM Downloading OVS package from https://downloads.antrea.io/ovs/ovs-2.14.2-antrea.1-win64.zip to C:\k\antrea\ovs-win64.zip 

8/31/2021 11:41 AM Download OVS package success. 

8/31/2021 11:41 AM Extract C:\k\antrea\ovs-win64.zip to C:\k\antrea 

8/31/2021 11:41 AM Copying OVS package from C:\k\antrea\openvswitch to C:\openvswitch 

8/31/2021 11:41 AM Installing OVS kernel driver 

8/31/2021 11:41 AM Hyper-V Virtual Machine Management service status: Running 


C:\openvswitch\driver>netcfg -l .\ovsext.inf -c s -i OVSExt 
Trying to install OVSExt ...

... failed. Error code: 0xe0000247.


C:\openvswitch\driver>net stop vmms 
The Hyper-V Virtual Machine Management service is stopping.
The Hyper-V Virtual Machine Management service was stopped successfully.


C:\openvswitch\driver>net start vmms 
The Hyper-V Virtual Machine Management service is starting.
The Hyper-V Virtual Machine Management service was started successfully.

8/31/2021 11:41 AM Creating ovsdb file 

8/31/2021 11:41 AM Create and start ovsdb-server service 

[SC] CreateService SUCCESS
[SC] ChangeServiceConfig2 SUCCESS


Stderr from the command:

Start-Service : Service 'ovsdb-server (ovsdb-server)' cannot be started due to the following error: Cannot start 
service ovsdb-server on computer '.'.
At C:\k\antrea\Install-OVS.ps1:214 char:5
+     Start-Service ovsdb-server
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service],  
   ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

Rerunning solved the issue, still investigating if related with last 1.3 release change

[knabben]

Testing the new version on:
#92

[wenyingd]

Hi, the error occurs if you install the upstream OVS driver on Windows host for the first time. It is because the upstream OVS driver is not signed, and Windows doesn't trust the publisher. To resolve it, you need to enable testsigning mode on the Windows host in advance, and it should be safe to use a console for the installation at the first time, because Windows might give a popup window to ask if the user trusts the vendor or not. The user should confirm the installation.

[knabben]

@wenyingd thanks.

will try these steps
https://docs.microsoft.com/en-us/windows-hardware/drivers/install/the-testsigning-boot-configuration-option

[perithompson]

Bcdedit.exe -set TESTSIGNING ON
Restart-Computer -Force

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.