netlink.LinkSetMaster() arbitrarily changes MAC addresses

[twelho]

I'm importing netlink v1.1.0 to create TAP adapters and a bridge device, and using netlink.LinkSetMaster(tapAdapter, bridge) to bind them together. This operation however results in the MAC addresses of the TAP adapter and the bridge arbitrarily changing, sometimes to the same MAC address which breaks network connectivity.

What I've been able to deduce:

• The MAC address changes are not consistent, sometimes only one interface changes, other times none change or then both of them change. If any of them change there's a relatively high chance that both will have the same MAC in the end.
• This seems to only happen when attaching only virtual devices to the bridge. I was not able to reproduce this if I set the bridge to be the master of my physical ethernet card first.
• This affects netlink.LinkSetMasterByIndex() as well, and it doesn't matter what handle is used to perform the call.
• I've reproduced this on bare metal as well as in a containerd container in Ignite, see ignite-spawn container has multiple interfaces with same MAC, networking (randomly) broken weaveworks/ignite#633.

I've created a small test binary isolating the issue here: https://github.com/twelho/netlinktest.
Sample output of what I'm seeing:

WARN[0000] Attach first adapter to bridge               
INFO[0000] Pre-LinkSetMaster "nltest_tp0" MAC address: f2:23:a7:6b:41:27 
INFO[0000] Pre-LinkSetMaster "nltest_br0" MAC address: 02:63:93:fa:61:32 
INFO[0000] LinkSetMaster: "nltest_tp0" -> "nltest_br0"  
INFO[0000] Post-LinkSetMaster "nltest_tp0" MAC address: fe:2e:d5:93:65:bb 
INFO[0000] Post-LinkSetMaster "nltest_br0" MAC address: fe:2e:d5:93:65:bb 
ERRO[0000] "nltest_tp0" changed MAC address: f2:23:a7:6b:41:27 -> fe:2e:d5:93:65:bb 
ERRO[0000] "nltest_br0" changed MAC address: 02:63:93:fa:61:32 -> fe:2e:d5:93:65:bb 
ERRO[0000] Duplicate MAC address: fe:2e:d5:93:65:bb     
WARN[0000] Attach second adapter to bridge              
INFO[0000] Pre-LinkSetMaster "nltest_tp1" MAC address: 72:e2:26:a9:fb:9c 
INFO[0000] Pre-LinkSetMaster "nltest_br0" MAC address: fe:2e:d5:93:65:bb 
INFO[0000] LinkSetMaster: "nltest_tp1" -> "nltest_br0"  
INFO[0000] Post-LinkSetMaster "nltest_tp1" MAC address: 72:e2:26:a9:fb:9c 
INFO[0000] Post-LinkSetMaster "nltest_br0" MAC address: c6:1b:55:d0:39:67 
ERRO[0000] "nltest_br0" changed MAC address: fe:2e:d5:93:65:bb -> c6:1b:55:d0:39:67

[bboreham]

I expect that it's the Linux kernel which is doing the changes.
This library's job is formatting requests and sending them to the kernel.

[bboreham]

Couldn't find a reference yesterday, sorry, but today I have dozens:

by default bridge interfaces in Linux use, for their MAC address, the lowest MAC address among the enslaved interfaces.
https://backreference.org/2010/07/28/linux-bridge-mac-addresses-and-dynamic-ports/

Unless overridden, the bridge device takes the lowest MAC address of all the devices in the bridge.
[...] you can solve the problem by assigning a MAC address to the bridge device
https://lists.linuxfoundation.org/pipermail/bridge/2010-May/007204.html

[twelho]

I wasn't aware that it's intended functionality for the bridge to change its MAC, especially with the ip tool seemingly always defining a persistent MAC for the bridge as soon as it's created. There's still something odd going on however, since it seems like the first TAP device (sometimes) also magically changes its MAC:

WARN[0000] Attach first adapter to bridge               
INFO[0000] Pre-LinkSetMaster "nltest_tp0" MAC address: 32:db:d9:bb:23:2c 
INFO[0000] Pre-LinkSetMaster "nltest_br0" MAC address: 12:3c:86:f7:77:b0 
INFO[0000] LinkSetMaster: "nltest_tp0" -> "nltest_br0"  
INFO[0000] Post-LinkSetMaster "nltest_tp0" MAC address: fe:2e:d5:93:65:bb 
INFO[0000] Post-LinkSetMaster "nltest_br0" MAC address: c6:1b:55:d0:39:67 
ERRO[0000] "nltest_tp0" changed MAC address: 32:db:d9:bb:23:2c -> fe:2e:d5:93:65:bb 
WARN[0000] Attach second adapter to bridge              
INFO[0000] Pre-LinkSetMaster "nltest_tp1" MAC address: 72:e2:26:a9:fb:9c 
INFO[0000] Pre-LinkSetMaster "nltest_br0" MAC address: c6:1b:55:d0:39:67 
INFO[0000] LinkSetMaster: "nltest_tp1" -> "nltest_br0"  
INFO[0000] Post-LinkSetMaster "nltest_tp1" MAC address: 72:e2:26:a9:fb:9c 
INFO[0000] Post-LinkSetMaster "nltest_br0" MAC address: c6:1b:55:d0:39:67

I can get behind the logic for the bridge, but where is the TAP interfaces changing MACs documented? I've updated https://github.com/twelho/netlinktest accordingly to ignore the bridge's changes and only look at the interfaces being attached to it, the above snippet shows what the issue looks like now.

[vishvananda]

I don't believe this is a netlink library issue. I ran into this quite frequently in my early days of linux kernel networking.
Example bug:
https://bugzilla.redhat.com/show_bug.cgi?id=663933

The tap device changing automatically does seem new. I wonder if there is some newer code in the kernel that is detecting if the mac of the bridge is lower than the tap device and changing it automatically. Your best bet here is to explicitly set the mac of the mac address of one of the devices in the bridge, and then explictly set the mac of the bridge to that value:

https://backreference.org/2010/07/28/linux-bridge-mac-addresses-and-dynamic-ports/