0007460: NetworkManager bug after upgrading from Rocky Linux 8.8 to 8.9 - Rocky Linux BugTracker

ID	Project	Category	View Status	Date Submitted	Last Update

0007460	Rocky-Linux-8	NetworkManager	public	2024-07-16 09:23	2024-07-29 14:49

Reporter	Kevin Pemonon	Assigned To	Louis Abel
Priority	normal	Severity	major	Reproducibility	always
Status	needinfo	Resolution	open
OS	Rocky Linux 8.9	OS Version	8.9

Summary	0007460: NetworkManager bug after upgrading from Rocky Linux 8.8 to 8.9
Description	Hello, The purpose of this ticket is to report a change in the behavior of the NetworkManager. A change in NetworkManager behavior was observed when updating a VM running Rocky Linux 8.8 to version 8.9. Bonding network interfaces configured in LACP mode (== 802.3ad) no longer work after updating the NetworkManager rpm from version 1.40.16-4.el8_8 to 1.40.16-9.el8. The logs are filled with error messages: bond0: (slave eth1): failed to get link speed/duplex bond0: (slave eth2): failed to get link speed/duplex This new behavior seems to be specific to VMs using the virtio_net driver for network interfaces and LACP mode for bonding. Other modes (active-backup, balance-tlb, etc.) are not affected. Would you be able to correct this change in behavior? Thank you for your time and help. Best regards, Kevin Pemonon
Additional Information	I - VM configuration # cat /etc/system-release Rocky Linux release 8.8 (Green Obsidian) # uname -rs Linux 4.18.0-477.27.1.el8_8.x86_64 # cat /etc/sysconfig/network-scripts/ifcfg-eth1 TYPE=Ethernet NAME=eth1 UUID=e0be58e8-2415-4f32-8d42-d8f561871635 DEVICE=eth1 ONBOOT=yes MASTER=bond0 SLAVE=yes # cat /etc/sysconfig/network-scripts/ifcfg-eth2 TYPE=Ethernet NAME=eth2 UUID=eac7d775-156a-47e7-a43c-9da3786dc098 DEVICE=eth2 ONBOOT=yes MASTER=bond0 SLAVE=yes # cat /etc/sysconfig/network-scripts/ifcfg-bond0 BONDING_OPTS="mode=802.3ad miimon=100" TYPE=Bond BONDING_MASTER=yes PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none IPADDR=172.28.128.158 PREFIX=27 DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6_DISABLED=yes IPV6INIT=no NAME=bond0 UUID=c1475a17-e28a-4fb4-9cbb-27902a1e7485 DEVICE=bond0 ONBOOT=yes Please note: The VM uses the 'virtio_net' driver for the network interfaces, so the 'speed' and 'duplex' parameters are set to 'Unknown' (which is normal with this driver, see the RedHat knowledge base : https://access.redhat.com/solutions/7051532) # nmcli bond0: connecté à bond0 "bond0" bond, 02:00:66:68:A4:E0, sw, mtu 1500 inet4 172.28.128.158/27 route4 172.28.128.128/27 metric 301 eth1: connecté à eth1 "Red Hat Virtio" ethernet (virtio_net), 02:00:66:68:A4:E0, hw, mtu 1500 master bond0 eth2: connecté à eth2 "Red Hat Virtio" ethernet (virtio_net), 02:00:66:68:A4:E0, hw, mtu 1500 master bond0 # ethtool eth1 Settings for eth1: Supported ports: [ ] Supported link modes: Not reported Supported pause frame use: No Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: Unknown! Duplex: Unknown! (255) Auto-negotiation: off Port: Other PHYAD: 0 Transceiver: internal Link detected: yes # ethtool eth2 Settings for eth2: Supported ports: [ ] Supported link modes: Not reported Supported pause frame use: No Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: Unknown! Duplex: Unknown! (255) Auto-negotiation: off Port: Other PHYAD: 0 Transceiver: internal Link detected: yes ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- I.1 - With NetworkManager-1.40.16-4 : # rpm -q NetworkManager NetworkManager-1.40.16-4.el8_8.x86_64 Only the following messages are found in the logs: kernel: bond0: (slave eth1): Enslaving as a backup interface with an up link kernel: bond0: (slave eth2): Enslaving as a backup interface with an up link kernel: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond And the bond0 interface works: # ip a show bond0 8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 02:00:66:68:a4:e0 brd ff:ff:ff:ff:ff:ff inet 172.28.128.158/27 brd 172.28.128.159 scope global noprefixroute bond0 valid_lft forever preferred_lft forever # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Peer Notification Delay (ms): 0 802.3ad info LACP active: on LACP rate: slow Min links: 0 Aggregator selection policy (ad_select): stable System priority: 65535 System MAC address: 02:00:66:68:a4:e0 Active Aggregator Info: Aggregator ID: 1 Number of ports: 1 Actor Key: 0 Partner Key: 1 Partner Mac Address: 00:00:00:00:00:00 Slave Interface: eth1 MII Status: up Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 02:00:66:68:a4:e0 Slave queue ID: 0 Aggregator ID: 1 Actor Churn State: none Partner Churn State: churned Actor Churned Count: 0 Partner Churned Count: 1 details actor lacp pdu: system priority: 65535 system mac address: 02:00:66:68:a4:e0 port key: 0 port priority: 255 port number: 1 port state: 77 details partner lacp pdu: system priority: 65535 system mac address: 00:00:00:00:00:00 oper key: 1 port priority: 255 port number: 1 port state: 1 Slave Interface: eth2 MII Status: up Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 02:00:66:68:a4:e1 Slave queue ID: 0 Aggregator ID: 2 Actor Churn State: churned Partner Churn State: churned Actor Churned Count: 1 Partner Churned Count: 1 details actor lacp pdu: system priority: 65535 system mac address: 02:00:66:68:a4:e0 port key: 0 port priority: 255 port number: 2 port state: 69 details partner lacp pdu: system priority: 65535 system mac address: 00:00:00:00:00:00 oper key: 1 port priority: 255 port number: 1 port state: 1 # ping -c3 172.28.128.133 PING 172.28.128.133 (172.28.128.133) 56(84) bytes of data. 64 bytes from 172.28.128.133: icmp_seq=1 ttl=64 time=1.08 ms 64 bytes from 172.28.128.133: icmp_seq=2 ttl=64 time=0.459 ms 64 bytes from 172.28.128.133: icmp_seq=3 ttl=64 time=0.467 ms --- 172.28.128.133 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2036ms rtt min/avg/max/mdev = 0.459/0.667/1.076/0.289 ms ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- I.2 - With NetworkManager-1.40.16-4 : # rpm -q NetworkManager NetworkManager-1.40.16-9.el8.x86_64 # ip a show bond0 8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 02:00:66:68:a4:e0 brd ff:ff:ff:ff:ff:ff inet 172.28.128.158/27 brd 172.28.128.159 scope global noprefixroute bond0 valid_lft forever preferred_lft forever # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Peer Notification Delay (ms): 0 802.3ad info LACP active: on LACP rate: slow Min links: 0 Aggregator selection policy (ad_select): stable System priority: 65535 System MAC address: 02:00:66:68:a4:e0 Active Aggregator Info: Aggregator ID: 1 Number of ports: 1 Actor Key: 0 Partner Key: 1 Partner Mac Address: 00:00:00:00:00:00 Slave Interface: eth1 MII Status: down Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 02:00:66:68:a4:e0 Slave queue ID: 0 Aggregator ID: 1 Actor Churn State: none Partner Churn State: churned Actor Churned Count: 0 Partner Churned Count: 1 details actor lacp pdu: system priority: 65535 system mac address: 02:00:66:68:a4:e0 port key: 0 port priority: 255 port number: 1 port state: 77 details partner lacp pdu: system priority: 65535 system mac address: 00:00:00:00:00:00 oper key: 1 port priority: 255 port number: 1 port state: 1 Slave Interface: eth2 MII Status: down Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 02:00:66:68:a4:e1 Slave queue ID: 0 Aggregator ID: 2 Actor Churn State: churned Partner Churn State: churned Actor Churned Count: 1 Partner Churned Count: 1 details actor lacp pdu: system priority: 65535 system mac address: 02:00:66:68:a4:e0 port key: 0 port priority: 255 port number: 2 port state: 69 details partner lacp pdu: system priority: 65535 system mac address: 00:00:00:00:00:00 oper key: 1 port priority: 255 port number: 1 port state: 1 The bond0 interface does not work: # ping -c3 172.28.128.133 PING 172.28.128.133 (172.28.128.133) 56(84) bytes of data. From 172.28.128.158 icmp_seq=1 Destination Host Unreachable From 172.28.128.158 icmp_seq=2 Destination Host Unreachable From 172.28.128.158 icmp_seq=3 Destination Host Unreachable --- 172.28.128.133 ping statistics --- 3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2039ms pipe 3 And in the logs : kernel: bond0: (slave eth1): Enslaving as a backup interface with an up link kernel: bond0: (slave eth2): Enslaving as a backup interface with an up link kernel: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond kernel: bond0: (slave eth1): link status up again after 0 ms kernel: bond0: (slave eth2): link status up again after 0 ms kernel: bond0: (slave eth1): failed to get link speed/duplex kernel: bond0: (slave eth2): failed to get link speed/duplex kernel: bond0: (slave eth1): failed to get link speed/duplex kernel: bond0: (slave eth2): failed to get link speed/duplex kernel: bond0: (slave eth1): failed to get link speed/duplex kernel: bond0: (slave eth2): failed to get link speed/duplex ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- II - Workaround solution Note that there is a workaround which consists of configuring the parameters speed and duplex parameters with the ethtool command: # ethtool -s eth1 speed 1000 duplex full # ethtool -s eth2 speed 1000 duplex full In the logs (journalctl --dmesg): kernel: bond0: (slave eth1): link status definitely up, 1000 Mbps full duplex kernel: bond0: active interface up! kernel: bond0: (slave eth2): link status definitely up, 1000 Mbps full duplex # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Peer Notification Delay (ms): 0 802.3ad info LACP active: on LACP rate: slow Min links: 0 Aggregator selection policy (ad_select): stable System priority: 65535 System MAC address: 02:00:66:68:a4:e0 Active Aggregator Info: Aggregator ID: 1 Number of ports: 1 Actor Key: 9 Partner Key: 1 Partner Mac Address: 00:00:00:00:00:00 Slave Interface: eth1 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 02:00:66:68:a4:e0 Slave queue ID: 0 Aggregator ID: 1 Actor Churn State: none Partner Churn State: churned Actor Churned Count: 0 Partner Churned Count: 2 details actor lacp pdu: system priority: 65535 system mac address: 02:00:66:68:a4:e0 port key: 9 port priority: 255 port number: 1 port state: 77 details partner lacp pdu: system priority: 65535 system mac address: 00:00:00:00:00:00 oper key: 1 port priority: 255 port number: 1 port state: 1 Slave Interface: eth2 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 02:00:66:68:a4:e1 Slave queue ID: 0 Aggregator ID: 2 Actor Churn State: churned Partner Churn State: churned Actor Churned Count: 2 Partner Churned Count: 2 details actor lacp pdu: system priority: 65535 system mac address: 02:00:66:68:a4:e0 port key: 9 port priority: 255 port number: 2 port state: 69 details partner lacp pdu: system priority: 65535 system mac address: 00:00:00:00:00:00 oper key: 1 port priority: 255 port number: 1 port state: 1 # ping -c3 172.28.128.133 PING 172.28.128.133 (172.28.128.133) 56(84) bytes of data. 64 bytes from 172.28.128.133: icmp_seq=1 ttl=64 time=0.504 ms 64 bytes from 172.28.128.133: icmp_seq=2 ttl=64 time=0.343 ms 64 bytes from 172.28.128.133: icmp_seq=3 ttl=64 time=0.462 ms --- 172.28.128.133 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2056ms rtt min/avg/max/mdev = 0.343/0.436/0.504/0.070 ms
Tags	No tags attached.

Kevin Pemonon 2024-07-16 09:27 reporter ~0007889	Sorry, the title "I.2 - With NetworkManager-1.40.16-4" is wrong The good title is "I.2 - With NetworkManager-1.40.16-9 :"

Louis Abel 2024-07-16 15:04 administrator ~0007890	Rocky Linux 8.9 is end of life. You are recommended to fully update your system to get to what's current in 8.10. We will not be able to try to reproduce your issue until then. Setting to need info.

Kevin Pemonon 2024-07-17 07:38 reporter ~0007921	Hello, Thank you for your answer. We tried again on an OS running Rocky Linux 8.10 and using the version 1.40.16-15.el8_9 of NetworkManager installed and available by default, but we encountered the same problem. Could you help us with this?

Kevin Pemonon 2024-07-22 13:23 reporter ~0007987	Hello, For our information, has our request been processed?

Louis Abel 2024-07-25 16:32 administrator ~0008053	Can you provide information on how and where this VM is connecting? You've provided client information but you have not provided information as to where this VM is connecting or how the host is handling this VM's networking. Without that information, we cannot sufficiently reproduce your issue. It also seems rather unusual to be setting up LACP on a virtual machine, so it would also help to understand what this setup is for and why it's necessary. >kernel: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond This tells me the other end is not setup properly. But again, it's not clear how you have your environment setup and performing LACP in a virtual machine is not a normal thing to do. >kernel: bond0: (slave eth1): failed to get link speed/duplex >kernel: bond0: (slave eth2): failed to get link speed/duplex This is not behavior from Network Manager. As for your workaround: If manually setting the speed via ethtool is working, it is possible that setting it via nmcli would also work. Even so, it's still not clear how you have your environment setup and may speak to completely different issue altogether. ---- >For our information, has our request been processed? This is not a support venue. This is for reporting bugs and confirming the bugs in our upstreams (CentOS Stream and RHEL) if applicable. If you are seeking support, please use our mattermost or our forums. https://forums.rockylinux.org If you are sure this is a bug, please provide applicable information as requested above.

Jean-Luc HAVY 2024-07-29 14:49 reporter ~0008086	This VM is on a Linux KVM (+ openvswitch) Hypervisor and the bond0 interface is used to mount a NFS share on another VM which acts as a NFS server and runs on a CentOS 7.9. NFS Server : - CentOS Linux release 7.9.2009 (Core) - Linux 3.10.0-1160.76.1.el7.x86_64 - no bonding on this side : # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 02:00:66:68:a4:d6 brd ff:ff:ff:ff:ff:ff inet 172.28.128.133/27 brd 172.28.128.159 scope global eth1 valid_lft forever preferred_lft forever The setup is unusual because the VM where i first encountered this problem is used as a testbed in a pre-prod environment and the configuration has to be close enough (but not match completly) to that of a physical server, hence the LACP config of the bond interface. I have tried an succeeded in reproducing this change of behavior of NetwokManager on another VM built from scratch on a RockyLinux 8.8 updated to RockyLinux 8.9 then to 8.10, where i have kept just the necessary configuration. The original VM was built on RockyLinux 8.4 and has been updated regularly and was working fine until the last update to 8.9. Those messages : > >kernel: bond0: (slave eth1): failed to get link speed/duplex are not present in the logs when NetworkManager-1.40.16-4 is installed and should be expected because the virtio_net driver (used by all the network interfaces) does not advertised any value for these parameters (https://access.redhat.com/solutions/7051532). And that message : > kernel: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond also appears in the logs of Rocky Linux 8.8 with NetworkManager-1.40.16-4 but everything works as expected. What other information do you need ?

Date Modified	Username	Field	Change
2024-07-16 09:23	Kevin Pemonon	New Issue
2024-07-16 09:27	Kevin Pemonon	Note Added: 0007889
2024-07-16 15:04	Louis Abel	Assigned To	=> Louis Abel
2024-07-16 15:04	Louis Abel	Status	new => needinfo
2024-07-16 15:04	Louis Abel	Note Added: 0007890
2024-07-17 07:38	Kevin Pemonon	Note Added: 0007921
2024-07-22 13:23	Kevin Pemonon	Note Added: 0007987
2024-07-25 16:32	Louis Abel	Note Added: 0008053
2024-07-29 14:49	Jean-Luc HAVY	Note Added: 0008086