Hello Kolla Ansible community,
We are experiencing difficulties with VM IP address assignment in our OpenStack deployment using Kolla Ansible 2024.1 on Rocky Linux 9.2. Here are the details of the issue:
1. Problem description: IP addresses are successfully assigned to VMs created on control nodes, but VMs created on compute-only nodes fail to receive IP addresses. This suggests that DHCP requests originating from the compute nodes are not reaching the DHCP
agents on the control nodes.
2. Error messages/logs: Console log of a Cirros VM on a compute node shows:
checking
http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 0.88. request failed
...
failed 20/20: up 54.89. request failed
failed to read iid from metadata. tried 20
failed to get instance-id of datasource
3. Configuration:
- 7 nodes in total: 5 control/compute/network nodes, 2 compute-only nodes
- Single bonded interface (bond0) with 3 VLANs (bond0.10, bond0.20, bond0.30)
- network_interface: bond0.10
- api_interface: bond0.20
- tunnel_interface: same as network_interface
- neutron_external_interface: bond0
- Provider network: VLAN 41
- ml2_conf.ini
[ml2_type_vlan]
network_vlan_ranges = physnet1:41:41
- 5 neutron DHCP agents, one on each control node
4. Troubleshooting steps taken:
- Verified br-tun OpenFlow rules are dropping packets
- Confirmed 5 DHCP agents are running, one on each control node
- Openvswitch containers: agent and
Here are the openvswitch open flow rules (dump) from the compute node (node16) for br-ex, br-in and br-tun
(openvswitch-vswitchd)# ovs-ofctl dump-flows br-ex
cookie=0xa9774500bf4a6912, duration=175366.856s, table=0, n_packets=3093, n_bytes=945104, priority=4,in_port="phy-br-ex",dl_vlan=1 actions=mod_vlan_vid:41,NORMAL
cookie=0xa9774500bf4a6912, duration=177050.619s, table=0, n_packets=2, n_bytes=176, priority=2,in_port="phy-br-ex" actions=drop
cookie=0xa9774500bf4a6912, duration=177050.621s, table=0, n_packets=8178, n_bytes=2363856, priority=0 actions=NORMAL
(openvswitch-vswitchd)# ovs-ofctl dump-flows br-int
cookie=0x98223299c10a59d3, duration=177059.641s, table=0, n_packets=0, n_bytes=0, priority=65535,dl_vlan=4095 actions=drop
cookie=0x98223299c10a59d3, duration=175373.945s, table=0, n_packets=123, n_bytes=9594, priority=10,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136 actions=resubmit(,24)
cookie=0x98223299c10a59d3, duration=175373.944s, table=0, n_packets=0, n_bytes=0, priority=10,arp,in_port="qvoab4fc3cb-a2" actions=resubmit(,24)
cookie=0x98223299c10a59d3, duration=177059.625s, table=0, n_packets=20, n_bytes=1668, priority=2,in_port="int-br-ex" actions=drop
cookie=0x98223299c10a59d3, duration=175373.946s, table=0, n_packets=3028, n_bytes=939610, priority=9,in_port="qvoab4fc3cb-a2" actions=resubmit(,25)
cookie=0x98223299c10a59d3, duration=175375.861s, table=0, n_packets=2256, n_bytes=219762, priority=3,in_port="int-br-ex",dl_vlan=41 actions=mod_vlan_vid:1,resubmit(,58)
cookie=0x98223299c10a59d3, duration=177059.643s, table=0, n_packets=2, n_bytes=176, priority=0 actions=resubmit(,58)
cookie=0x98223299c10a59d3, duration=177059.644s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x98223299c10a59d3, duration=175373.946s, table=24, n_packets=123, n_bytes=9594, priority=2,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136,nd_target=fe80::f816:3eff:fe4f:8020 actions=resubmit(,58)
cookie=0x98223299c10a59d3, duration=175373.945s, table=24, n_packets=0, n_bytes=0, priority=2,arp,in_port="qvoab4fc3cb-a2",arp_spa=141.52.56.110 actions=resubmit(,25)
cookie=0x98223299c10a59d3, duration=177059.641s, table=24, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x98223299c10a59d3, duration=175373.947s, table=25, n_packets=2970, n_bytes=935510, priority=2,in_port="qvoab4fc3cb-a2",dl_src=fa:16:3e:4f:80:20 actions=resubmit(,30)
cookie=0x98223299c10a59d3, duration=177059.640s, table=30, n_packets=2970, n_bytes=935510, priority=0 actions=resubmit(,58)
cookie=0x98223299c10a59d3, duration=177059.639s, table=31, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,58)
cookie=0x98223299c10a59d3, duration=177059.642s, table=58, n_packets=5351, n_bytes=1165042, priority=0 actions=resubmit(,60)
cookie=0x98223299c10a59d3, duration=177059.642s, table=60, n_packets=5351, n_bytes=1165042, priority=1 actions=NORMAL
cookie=0x98223299c10a59d3, duration=177059.640s, table=62, n_packets=0, n_bytes=0, priority=3 actions=NORMAL
(openvswitch-vswitchd)[root@os16 /]# ovs-ofctl dump-flows br-tun
cookie=0xad1cb3762184465d, duration=178072.704s, table=0, n_packets=4815, n_bytes=1114796, priority=1,in_port="patch-int" actions=resubmit(,2)
cookie=0xad1cb3762184465d, duration=178072.703s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0xad1cb3762184465d, duration=178072.703s, table=2, n_packets=1164, n_bytes=69852, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21)
cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=7, n_bytes=528, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=3644, n_bytes=1044416, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
cookie=0xad1cb3762184465d, duration=178072.701s, table=3, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0xad1cb3762184465d, duration=178072.701s, table=4, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0xad1cb3762184465d, duration=178072.701s, table=6, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0xad1cb3762184465d, duration=178072.700s, table=10, n_packets=0, n_bytes=0, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xad1cb3762184465d,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:"patch-int"
cookie=0xad1cb3762184465d, duration=178072.699s, table=20, n_packets=7, n_bytes=528, priority=0 actions=resubmit(,22)
cookie=0xad1cb3762184465d, duration=178072.699s, table=21, n_packets=1164, n_bytes=69852, priority=0 actions=resubmit(,22)
cookie=0xad1cb3762184465d, duration=178072.699s, table=22, n_packets=4815, n_bytes=1114796, priority=0 actions=drop
I would greatly appreciate any guidance on resolving this.
Best regards,
le3188