Open Stack

Hi all,

Would appreciate your networking expertise.

I have a 7 node environment, 3 controllers which run also as network node
and 4 compute nodes. There are 2 compute nodes which are behaving correctly
both sending and receiving packets, but the other 2 compute nodes can sent
broadcasts such as ARP request and DHCP requests but the responses is not
received. Also note that previously this was working but out of nowhere
this issue started happening.

I've debugged this using tcpdump on both the compute node and the
controller node. The VM's DHCP request gets successfully sent to the
controller node and the controller node responds with a dhcp response. It
also get's successfully sent out of eth3 (vlan trunk), but this dhcp
response never arrives on the compute node. So then I tried to login via
VNC and manually set the IP and then try pinging, the result was outgoing
arp request seen by all nodes but the arp response was not received by the
compute node.

I've tried disabling hardware offloading as I thought this may be the nic
discarding packets, but that didn't help. I have spent about a day
debugging with tcpdump but running out of clueues. Anybody with a similar
experience? It's weird that the node can sent out packets seen by other
nodes but it can not receive the responses from other nodes.

tcpdump on compute which isn't receiving packets
tcpdump -i eth3 -nnNs 512
tcpdump: WARNING: eth3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), capture size 512 bytes
08:02:02.232749 STP 802.1s, Rapid STP, CIST Flags [Learn, Forward,
Agreement], length 102
08:02:03.990588 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request
from fa:16:3e:0d:26:a5 (this is a VM), length 300
08:02:04.232797 STP 802.1s, Rapid STP, CIST Flags [Learn, Forward,
Agreement], length 102
08:02:11.393997 LLDP, length 318: HP
08:02:12.232627 STP 802.1s, Rapid STP, CIST Flags [Learn, Forward,
Agreement], length 102
08:02:13.118051 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request
from fa:16:3e:0d:26:a5, length 300
08:02:14.232638 STP 802.1s, Rapid STP, CIST Flags [Learn, Forward,
Agreement], length 102

tcpdump on controller
ip netns exec qdhcp-878e1f0a-abba-4637-8afd-2814a38136a5 tcpdump -i
tape2f5fa09-19 -nnNs 512
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tape2f5fa09-19, link-type EN10MB (Ethernet), capture size 512
bytes
08:01:32.591906 ARP, Request who-has 192.168.111.122 tell 192.168.111.3,
length 28
08:01:33.033029 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request
from fa:16:3e:0d:26:a5, length 300
08:01:33.033327 IP 192.168.111.3.67 > 192.168.111.122.68: BOOTP/DHCP,
Reply, length 328
08:01:38.048136 ARP, Request who-has 192.168.111.122 tell 192.168.111.3,
length 28
08:01:39.051897 ARP, Request who-has 192.168.111.122 tell 192.168.111.3,
length 28
08:01:40.051901 ARP, Request who-has 192.168.111.122 tell 192.168.111.3,
length 28
08:01:45.168519 ARP, Request who-has 192.168.111.138 tell 192.168.111.138,
length 46
08:01:45.168546 ARP, Request who-has 192.168.111.138 tell 192.168.111.138,
length 46
08:01:46.962341 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request
from fa:16:3e:0d:26:a5, length 300
08:01:46.962584 IP 192.168.111.3.67 > 192.168.111.122.68: BOOTP/DHCP,
Reply, length 328
08:01:49.438756 ARP, Request who-has 192.168.111.138 tell 192.168.111.138,
length 46
08:01:49.438780 ARP, Request who-has 192.168.111.138 tell 192.168.111.138,
length 46
08:01:51.346561 ARP, Request who-has 192.168.111.112 tell 192.168.111.136,
length 46
08:01:51.967905 ARP, Request who-has 192.168.111.122 tell 192.168.111.3,
length 28

Thanks,
Sam Stoelinga
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150922/f0ce3bfc/attachment.html>