Open Stack

The issue was related to SRIOV, but not sure why exactly. After disabling
all VFs things started to work like normal again. I was using the PF as
trunk for VM with OVS + vlan, weird is that it worked for some compute
nodes.

The command I used to disable all VFs: echo '0' >
/sys/class/net/eth3/device/sriov_numvfs

On Tue, Sep 22, 2015 at 4:47 PM, Nasir Mahmood <nasir.mahmood at gmail.com>
wrote:

> try this
>
> http://2014.texaslinuxfest.org/sites/default/files/HopkinsPPTdeck.pdf
>
> On Tue, Sep 22, 2015 at 1:16 PM, Sam Stoelinga <sammiestoel at gmail.com>
> wrote:
>
>> Hi all,
>>
>> Would appreciate your networking expertise.
>>
>> I have a 7 node environment, 3 controllers which run also as network node
>> and 4 compute nodes. There are 2 compute nodes which are behaving correctly
>> both sending and receiving packets, but the other 2 compute nodes can sent
>> broadcasts such as ARP request and DHCP requests but the responses is not
>> received. Also note that previously this was working but out of nowhere
>> this issue started happening.
>>
>> I've debugged this using tcpdump on both the compute node and the
>> controller node. The VM's DHCP request gets successfully sent to the
>> controller node and the controller node responds with a dhcp response. It
>> also get's successfully sent out of eth3 (vlan trunk), but this dhcp
>> response never arrives on the compute node. So then I tried to login via
>> VNC and manually set the IP and then try pinging, the result was outgoing
>> arp request seen by all nodes but the arp response was not received by the
>> compute node.
>>
>> I've tried disabling hardware offloading as I thought this may be the nic
>> discarding packets, but that didn't help. I have spent about a day
>> debugging with tcpdump but running out of clueues. Anybody with a similar
>> experience? It's weird that the node can sent out packets seen by other
>> nodes but it can not receive the responses from other nodes.
>>
>> tcpdump on compute which isn't receiving packets
>> tcpdump -i eth3 -nnNs 512
>> tcpdump: WARNING: eth3: no IPv4 address assigned
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on eth3, link-type EN10MB (Ethernet), capture size 512 bytes
>> 08:02:02.232749 STP 802.1s, Rapid STP, CIST Flags [Learn, Forward,
>> Agreement], length 102
>> 08:02:03.990588 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request
>> from fa:16:3e:0d:26:a5 (this is a VM), length 300
>> 08:02:04.232797 STP 802.1s, Rapid STP, CIST Flags [Learn, Forward,
>> Agreement], length 102
>> 08:02:11.393997 LLDP, length 318: HP
>> 08:02:12.232627 STP 802.1s, Rapid STP, CIST Flags [Learn, Forward,
>> Agreement], length 102
>> 08:02:13.118051 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request
>> from fa:16:3e:0d:26:a5, length 300
>> 08:02:14.232638 STP 802.1s, Rapid STP, CIST Flags [Learn, Forward,
>> Agreement], length 102
>>
>> tcpdump on controller
>> ip netns exec qdhcp-878e1f0a-abba-4637-8afd-2814a38136a5 tcpdump -i
>> tape2f5fa09-19 -nnNs 512
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on tape2f5fa09-19, link-type EN10MB (Ethernet), capture size
>> 512 bytes
>> 08:01:32.591906 ARP, Request who-has 192.168.111.122 tell 192.168.111.3,
>> length 28
>> 08:01:33.033029 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request
>> from fa:16:3e:0d:26:a5, length 300
>> 08:01:33.033327 IP 192.168.111.3.67 > 192.168.111.122.68: BOOTP/DHCP,
>> Reply, length 328
>> 08:01:38.048136 ARP, Request who-has 192.168.111.122 tell 192.168.111.3,
>> length 28
>> 08:01:39.051897 ARP, Request who-has 192.168.111.122 tell 192.168.111.3,
>> length 28
>> 08:01:40.051901 ARP, Request who-has 192.168.111.122 tell 192.168.111.3,
>> length 28
>> 08:01:45.168519 ARP, Request who-has 192.168.111.138 tell
>> 192.168.111.138, length 46
>> 08:01:45.168546 ARP, Request who-has 192.168.111.138 tell
>> 192.168.111.138, length 46
>> 08:01:46.962341 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request
>> from fa:16:3e:0d:26:a5, length 300
>> 08:01:46.962584 IP 192.168.111.3.67 > 192.168.111.122.68: BOOTP/DHCP,
>> Reply, length 328
>> 08:01:49.438756 ARP, Request who-has 192.168.111.138 tell
>> 192.168.111.138, length 46
>> 08:01:49.438780 ARP, Request who-has 192.168.111.138 tell
>> 192.168.111.138, length 46
>> 08:01:51.346561 ARP, Request who-has 192.168.111.112 tell
>> 192.168.111.136, length 46
>> 08:01:51.967905 ARP, Request who-has 192.168.111.122 tell 192.168.111.3,
>> length 28
>>
>> Thanks,
>> Sam Stoelinga
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>
>
> --
> Nasir Mahmood
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150922/92833266/attachment.html>