<div dir="ltr">Hello,<div><br></div><div> Do you know the L2 switch model you are using in the blade center and how much traffic you are pushing through?</div><div><br></div><div>If you have netflow or some other type of traffic analysis I would look for changes and sudden increases around this time:</div><div>- difference in packets per second</div><div>- difference in bytes per second</div><div><br></div><div>I have seen something similar in the past under two conditions:</div><div>1. When a switch buffers have been overloaded due to excessive UDP traffic, the switch ended up sending out the data on all ports.</div><div>2. Someone created a switching loop by plugging in both ends of the cable into a switch with spanning tree disabled.</div><div><br></div><div>Michael</div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 14, 2014 at 9:25 PM, Don Waterloo <span dir="ltr"><<a href="mailto:don.waterloo@gmail.com" target="_blank">don.waterloo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div><div class="h5"><div class="gmail_quote">On 14 October 2014 22:45, Don Waterloo <span dir="ltr"><<a href="mailto:don.waterloo@gmail.com" target="_blank">don.waterloo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>I have a system w/ 6 blades interconnected on L2. 1 controller, 5 compute/network nodes.</div><div>they are using icehouse, neutron, ovs, with vxlan on ubuntu 14.04.</div><div><br></div><div>Once in a while, one of the compute blades will declare a bunch of lines like this:</div><div><br></div><div>129749.691102] qbrfc96cb3f-f3: received packet on qvbfc96cb3f-f3 with own address as source address</div><div>[129749.691112] qbrcdd85970-14: received packet on qvbcdd85970-14 with own address as source address</div><div>[129749.691471] qbr8e0cc5fd-17: received packet on qvb8e0cc5fd-17 with own address as source address</div><div>[129749.691620] qbr270b38d8-4d: received packet on qvb270b38d8-4d with own address as source address</div><div><br></div><div>in its dmesg, and during that time (1-2 seconds usually), packets cease to be forwarded from the controller (the l3 agent).</div><div><br></div><div>what could be causing this packet storm? of 'own address' packets? any suggestions on how to debug? there's a lot of traffic flying around, so packet captures need to be tactical.</div><div><br></div><div>if i do a tcpdump for its src address, eg.</div><div><div>tcpdump -i qvbcdd85970-14 ether src f2:4b:2d:1f:e0:9b </div></div><div>i'm not seeing it.</div><div><br></div><div><br></div></div>
</blockquote></div></div></div>I am also (and certainly related) getting this problem:</div><div class="gmail_extra">IPv4: martian source 10.129.247.4 from 10.129.247.5, on dev br-ex<br></div><div class="gmail_extra">ll header: 00000000: ff ff ff ff ff ff fa 16 3e e7 97 7a 08 06<br></div><div class="gmail_extra"><br></div><div class="gmail_extra">on the compute nodes running neutron ovs agent.</div><div class="gmail_extra"><br></div><div class="gmail_extra">if i hunt down the mac address in my port-list, it is my 'ext-net', the network that is used to reach my physical network. its created as:</div><div class="gmail_extra"><br></div><div class="gmail_extra">neutron net-create --shared --router:external=True ext-net<br></div><div class="gmail_extra"><div class="gmail_extra">neutron subnet-create --name ext-subnet \</div><div class="gmail_extra"> --host-route destination=$CONTROL_IP/32,nexthop=$DATA_IP \</div><div class="gmail_extra"> --dns-nameserver $DATA_IP \</div><div class="gmail_extra"> --disable-dhcp \</div><div class="gmail_extra"> ext-net $BR_EX_IP</div><div><br></div><div>so i have to believe this is related.</div><div><br></div><div>the blade complaining about this is 247.5, and all of the others (247.4, 247.3 etc) are present in the log.</div><div><br></div><div>so if i read this correctly, a broadcast from my 'ext-net' port is arriving with the src-ip/dest-ip filled in</div><div><br></div><div>so:</div><div>src-mac: ext-net</div><div>dst-mac: all 1 broadcast</div><div>src-ip: slot X</div><div>dst-ip: slot 4</div><div><br></div><div><br></div><div>?</div><div><br></div></div><div class="gmail_extra"><br></div></div>
<br>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><span style="font-size:16px"><h2 style="font-family:Verdana,Arial,Helvetica,sans-serif;font-weight:bold;font-size:16px"><span style="font-family:verdana,sans-serif;color:rgb(24,24,24);font-size:14px;font-weight:normal;line-height:18px">“We, the unwilling, led by the unknowing, are doing the impossible for the ungrateful. We have done so much, for so long, with so little, we are now qualified to do anything with nothing.”</span><br></h2><div><p><font face="verdana, sans-serif"><span style="color:rgb(24,24,24);font-size:14px;line-height:18px">― </span><a href="https://www.goodreads.com/author/show/4666841.Konstantin_Josef_Jire_ek" style="color:rgb(102,102,0);text-decoration:none;font-size:14px;line-height:18px" target="_blank">Konstantin Josef Jireček</a></font><br></p></div></span></div>
</div>