Open Stack

Hello,

    Do you know the L2 switch model you are using in the blade center and
how much traffic you are pushing through?

If you have netflow or some other type of traffic analysis I would look for
changes and sudden increases around this time:
- difference in packets per second
- difference in bytes per second

I have seen something similar in the past under two conditions:
1. When a switch buffers have been overloaded due to excessive UDP traffic,
the switch ended up sending out the data on all ports.
2. Someone created a switching loop by plugging in both ends of the cable
into a switch with spanning tree disabled.

Michael




On Tue, Oct 14, 2014 at 9:25 PM, Don Waterloo <don.waterloo at gmail.com>
wrote:

> On 14 October 2014 22:45, Don Waterloo <don.waterloo at gmail.com> wrote:
>
>> I  have a system w/ 6 blades interconnected on L2. 1 controller, 5
>> compute/network nodes.
>> they are using icehouse, neutron, ovs, with vxlan on ubuntu 14.04.
>>
>> Once in a while, one of the compute blades will declare a bunch of lines
>> like this:
>>
>> 129749.691102] qbrfc96cb3f-f3: received packet on qvbfc96cb3f-f3 with own
>> address as source address
>> [129749.691112] qbrcdd85970-14: received packet on qvbcdd85970-14 with
>> own address as source address
>> [129749.691471] qbr8e0cc5fd-17: received packet on qvb8e0cc5fd-17 with
>> own address as source address
>> [129749.691620] qbr270b38d8-4d: received packet on qvb270b38d8-4d with
>> own address as source address
>>
>> in its dmesg, and during that time (1-2 seconds usually), packets cease
>> to be forwarded from the controller (the l3 agent).
>>
>> what could be causing this packet storm? of 'own address' packets? any
>> suggestions on how to debug? there's a lot of traffic flying around, so
>> packet captures need to be tactical.
>>
>> if i do a tcpdump for its src address, eg.
>> tcpdump -i qvbcdd85970-14 ether src f2:4b:2d:1f:e0:9b
>> i'm not seeing it.
>>
>>
>> I am also (and certainly related) getting this problem:
> IPv4: martian source 10.129.247.4 from 10.129.247.5, on dev br-ex
> ll header: 00000000: ff ff ff ff ff ff fa 16 3e e7 97 7a 08 06
>
> on the compute nodes running neutron ovs agent.
>
> if i hunt down the mac address in my port-list, it is my 'ext-net', the
> network that is used to reach my physical network. its created as:
>
> neutron net-create  --shared --router:external=True ext-net
> neutron subnet-create  --name ext-subnet \
>     --host-route destination=$CONTROL_IP/32,nexthop=$DATA_IP \
>     --dns-nameserver $DATA_IP \
>     --disable-dhcp \
>     ext-net $BR_EX_IP
>
> so i have to believe this is related.
>
> the blade complaining about this is 247.5, and all of the others (247.4,
> 247.3 etc) are present in the log.
>
> so if i read this correctly, a broadcast from my 'ext-net' port is
> arriving with the src-ip/dest-ip filled in
>
> so:
> src-mac: ext-net
> dst-mac: all 1 broadcast
> src-ip: slot X
> dst-ip: slot 4
>
>
> ?
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>


-- 
“We, the unwilling, led by the unknowing, are doing the impossible for the
ungrateful. We have done so much, for so long, with so little, we are now
qualified to do anything with nothing.”

― Konstantin Josef Jireček
<https://www.goodreads.com/author/show/4666841.Konstantin_Josef_Jire_ek>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20141014/62b91a33/attachment.html>