[Openstack] 99.5% of packets are disappearing somewhere between the Linux Bridge (brqxxxxzzzz-yy) and the tap (tapxxxxzzzz-yy).

Kevin Benton blak111 at gmail.com
Fri Jul 10 18:54:11 UTC 2015


You should be able to confirm this is the issue by running brctl showmacs
<bridgename>. If the mac addresses is learned in that table on the physical
interface, any traffic to that mac wouldn't be flooded to your VM.
On Jul 10, 2015 12:16 PM, "Kevin Benton" <blak111 at gmail.com> wrote:

> Am I correct in understanding that the traffic you are expecting to see is
> mirrored traffic not actually destined to the VM?
>
> If so, linuxbridge is probably going to learn that the port all observed
> macs belong to is the physical one so nothing will be flooded on the bridge.
>
> Try disabling mac learning on that bridge with the follow command so
> everything is flooded: brctl setageing <bridgename> 0
>
> On Jul 10, 2015 1:01 AM, "Martinx - ジェームズ" <thiagocmartinsc at gmail.com>
> wrote:
>
>> Thank you James!
>>
>> Listen, trying to resume my topology for you, my Instance have two
>> interfaces.
>>
>> 1- eth0 - its default gateway - vxlan with dhcp (okay);
>> 2- eth1 - vlan provider network (the one that doesn't work) (eth3 if
>> instance have 4 vNIC).
>>
>> I can access the Instance via ssh through its namespace router normally
>> (or VNC Console).
>>
>> For "eth1", the physical switch port sends a mirrored tagged vlan traffic
>> to it. My instance just (wants to) consumes that traffic (untagged)... My
>> instance is a kind of NFV (in "offline" mode)...
>>
>> Do you still think that makes sense to capture the traffic simultaneously?
>>
>> You're right, "eth1" have no IP, it is just UP, reading packets...
>>
>> Thanks!
>> Thiago
>>
>> On 10 July 2015 at 00:12, James Denton <james.denton at rackspace.com>
>> wrote:
>>
>>>  Thanks, Thiago.
>>>
>>>
>>>  Do you mind running a capture across the 3 interfaces (eth, bridge,
>>> tap) simultaneously? In particular, traffic generated outside of the
>>> node that demonstrates connection attempts to your instance. It will be
>>> helpful to see if there are continuous ARP requests without replies, or a
>>> reply and continuous TCP SYN packets and whatnot. On the tap interface you
>>> should only expect to see broadcast, multicast, and unicast traffic to the
>>> MAC address of the instance. Because the MAC addresses are masqueraded in
>>> those captures, and they're not related, it's hard to tell what you're
>>> seeing. Do you mind not masking them this time around?
>>>
>>>
>>>  Also, what is the IP address of the instance? Seeing that this is an
>>> all-in-one, I'm guessing you didn't having issues with DHCP?
>>>
>>>
>>>  Thanks,
>>>
>>> James
>>>
>>>>>>  ------------------------------
>>> *From:* Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>>> *Sent:* Thursday, July 9, 2015 8:51 PM
>>> *To:* James Denton
>>> *Cc:* openstack at lists.openstack.org
>>> *Subject:* Re: [Openstack] 99.5% of packets are disappearing somewhere
>>> between the Linux Bridge (brqxxxxzzzz-yy) and the tap (tapxxxxzzzz-yy).
>>>
>>>  Hello James!
>>>
>>> On 9 July 2015 at 11:17, James Denton <james.denton at rackspace.com>
>>> wrote:
>>>
>>>> Hi Thiago,
>>>>
>>>>    * I can see the untagged packets arriving at "brq50b13311-fa", by
>>>> using "tcpdump -eni brq50b13311-fa";
>>>>
>>>>
>>>>   Do you mind posting the packet capture from eth3 and the bridge on
>>>> pastebin?
>>>>
>>>
>>>
>>>  I don't mind, I'll just replace the public IPs before posting (and
>>> possibly MAC)...
>>>
>>>
>>>  * Actual traffic hitting physical "eth3" with VLAN tag (OK):
>>>
>>>  http://paste.openstack.org/show/360214/
>>>
>>>
>>>  * Actual traffic hitting "brq50b13311-fa" without tag (OK):
>>>
>>>  http://paste.openstack.org/show/360249/
>>>
>>>
>>>  * Actual traffic hitting "tap9a546be0-d6" without tag (BUGGED -
>>> missing packets):
>>>
>>>  http://paste.openstack.org/show/360274/
>>>
>>>
>>>  * Actual traffic hitting vNIC "eth3" without tag (BUGGED - missing
>>> packets):
>>>
>>>  http://paste.openstack.org/show/360275/
>>>
>>>
>>>  *** Only PVST, OSPF and ICMP are appearing inside the Instance (and
>>> its tap, of course) ***
>>>
>>>
>>>
>>>>
>>>>    For example, I can not see the string "Cisco" while running
>>>> "tcpdump -eni brq50b13311-fa | grep -i cisco", so, where those packets come
>>>> from (that I'm seeing on tap9a546be0-d6 and within its instance - pastebin
>>>> above) ???
>>>>
>>>>
>>>>   Those are multicast packets for PVST and OSPF from the switch and
>>>> router, respectively. You might try filtering by MAC on the bridge instead
>>>> of using grep to isolate those packets:
>>>>
>>>>  tcpdump -eni brq50b13311-fa ether dst 01:00:0c:cc:cc:cd
>>>>
>>>>  I would expect to see those packets on eth3 as well.
>>>>
>>>
>>>  You're absolutely right!
>>>
>>>  The PVST, OSPF (and very rare ICMP) are appearing @ eth3 too (with
>>> "vlan XXXX" tagged), my bad (that grep, "ether dst" is much better, tks).
>>>
>>>  Look, inside the Instance - vNIC eth3:
>>>
>>>  tcpdump -eni eth3
>>>
>>>  http://paste.openstack.org/show/360127/
>>>
>>>
>>> Only the PVST, OSPF and ICMP packets are hitting the tapxxxxzzzz-yy
>>> interface! As expected, I can see those packets inside of the Instance as
>>> well (Pastebin above).
>>>
>>>  Why TCP/UDP isn't passing?
>>>
>>>
>>>>    * I CAN NOT see the untagged packets arriving at "tap9a546be0-d6",
>>>> by using "tcpdump -eni tap9a546be0-d6"!
>>>>
>>>>
>>>>  What do your security group rules look like?
>>>>
>>>
>>>  I have no Security Groups, no Firewall, no ipset...
>>>
>>>
>>>  ML2 configuration contains:
>>>
>>>  http://paste.openstack.org/show/356860/
>>>
>>>
>>>
>>>>
>>>>    What is driving me crazy is that, on top of this very same setup
>>>> (including e1000 driver), but with different vlan tag, it works!
>>>>
>>>>
>>>>  Is it the same eth3 interface? You may want to avoid vlan 666,
>>>> anyway. Never known those numbers to be lucky.
>>>>
>>>
>>>  Yes, very same eth3.
>>>
>>>  LOL... I just posted this number here, to not publish private data,
>>> actual VLAN ID is different.   :-P
>>>
>>>  Why it works for "VLAN X", but not for "VLAN Y", is a mystery for me.
>>>
>>>  Thank you so much for your help!
>>>
>>>  I'm seeing some debugging progress here...
>>>
>>>  Hopping to get this fixed! It is very important for the project that
>>> I'm working on.
>>>
>>>
>>>>
>>>>  James
>>>>
>>>
>>>  Thiago
>>>
>>
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150710/36b30041/attachment.html>


More information about the Openstack mailing list