[Openstack] 99.5% of packets are disappearing somewhere between the Linux Bridge (brqxxxxzzzz-yy) and the tap (tapxxxxzzzz-yy).

Kevin Benton blak111 at gmail.com
Fri Jul 10 18:16:04 UTC 2015


Am I correct in understanding that the traffic you are expecting to see is
mirrored traffic not actually destined to the VM?

If so, linuxbridge is probably going to learn that the port all observed
macs belong to is the physical one so nothing will be flooded on the bridge.

Try disabling mac learning on that bridge with the follow command so
everything is flooded: brctl setageing <bridgename> 0

On Jul 10, 2015 1:01 AM, "Martinx - ジェームズ" <thiagocmartinsc at gmail.com>
wrote:

> Thank you James!
>
> Listen, trying to resume my topology for you, my Instance have two
> interfaces.
>
> 1- eth0 - its default gateway - vxlan with dhcp (okay);
> 2- eth1 - vlan provider network (the one that doesn't work) (eth3 if
> instance have 4 vNIC).
>
> I can access the Instance via ssh through its namespace router normally
> (or VNC Console).
>
> For "eth1", the physical switch port sends a mirrored tagged vlan traffic
> to it. My instance just (wants to) consumes that traffic (untagged)... My
> instance is a kind of NFV (in "offline" mode)...
>
> Do you still think that makes sense to capture the traffic simultaneously?
>
> You're right, "eth1" have no IP, it is just UP, reading packets...
>
> Thanks!
> Thiago
>
> On 10 July 2015 at 00:12, James Denton <james.denton at rackspace.com> wrote:
>
>>  Thanks, Thiago.
>>
>>
>>  Do you mind running a capture across the 3 interfaces (eth, bridge,
>> tap) simultaneously? In particular, traffic generated outside of the
>> node that demonstrates connection attempts to your instance. It will be
>> helpful to see if there are continuous ARP requests without replies, or a
>> reply and continuous TCP SYN packets and whatnot. On the tap interface you
>> should only expect to see broadcast, multicast, and unicast traffic to the
>> MAC address of the instance. Because the MAC addresses are masqueraded in
>> those captures, and they're not related, it's hard to tell what you're
>> seeing. Do you mind not masking them this time around?
>>
>>
>>  Also, what is the IP address of the instance? Seeing that this is an
>> all-in-one, I'm guessing you didn't having issues with DHCP?
>>
>>
>>  Thanks,
>>
>> James
>>
>>>>  ------------------------------
>> *From:* Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>> *Sent:* Thursday, July 9, 2015 8:51 PM
>> *To:* James Denton
>> *Cc:* openstack at lists.openstack.org
>> *Subject:* Re: [Openstack] 99.5% of packets are disappearing somewhere
>> between the Linux Bridge (brqxxxxzzzz-yy) and the tap (tapxxxxzzzz-yy).
>>
>>  Hello James!
>>
>> On 9 July 2015 at 11:17, James Denton <james.denton at rackspace.com> wrote:
>>
>>> Hi Thiago,
>>>
>>>    * I can see the untagged packets arriving at "brq50b13311-fa", by
>>> using "tcpdump -eni brq50b13311-fa";
>>>
>>>
>>>   Do you mind posting the packet capture from eth3 and the bridge on
>>> pastebin?
>>>
>>
>>
>>  I don't mind, I'll just replace the public IPs before posting (and
>> possibly MAC)...
>>
>>
>>  * Actual traffic hitting physical "eth3" with VLAN tag (OK):
>>
>>  http://paste.openstack.org/show/360214/
>>
>>
>>  * Actual traffic hitting "brq50b13311-fa" without tag (OK):
>>
>>  http://paste.openstack.org/show/360249/
>>
>>
>>  * Actual traffic hitting "tap9a546be0-d6" without tag (BUGGED - missing
>> packets):
>>
>>  http://paste.openstack.org/show/360274/
>>
>>
>>  * Actual traffic hitting vNIC "eth3" without tag (BUGGED - missing
>> packets):
>>
>>  http://paste.openstack.org/show/360275/
>>
>>
>>  *** Only PVST, OSPF and ICMP are appearing inside the Instance (and its
>> tap, of course) ***
>>
>>
>>
>>>
>>>    For example, I can not see the string "Cisco" while running "tcpdump
>>> -eni brq50b13311-fa | grep -i cisco", so, where those packets come from
>>> (that I'm seeing on tap9a546be0-d6 and within its instance - pastebin
>>> above) ???
>>>
>>>
>>>   Those are multicast packets for PVST and OSPF from the switch and
>>> router, respectively. You might try filtering by MAC on the bridge instead
>>> of using grep to isolate those packets:
>>>
>>>  tcpdump -eni brq50b13311-fa ether dst 01:00:0c:cc:cc:cd
>>>
>>>  I would expect to see those packets on eth3 as well.
>>>
>>
>>  You're absolutely right!
>>
>>  The PVST, OSPF (and very rare ICMP) are appearing @ eth3 too (with
>> "vlan XXXX" tagged), my bad (that grep, "ether dst" is much better, tks).
>>
>>  Look, inside the Instance - vNIC eth3:
>>
>>  tcpdump -eni eth3
>>
>>  http://paste.openstack.org/show/360127/
>>
>>
>> Only the PVST, OSPF and ICMP packets are hitting the tapxxxxzzzz-yy
>> interface! As expected, I can see those packets inside of the Instance as
>> well (Pastebin above).
>>
>>  Why TCP/UDP isn't passing?
>>
>>
>>>    * I CAN NOT see the untagged packets arriving at "tap9a546be0-d6",
>>> by using "tcpdump -eni tap9a546be0-d6"!
>>>
>>>
>>>  What do your security group rules look like?
>>>
>>
>>  I have no Security Groups, no Firewall, no ipset...
>>
>>
>>  ML2 configuration contains:
>>
>>  http://paste.openstack.org/show/356860/
>>
>>
>>
>>>
>>>    What is driving me crazy is that, on top of this very same setup
>>> (including e1000 driver), but with different vlan tag, it works!
>>>
>>>
>>>  Is it the same eth3 interface? You may want to avoid vlan 666, anyway.
>>> Never known those numbers to be lucky.
>>>
>>
>>  Yes, very same eth3.
>>
>>  LOL... I just posted this number here, to not publish private data,
>> actual VLAN ID is different.   :-P
>>
>>  Why it works for "VLAN X", but not for "VLAN Y", is a mystery for me.
>>
>>  Thank you so much for your help!
>>
>>  I'm seeing some debugging progress here...
>>
>>  Hopping to get this fixed! It is very important for the project that
>> I'm working on.
>>
>>
>>>
>>>  James
>>>
>>
>>  Thiago
>>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150710/7af5a163/attachment.html>


More information about the Openstack mailing list