[Openstack] 99.5% of packets are disappearing somewhere between the Linux Bridge (brqxxxxzzzz-yy) and the tap (tapxxxxzzzz-yy).

Kevin Benton blak111 at gmail.com
Fri Jul 10 19:01:05 UTC 2015


> Now, how to configure "setageing" via Neutron ML2?

I don't think we have a nice place to put a hook for this right now. Feel
free to file a request for enhancement bug (regular bug process with the
tag 'rfe' and the use case in the body) and maybe we can get it addressed
in a future version.

Cheers
On Jul 10, 2015 12:56 PM, "Martinx - ジェームズ" <thiagocmartinsc at gmail.com>
wrote:

> Hello Kevin!
>
> First of all... THANK YOU SO MUCH!!!   :-D
>
> On 10 July 2015 at 15:16, Kevin Benton <blak111 at gmail.com> wrote:
>
>> Am I correct in understanding that the traffic you are expecting to see
>> is mirrored traffic not actually destined to the VM?
>>
> Yes.
>
>
>>  If so, linuxbridge is probably going to learn that the port all
>> observed macs belong to is the physical one so nothing will be flooded on
>> the bridge.
>>
> Mmmm...
>
>
>> Try disabling mac learning on that bridge with the follow command so
>> everything is flooded: brctl setageing <bridgename> 0
>>
>
> Perfect! It worked like a charm!
>
> Now, how to configure "setageing" via Neutron ML2?     :-)
>
> Thank you again!
>
>>
>> On Jul 10, 2015 1:01 AM, "Martinx - ジェームズ" <thiagocmartinsc at gmail.com>
>> wrote:
>>
>>> Thank you James!
>>>
>>> Listen, trying to resume my topology for you, my Instance have two
>>> interfaces.
>>>
>>> 1- eth0 - its default gateway - vxlan with dhcp (okay);
>>> 2- eth1 - vlan provider network (the one that doesn't work) (eth3 if
>>> instance have 4 vNIC).
>>>
>>> I can access the Instance via ssh through its namespace router normally
>>> (or VNC Console).
>>>
>>> For "eth1", the physical switch port sends a mirrored tagged vlan
>>> traffic to it. My instance just (wants to) consumes that traffic
>>> (untagged)... My instance is a kind of NFV (in "offline" mode)...
>>>
>>> Do you still think that makes sense to capture the traffic
>>> simultaneously?
>>>
>>> You're right, "eth1" have no IP, it is just UP, reading packets...
>>>
>>> Thanks!
>>> Thiago
>>>
>>> On 10 July 2015 at 00:12, James Denton <james.denton at rackspace.com>
>>> wrote:
>>>
>>>>  Thanks, Thiago.
>>>>
>>>>
>>>>  Do you mind running a capture across the 3 interfaces (eth, bridge,
>>>> tap) simultaneously? In particular, traffic generated outside of the
>>>> node that demonstrates connection attempts to your instance. It will be
>>>> helpful to see if there are continuous ARP requests without replies, or a
>>>> reply and continuous TCP SYN packets and whatnot. On the tap interface you
>>>> should only expect to see broadcast, multicast, and unicast traffic to the
>>>> MAC address of the instance. Because the MAC addresses are masqueraded in
>>>> those captures, and they're not related, it's hard to tell what you're
>>>> seeing. Do you mind not masking them this time around?
>>>>
>>>>
>>>>  Also, what is the IP address of the instance? Seeing that this is an
>>>> all-in-one, I'm guessing you didn't having issues with DHCP?
>>>>
>>>>
>>>>  Thanks,
>>>>
>>>> James
>>>>
>>>>>>>>  ------------------------------
>>>> *From:* Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>>>> *Sent:* Thursday, July 9, 2015 8:51 PM
>>>> *To:* James Denton
>>>> *Cc:* openstack at lists.openstack.org
>>>> *Subject:* Re: [Openstack] 99.5% of packets are disappearing somewhere
>>>> between the Linux Bridge (brqxxxxzzzz-yy) and the tap (tapxxxxzzzz-yy).
>>>>
>>>>  Hello James!
>>>>
>>>> On 9 July 2015 at 11:17, James Denton <james.denton at rackspace.com>
>>>> wrote:
>>>>
>>>>> Hi Thiago,
>>>>>
>>>>>    * I can see the untagged packets arriving at "brq50b13311-fa", by
>>>>> using "tcpdump -eni brq50b13311-fa";
>>>>>
>>>>>
>>>>>   Do you mind posting the packet capture from eth3 and the bridge on
>>>>> pastebin?
>>>>>
>>>>
>>>>
>>>>  I don't mind, I'll just replace the public IPs before posting (and
>>>> possibly MAC)...
>>>>
>>>>
>>>>  * Actual traffic hitting physical "eth3" with VLAN tag (OK):
>>>>
>>>>  http://paste.openstack.org/show/360214/
>>>>
>>>>
>>>>  * Actual traffic hitting "brq50b13311-fa" without tag (OK):
>>>>
>>>>  http://paste.openstack.org/show/360249/
>>>>
>>>>
>>>>  * Actual traffic hitting "tap9a546be0-d6" without tag (BUGGED -
>>>> missing packets):
>>>>
>>>>  http://paste.openstack.org/show/360274/
>>>>
>>>>
>>>>  * Actual traffic hitting vNIC "eth3" without tag (BUGGED - missing
>>>> packets):
>>>>
>>>>  http://paste.openstack.org/show/360275/
>>>>
>>>>
>>>>  *** Only PVST, OSPF and ICMP are appearing inside the Instance (and
>>>> its tap, of course) ***
>>>>
>>>>
>>>>
>>>>>
>>>>>    For example, I can not see the string "Cisco" while running
>>>>> "tcpdump -eni brq50b13311-fa | grep -i cisco", so, where those packets come
>>>>> from (that I'm seeing on tap9a546be0-d6 and within its instance - pastebin
>>>>> above) ???
>>>>>
>>>>>
>>>>>   Those are multicast packets for PVST and OSPF from the switch and
>>>>> router, respectively. You might try filtering by MAC on the bridge instead
>>>>> of using grep to isolate those packets:
>>>>>
>>>>>  tcpdump -eni brq50b13311-fa ether dst 01:00:0c:cc:cc:cd
>>>>>
>>>>>  I would expect to see those packets on eth3 as well.
>>>>>
>>>>
>>>>  You're absolutely right!
>>>>
>>>>  The PVST, OSPF (and very rare ICMP) are appearing @ eth3 too (with
>>>> "vlan XXXX" tagged), my bad (that grep, "ether dst" is much better, tks).
>>>>
>>>>  Look, inside the Instance - vNIC eth3:
>>>>
>>>>  tcpdump -eni eth3
>>>>
>>>>  http://paste.openstack.org/show/360127/
>>>>
>>>>
>>>> Only the PVST, OSPF and ICMP packets are hitting the tapxxxxzzzz-yy
>>>> interface! As expected, I can see those packets inside of the Instance as
>>>> well (Pastebin above).
>>>>
>>>>  Why TCP/UDP isn't passing?
>>>>
>>>>
>>>>>    * I CAN NOT see the untagged packets arriving at "tap9a546be0-d6",
>>>>> by using "tcpdump -eni tap9a546be0-d6"!
>>>>>
>>>>>
>>>>>  What do your security group rules look like?
>>>>>
>>>>
>>>>  I have no Security Groups, no Firewall, no ipset...
>>>>
>>>>
>>>>  ML2 configuration contains:
>>>>
>>>>  http://paste.openstack.org/show/356860/
>>>>
>>>>
>>>>
>>>>>
>>>>>    What is driving me crazy is that, on top of this very same setup
>>>>> (including e1000 driver), but with different vlan tag, it works!
>>>>>
>>>>>
>>>>>  Is it the same eth3 interface? You may want to avoid vlan 666,
>>>>> anyway. Never known those numbers to be lucky.
>>>>>
>>>>
>>>>  Yes, very same eth3.
>>>>
>>>>  LOL... I just posted this number here, to not publish private data,
>>>> actual VLAN ID is different.   :-P
>>>>
>>>>  Why it works for "VLAN X", but not for "VLAN Y", is a mystery for me.
>>>>
>>>>  Thank you so much for your help!
>>>>
>>>>  I'm seeing some debugging progress here...
>>>>
>>>>  Hopping to get this fixed! It is very important for the project that
>>>> I'm working on.
>>>>
>>>>
>>>>>
>>>>>  James
>>>>>
>>>>
>>>>  Thiago
>>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     : openstack at lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150710/51ee6586/attachment.html>


More information about the Openstack mailing list