<p>Narayan,</p>
<p>Are you doing bonding in conjunction with your bridging + vlans? Or is it just a single interface backing the vlan_interface?</p>
<p>Nate</p>
<div class="gmail_quote">On Jul 16, 2012 9:55 PM, "Narayan Desai" <<a href="mailto:narayan.desai@gmail.com" target="_blank">narayan.desai@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
We're running into what looks like a linux bridging bug, which causes<br>
both substantial (20-40%) packet loss, and DNS to fail about that same<br>
fraction of the time. We're running essex on precise, with dedicated<br>
nova-network servers and VLANManager. On either of our nova-network<br>
servers, we see the same behavior. When tracking this down, I found<br>
the following, when tcpdump'ing along the path between vm instance and<br>
n-net gateway.<br>
<br>
The packets appear to make it to the nova-network server, and are<br>
properly pulled out of dot1q tagging:<br>
root@m5-p:~# tcpdump -K -p -i vlan200 -v -vv udp port 53<br>
tcpdump: WARNING: vlan200: no IPv4 address assigned<br>
tcpdump: listening on vlan200, link-type EN10MB (Ethernet), capture<br>
size 65535 bytes<br>
20:34:02.377711 IP (tos 0x0, ttl 64, id 59761, offset 0, flags [none],<br>
proto UDP (17), length 60)<br>
10.0.0.3.54937 > 10.0.0.1.domain: 52874+ A? <a href="http://www.google.com" target="_blank">www.google.com</a>. (32)<br>
20:34:07.377942 IP (tos 0x0, ttl 64, id 59762, offset 0, flags [none],<br>
proto UDP (17), length 60) 10.0.0.3.54937 > 10.0.0.1.domain: 52874+<br>
A? <a href="http://www.google.com" target="_blank">www.google.com</a>. (32)<br>
20:34:12.378248 IP (tos 0x0, ttl 64, id 59763, offset 0, flags [none],<br>
proto UDP (17), length 60) 10.0.0.3.54937 > 10.0.0.1.domain: 52874+<br>
A? <a href="http://www.google.com" target="_blank">www.google.com</a>. (32)<br>
20:34:12.378428 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto<br>
UDP (17), length 170) 10.0.0.1.domain > 10.0.0.3.54937: 52874 q: A?<br>
<a href="http://www.google.com" target="_blank">www.google.com</a>. 6/0/0 <a href="http://www.google.com" target="_blank">www.google.com</a>. [1d3h55m19s] CNAME<br>
<a href="http://www.l.google.com" target="_blank">www.l.google.com</a>., <a href="http://www.l.google.com" target="_blank">www.l.google.com</a>. [1m33s] A 74.125.225.209,<br>
<a href="http://www.l.google.com" target="_blank">www.l.google.com</a>. [1m33s] A 74.125.225.208, <a href="http://www.l.google.com" target="_blank">www.l.google.com</a>. [1m33s]<br>
A 74.125.225.212, <a href="http://www.l.google.com" target="_blank">www.l.google.com</a>. [1m33s] A 74.125.225.211,<br>
<a href="http://www.l.google.com" target="_blank">www.l.google.com</a>. [1m33s] A 74.125.225.210 (142)<br>
<br>
But some packets don't make it all of the way to the bridged interface:<br>
root@m5-p:~# brctl show<br>
bridge name bridge id STP enabled interfaces<br>
br200 8000.fa163e18927b no vlan200<br>
<br>
root@m5-p:~# tcpdump -K -p -i br200 -v -vv udp port 53<br>
tcpdump: listening on br200, link-type EN10MB (Ethernet), capture size<br>
65535 bytes<br>
20:34:12.378264 IP (tos 0x0, ttl 64, id 59763, offset 0, flags [none],<br>
proto UDP (17), length 60)<br>
10.0.0.3.54937 > 10.0.0.1.domain: 52874+ A? <a href="http://www.google.com" target="_blank">www.google.com</a>. (32)<br>
20:34:12.378424 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto<br>
UDP (17), length 170)<br>
10.0.0.1.domain > 10.0.0.3.54937: 52874 q: A? <a href="http://www.google.com" target="_blank">www.google.com</a>.<br>
6/0/0 <a href="http://www.google.com" target="_blank">www.google.com</a>. [1d3h55m19s] CNAME <a href="http://www.l.google.com" target="_blank">www.l.google.com</a>.,<br>
<a href="http://www.l.google.com" target="_blank">www.l.google.com</a>. [1m33s] A 74.125.225.209, <a href="http://www.l.google.com" target="_blank">www.l.google.com</a>. [1m33s]<br>
A 74.125.225.208, <a href="http://www.l.google.com" target="_blank">www.l.google.com</a>. [1m33s] A 74.125.225.212,<br>
<a href="http://www.l.google.com" target="_blank">www.l.google.com</a>. [1m33s] A 74.125.225.211, <a href="http://www.l.google.com" target="_blank">www.l.google.com</a>. [1m33s]<br>
A 74.125.225.210 (142)<br>
<br>
I can't find any way that ipfilter could be implicated in this; there<br>
aren't deny rules that are hitting.<br>
<br>
Oddly enough, this seems to cause no loss in icmp traffic, even with ping -f.<br>
<br>
So far, searching hasn't netted very much. I've found this similar<br>
sounding ubuntu bug report, but it looks like no one is working on it:<br>
<a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/986043" target="_blank">https://bugs.launchpad.net/ubuntu/+source/linux/+bug/986043</a><br>
<br>
We're at 3.2.0-24, and there is a 3.2.0-25, but it is reported to not<br>
fix this issue, and neither are 3.4 kernels.<br>
<br>
It seems sad to try backrevving to an onieric kernel, but that is on<br>
my list for tomorrow. If this is a kernel bug, it will make the<br>
precise default kernel unusable for nova-network servers with dot1q<br>
(or whatever the appropriate feature interaction is).<br>
<br>
Does this ring any bells, or is there another course of action I should attempt?<br>
thanks in advance for any suggestions.<br>
-nld<br>
<br>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
</blockquote></div>