<div dir="ltr">Hi Daniel,<div><br></div><div>I followed that page, my Instances MTU is lowered by DHCP Agent but, same result: poor network performance (internal between Instances and when trying to reach the Internet).</div>
<div><br></div><div>No matter if I use "dnsmasq_config_file=/etc/neutron/dnsmasq-neutron.conf + "dhcp-option-force=26,1400"" for my Neutron DHCP agent, or not (i.e. MTU = 1500), the result is almost the same.</div>
<div><br></div><div>I'll try VXLAN (or just VLANs) this weekend to see if I can get better results...</div><div><br></div><div>Thanks!</div><div>Thiago</div><div><br></div><div><br></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On 24 October 2013 17:38, Speichert,Daniel <span dir="ltr"><<a href="mailto:djs428@drexel.edu" target="_blank">djs428@drexel.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">We managed to bring the upload speed back to maximum on the instances through the use of this guide:<u></u><u></u></span></p>
<p class="MsoNormal"><a href="http://docs.openstack.org/trunk/openstack-network/admin/content/openvswitch_plugin.html" target="_blank">http://docs.openstack.org/trunk/openstack-network/admin/content/openvswitch_plugin.html</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Basically, the MTU needs to be lowered for GRE tunnels. It can be done with DHCP as explained in the new trunk manual.<span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Daniel<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <a href="mailto:annegentle@justwriteclick.com" target="_blank">annegentle@justwriteclick.com</a> [mailto:<a href="mailto:annegentle@justwriteclick.com" target="_blank">annegentle@justwriteclick.com</a>]
<b>On Behalf Of </b>Anne Gentle<br>
<b>Sent:</b> Thursday, October 24, 2013 12:08 PM<br>
<b>To:</b> Martinx - </span><span style="font-size:11.0pt;font-family:"MS Gothic"">$B%8%'!<%`%:(B</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><br>
<b>Cc:</b> Speichert,Daniel; <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a></span></p><div><div class="h5"><br>
<b>Subject:</b> Re: [Openstack] Directional network performance issues with Neutron + OpenvSwitch<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, Oct 24, 2013 at 10:37 AM, Martinx - <span style="font-family:"Calibri","sans-serif"">
$B%8%'!<%`%:(B</span> <<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Precisely!<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The doc currently says to disable Namespace when using GRE, never did this before, look:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://docs.openstack.org/trunk/install-guide/install/apt/content/install-neutron.install-plugin.ovs.gre.html" target="_blank">http://docs.openstack.org/trunk/install-guide/install/apt/content/install-neutron.install-plugin.ovs.gre.html</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">But on this very same doc, they say to enable it... Who knows?! =P<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://docs.openstack.org/trunk/install-guide/install/apt/content/section_networking-routers-with-private-networks.html" target="_blank">http://docs.openstack.org/trunk/install-guide/install/apt/content/section_networking-routers-with-private-networks.html</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I stick with Namespace enabled...<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Just a reminder, /trunk/ links are works in progress, thanks for bringing the mismatch to our attention, and we already have a doc bug filed:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="https://bugs.launchpad.net/openstack-manuals/+bug/1241056" target="_blank">https://bugs.launchpad.net/openstack-manuals/+bug/1241056</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Review this patch: <a href="https://review.openstack.org/#/c/53380/" target="_blank">https://review.openstack.org/#/c/53380/</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Anne<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Let me ask you something, when you enable ovs_use_veth, que Metadata and DHCP still works?!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thiago<u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On 24 October 2013 12:22, Speichert,Daniel <<a href="mailto:djs428@drexel.edu" target="_blank">djs428@drexel.edu</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Hello everyone,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">It seems we also ran into the same issue.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">We are running Ubuntu Saucy with OpenStack Havana from Ubuntu Cloud archives (precise-updates).</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">The download speed to the VMs increased from 5 Mbps to maximum after enabling
</span>ovs_use_veth. Upload speed from the VMs is still terrible (max 1 Mbps, usually 0.04 Mbps).<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Here is the iperf between the instance and L3 agent (network node) inside namespace.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">root@cloud:~# ip netns exec qrouter-a29e0200-d390-40d1-8cf7-7ac1cef5863a iperf -c 10.1.0.24 -r</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Server listening on TCP port 5001</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">TCP window size: 85.3 KByte (default)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">------------------------------------------------------------</span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Client connecting to 10.1.0.24, TCP port 5001</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">TCP window size: 585 KByte (default)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">[ 7] local 10.1.0.1 port 37520 connected with 10.1.0.24 port 5001</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">[ ID] Interval Transfer Bandwidth</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">[ 7] 0.0-10.0 sec 845 MBytes 708 Mbits/sec</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">[ 6] local 10.1.0.1 port 5001 connected with 10.1.0.24 port 53006</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">[ 6] 0.0-31.4 sec 256 KBytes 66.7 Kbits/sec</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">We are using Neutron OpenVSwitch with GRE and namespaces.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><br>
A side question: the documentation says to disable namespaces with GRE and enable them with VLANs. It was always working well for us on Grizzly with GRE and namespaces and we could never get it to work without namespaces. Is there any specific reason why the
documentation is advising to disable it?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="PL" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Regards,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="PL" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Daniel</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Martinx -
</span><span style="font-size:11.0pt;font-family:"MS Gothic"">$B%8%'!<%`%:(B</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> [mailto:<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>]
<br>
<b>Sent:</b> Thursday, October 24, 2013 3:58 AM<br>
<b>To:</b> Aaron Rosen<br>
<b>Cc:</b> <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a></span><u></u><u></u></p>
<div>
<p class="MsoNormal"><br>
<b>Subject:</b> Re: [Openstack] Directional network performance issues with Neutron + OpenvSwitch<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Hi Aaron,<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for answering! =)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Lets work...<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">---<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TEST #1 - iperf between Network Node and its Uplink router (Data Center's gateway "Internet") - OVS br-ex / eth2<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"># Tenant Namespace route table<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">root@net-node-1:~# ip netns exec qrouter-46cb8f7a-a3c5-4da7-ad69-4de63f7c34f1 ip route<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">default via 172.16.0.1 dev qg-50b615b7-c2 <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://172.16.0.0/20" target="_blank">172.16.0.0/20</a> dev qg-50b615b7-c2 proto kernel scope link src 172.16.0.2 <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://192.168.210.0/24" target="_blank">192.168.210.0/24</a> dev qr-a1376f61-05 proto kernel scope link src
<a href="tel:192.168.210.1" target="_blank">192.168.210.1</a> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"># there is a "iperf -s" running at 172.16.0.1 "Internet", testing it<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">root@net-node-1:~# ip netns exec qrouter-46cb8f7a-a3c5-4da7-ad69-4de63f7c34f1 iperf -c 172.16.0.1<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Client connecting to 172.16.0.1, TCP port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TCP window size: 22.9 KByte (default)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 5] local 172.16.0.2 port 58342 connected with 172.16.0.1 port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ ID] Interval Transfer Bandwidth<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 5] 0.0-10.0 sec 668 MBytes 559 Mbits/sec<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">---<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">---<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TEST #2 - iperf on one instance to the Namespace of the L3 agent + uplink router<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"># iperf server running within Tenant's Namespace router<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">root@net-node-1:~# ip netns exec qrouter-46cb8f7a-a3c5-4da7-ad69-4de63f7c34f1 iperf -s<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"># from instance-1<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">ubuntu@instance-1:~$ ip route<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">default via
<a href="tel:192.168.210.1" target="_blank">192.168.210.1</a> dev eth0 metric 100 <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://192.168.210.0/24" target="_blank">192.168.210.0/24</a> dev eth0 proto kernel scope link src
<a href="tel:192.168.210.2" target="_blank">192.168.210.2</a> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"># instance-1 performing tests against net-node-1 Namespace above<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">ubuntu@instance-1:~$ iperf -c
<a href="tel:192.168.210.1" target="_blank">192.168.210.1</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Client connecting to
<a href="tel:192.168.210.1" target="_blank">192.168.210.1</a>, TCP port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TCP window size: 21.0 KByte (default)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] local
<a href="tel:192.168.210.2" target="_blank">192.168.210.2</a> port 43739 connected with
<a href="tel:192.168.210.1" target="_blank">192.168.210.1</a> port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ ID] Interval Transfer Bandwidth<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] 0.0-10.0 sec 484 MBytes 406 Mbits/sec<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"># still on instance-1, now against "External IP" of its own Namespace / Router<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">ubuntu@instance-1:~$ iperf -c 172.16.0.2<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Client connecting to 172.16.0.2, TCP port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TCP window size: 21.0 KByte (default)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] local
<a href="tel:192.168.210.2" target="_blank">192.168.210.2</a> port 34703 connected with 172.16.0.2 port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ ID] Interval Transfer Bandwidth<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] 0.0-10.0 sec 520 MBytes 436 Mbits/sec<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"># still on instance-1, now against the Data Center UpLink Router<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">ubuntu@instance-1:~$ iperf -c 172.16.0.1<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Client connecting to 172.16.0.1, TCP port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TCP window size: 21.0 KByte (default)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] local
<a href="tel:192.168.210.4" target="_blank">192.168.210.4</a> port 38401 connected with 172.16.0.1 port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ ID] Interval Transfer Bandwidth<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] 0.0-10.0 sec <b><u> 324 MBytes 271 Mbits/sec</u></b><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">---<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">This latest test shows only 271 Mbits/s! I think it should be at least, 400~430 MBits/s... Right?!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">---<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TEST #3 - Two instances on the same hypervisor<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"># iperf server<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">ubuntu@instance-2:~$ ip route<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">default via
<a href="tel:192.168.210.1" target="_blank">192.168.210.1</a> dev eth0 metric 100 <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://192.168.210.0/24" target="_blank">192.168.210.0/24</a> dev eth0 proto kernel scope link src
<a href="tel:192.168.210.4" target="_blank">192.168.210.4</a> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">ubuntu@instance-2:~$ iperf -s<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Server listening on TCP port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TCP window size: 85.3 KByte (default)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 4] local
<a href="tel:192.168.210.4" target="_blank">192.168.210.4</a> port 5001 connected with
<a href="tel:192.168.210.2" target="_blank">192.168.210.2</a> port 45800<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ ID] Interval Transfer Bandwidth<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 4] 0.0-10.0 sec 4.61 GBytes 3.96 Gbits/sec<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"># iperf client<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">ubuntu@instance-1:~$ iperf -c
<a href="tel:192.168.210.4" target="_blank">192.168.210.4</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Client connecting to
<a href="tel:192.168.210.4" target="_blank">192.168.210.4</a>, TCP port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TCP window size: 21.0 KByte (default)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] local
<a href="tel:192.168.210.2" target="_blank">192.168.210.2</a> port 45800 connected with
<a href="tel:192.168.210.4" target="_blank">192.168.210.4</a> port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ ID] Interval Transfer Bandwidth<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] 0.0-10.0 sec 4.61 GBytes 3.96 Gbits/sec<u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal">---<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">---<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TEST #4 - Two instances on different hypervisors - over GRE<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">root@instance-2:~# iperf -s<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Server listening on TCP port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TCP window size: 85.3 KByte (default)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 4] local
<a href="tel:192.168.210.4" target="_blank">192.168.210.4</a> port 5001 connected with
<a href="tel:192.168.210.2" target="_blank">192.168.210.2</a> port 34640<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ ID] Interval Transfer Bandwidth<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 4] 0.0-10.0 sec 237 MBytes 198 Mbits/sec<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">root@instance-1:~# iperf -c
<a href="tel:192.168.210.4" target="_blank">192.168.210.4</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Client connecting to
<a href="tel:192.168.210.4" target="_blank">192.168.210.4</a>, TCP port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TCP window size: 21.0 KByte (default)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] local
<a href="tel:192.168.210.2" target="_blank">192.168.210.2</a> port 34640 connected with
<a href="tel:192.168.210.4" target="_blank">192.168.210.4</a> port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ ID] Interval Transfer Bandwidth<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] 0.0-10.0 sec 237 MBytes 198 Mbits/sec<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">---<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I just realized how slow is my intra-cloud (intra-VM) communication... :-/<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">---<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TEST #5 - Two hypervisors - "GRE TUNNEL LAN" - OVS local_ip / remote_ip<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"># Same path of "TEST #4" but, testing the physical GRE path (where GRE traffic flows)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">root@hypervisor-2:~$ iperf -s<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Server listening on TCP port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TCP window size: 85.3 KByte (default)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">n[ 4] local 10.20.2.57 port 5001 connected with 10.20.2.53 port 51694<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ ID] Interval Transfer Bandwidth<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 4] 0.0-10.0 sec 1.09 GBytes 939 Mbits/sec<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">root@hypervisor-1:~# iperf -c 10.20.2.57<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Client connecting to 10.20.2.57, TCP port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">TCP window size: 22.9 KByte (default)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] local 10.20.2.53 port 51694 connected with 10.20.2.57 port 5001<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ ID] Interval Transfer Bandwidth<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[ 3] 0.0-10.0 sec 1.09 GBytes 939 Mbits/sec<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">---<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">About Test #5, I don't know why the GRE traffic (Test #4) doesn't reach 1Gbit/sec (only ~200Mbit/s ?), since its physical path is much faster (GIGALan). Plus, Test #3 shows a pretty
fast speed when traffic flows only within a hypervisor (3.96Gbit/sec).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Tomorrow, I'll do this tests with netperf.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">NOTE: I'm using Open vSwitch 1.11.0, compiled for Ubuntu 12.04.3, via "dpkg-buildpackage" and installed via "Debian / Ubuntu way". If I downgrade to 1.10.2 from Havana Cloud Archive,
same results... I can downgrade it, if you guys tell me to do so.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">BTW, I'll install another "Region", based on Havana on Ubuntu 13.10, with exactly the same configurations from my current Havana + Ubuntu 12.04.3, on top of the same hardware, to
see if the problem still persist.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thiago<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On 23 October 2013 22:40, Aaron Rosen <<a href="mailto:arosen@nicira.com" target="_blank">arosen@nicira.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Mon, Oct 21, 2013 at 11:52 PM, Martinx -
<span style="font-family:"Calibri","sans-serif"">$B%8%'!<%`%:(B</span> <<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">James,<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I think I'm hitting this problem.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I'm using "Per-Tenant Routers with Private Networks", GRE tunnels and L3+DHCP Network Node.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The connectivity from behind my Instances is very slow. It takes an eternity to finish "apt-get update".<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">I'm curious if you can do the following tests to help pinpoint the bottle neck: <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Run iperf or netperf between:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">two instances on the same hypervisor - this will determine if it's a virtualization driver issue if the performance is bad. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">two instances on different hypervisors.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">one instance to the namespace of the l3 agent. <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If I run "apt-get update" from within tenant's Namespace, it goes fine.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If I enable "ovs_use_veth", Metadata (and/or DHCP) stops working and I and unable to start new Ubuntu Instances and login into them... Look:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">--<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">cloud-init start running: Tue, 22 Oct 2013 05:57:39 +0000. up 4.01 seconds<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2013-10-22 06:01:42,989 - util.py[WARNING]: '<a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>'
failed [3/120s]: url error [[Errno 113] No route to host]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2013-10-22 06:01:45,988 - util.py[WARNING]: '<a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>'
failed [6/120s]: url error [[Errno 113] No route to host]<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">--<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">Do you see anything interesting in the neutron-metadata-agent log? Or it looks like your instance doesn't have a route to the default gw? <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Is this problem still around?!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Should I stay away from GRE tunnels when with Havana + Ubuntu 12.04.3?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Is it possible to re-enable Metadata when ovs_use_veth = true ?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thiago<u></u><u></u></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On 3 October 2013 06:27, James Page <<a href="mailto:james.page@ubuntu.com" target="_blank">james.page@ubuntu.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA256<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">On 02/10/13 22:49, James Page wrote:<br>
>> sudo ip netns exec qrouter-d3baf1b1-55ee-42cb-a3f6-9629288e3221<br>
>>> traceroute -n 10.5.0.2 -p 44444 --mtu traceroute to 10.5.0.2<br>
>>> (10.5.0.2), 30 hops max, 65000 byte packets 1 10.5.0.2 0.950<br>
>>> ms F=1500 0.598 ms 0.566 ms<br>
>>><br>
>>> The PMTU from the l3 gateway to the instance looks OK to me.<br>
> I spent a bit more time debugging this; performance from within<br>
> the router netns on the L3 gateway node looks good in both<br>
> directions when accessing via the tenant network (10.5.0.2) over<br>
> the qr-XXXXX interface, but when accessing through the external<br>
> network from within the netns I see the same performance choke<br>
> upstream into the tenant network.<br>
><br>
> Which would indicate that my problem lies somewhere around the<br>
> qg-XXXXX interface in the router netns - just trying to figure out<br>
> exactly what - maybe iptables is doing something wonky?<u></u><u></u></p>
</div>
<p class="MsoNormal">OK - I found a fix but I'm not sure why this makes a difference;<br>
neither my l3-agent or dhcp-agent configuration had 'ovs_use_veth =<br>
True'; I switched this on, clearing everything down, rebooted and now<br>
I seem symmetric good performance across all neutron routers.<br>
<br>
This would point to some sort of underlying bug when ovs_use_veth = False.<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
- --<br>
James Page<br>
Ubuntu and Debian Developer<br>
<a href="mailto:james.page@ubuntu.com" target="_blank">james.page@ubuntu.com</a><br>
<a href="mailto:jamespage@debian.org" target="_blank">jamespage@debian.org</a><br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1.4.14 (GNU/Linux)<br>
Comment: Using GnuPG with Thunderbird - <a href="http://www.enigmail.net/" target="_blank">
http://www.enigmail.net/</a><u></u><u></u></p>
</div>
<p class="MsoNormal">iQIcBAEBCAAGBQJSTTh6AAoJEL/srsug59jDmpEP/jaB5/yn9+Xm12XrVu0Q3IV5<br>
fLGOuBboUgykVVsfkWccI/oygNlBaXIcDuak/E4jxPcoRhLAdY1zpX8MQ8wSsGKd<br>
CjSeuW8xxnXubdfzmsCKSs3FCIBhDkSYzyiJd/raLvCfflyy8Cl7KN2x22mGHJ6z<br>
qZ9APcYfm9qCVbEssA3BHcUL+st1iqMJ0YhVZBk03+QEXaWu3FFbjpjwx3X1ZvV5<br>
Vbac7enqy7Lr4DSAIJVldeVuRURfv3YE3iJZTIXjaoUCCVTQLm5OmP9TrwBNHLsA<br>
7W+LceQri+Vh0s4dHPKx5MiHsV3RCydcXkSQFYhx7390CXypMQ6WwXEY/a8Egssg<br>
SuxXByHwEcQFa+9sCwPQ+RXCmC0O6kUi8EPmwadjI5Gc1LoKw5Wov/SEen86fDUW<br>
P9pRXonseYyWN9I4MT4aG1ez8Dqq/SiZyWBHtcITxKI2smD92G9CwWGo4L9oGqJJ<br>
UcHRwQaTHgzy3yETPO25hjax8ZWZGNccHBixMCZKegr9p2dhR+7qF8G7mRtRQLxL<br>
0fgOAExn/SX59ZT4RaYi9fI6Gng13RtSyI87CJC/50vfTmqoraUUK1aoSjIY4Dt+<br>
DYEMMLp205uLEj2IyaNTzykR0yh3t6dvfpCCcRA/xPT9slfa0a7P8LafyiWa4/5c<br>
jkJM4Y1BUV+2L5Rrf3sc<br>
=4lO4<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">-----END PGP SIGNATURE-----<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>