<div dir="ltr">Hi Aaron,<div><br></div><div>Thanks for answering! =)</div><div><br></div><div>Lets work...</div><div class="gmail_extra"><br></div><div class="gmail_extra">---</div><div class="gmail_extra"><br></div><div class="gmail_extra">
TEST #1 - iperf between Network Node and its Uplink router (Data Center's gateway "Internet") - OVS br-ex / eth2</div><div class="gmail_extra"><br></div><div class="gmail_extra"># Tenant Namespace route table<br>
</div><div class="gmail_extra">
<br></div><div class="gmail_extra"><div class="gmail_extra">root@net-node-1:~# ip netns exec qrouter-46cb8f7a-a3c5-4da7-ad69-4de63f7c34f1 ip route</div><div class="gmail_extra">default via 172.16.0.1 dev qg-50b615b7-c2 </div>
<div class="gmail_extra"><a href="http://172.16.0.0/20" target="_blank">172.16.0.0/20</a> dev qg-50b615b7-c2 proto kernel scope link src 172.16.0.2 </div><div class="gmail_extra"><a href="http://192.168.210.0/24" target="_blank">192.168.210.0/24</a> dev qr-a1376f61-05 proto kernel scope link src <a href="tel:192.168.210.1" value="+551921682101" target="_blank">192.168.210.1</a> </div>
<div class="gmail_extra"><br></div><div class="gmail_extra"># there is a "iperf -s" running at 172.16.0.1 "Internet", testing it</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">
root@net-node-1:~# ip netns exec qrouter-46cb8f7a-a3c5-4da7-ad69-4de63f7c34f1 iperf -c 172.16.0.1</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">Client connecting to 172.16.0.1, TCP port 5001</div>
<div class="gmail_extra">TCP window size: 22.9 KByte (default)</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">[ 5] local 172.16.0.2 port 58342 connected with 172.16.0.1 port 5001</div>
<div class="gmail_extra">[ ID] Interval Transfer Bandwidth</div><div class="gmail_extra">[ 5] 0.0-10.0 sec 668 MBytes 559 Mbits/sec</div></div><div>---</div><div><br></div><div>---</div><div><br></div><div>
TEST #2 - iperf on one instance to the Namespace of the L3 agent + uplink router</div><div><br></div><div><div># iperf server running within Tenant's Namespace router</div></div><div><br></div><div><div>root@net-node-1:~# ip netns exec qrouter-46cb8f7a-a3c5-4da7-ad69-4de63f7c34f1 iperf -s</div>
</div><div><br></div><div>-</div><div><br></div><div># from instance-1</div><div><br></div><div><div>ubuntu@instance-1:~$ ip route</div><div>default via <a href="tel:192.168.210.1" value="+551921682101" target="_blank">192.168.210.1</a> dev eth0 metric 100 </div>
<div><a href="http://192.168.210.0/24" target="_blank">192.168.210.0/24</a> dev eth0 proto kernel scope link src <a href="tel:192.168.210.2" value="+551921682102" target="_blank">192.168.210.2</a> </div>
<div><br></div><div># instance-1 performing tests against net-node-1 Namespace above</div><div><br></div><div>ubuntu@instance-1:~$ iperf -c <a href="tel:192.168.210.1" value="+551921682101" target="_blank">192.168.210.1</a></div>
<div>------------------------------------------------------------</div>
<div>Client connecting to <a href="tel:192.168.210.1" value="+551921682101" target="_blank">192.168.210.1</a>, TCP port 5001</div><div>TCP window size: 21.0 KByte (default)</div><div>------------------------------------------------------------</div>
<div>[ 3] local <a href="tel:192.168.210.2" value="+551921682102" target="_blank">192.168.210.2</a> port 43739 connected with <a href="tel:192.168.210.1" value="+551921682101" target="_blank">192.168.210.1</a> port 5001</div>
<div>[ ID] Interval Transfer Bandwidth</div><div>[ 3] 0.0-10.0 sec 484 MBytes 406 Mbits/sec</div></div><div><br></div><div># still on instance-1, now against "External IP" of its own Namespace / Router</div>
<div><br></div><div><div>ubuntu@instance-1:~$ iperf -c 172.16.0.2</div><div>------------------------------------------------------------</div><div>Client connecting to 172.16.0.2, TCP port 5001</div><div>TCP window size: 21.0 KByte (default)</div>
<div>------------------------------------------------------------</div><div>[ 3] local <a href="tel:192.168.210.2" value="+551921682102" target="_blank">192.168.210.2</a> port 34703 connected with 172.16.0.2 port 5001</div>
<div>[ ID] Interval Transfer Bandwidth</div><div>[ 3] 0.0-10.0 sec 520 MBytes 436 Mbits/sec</div>
</div><div><br></div><div># still on instance-1, now against the Data Center UpLink Router</div><div><br></div><div><div>ubuntu@instance-1:~$ iperf -c 172.16.0.1</div><div>------------------------------------------------------------</div>
<div>Client connecting to 172.16.0.1, TCP port 5001</div><div>TCP window size: 21.0 KByte (default)</div><div>------------------------------------------------------------</div><div>[ 3] local <a href="tel:192.168.210.4" value="+551921682104" target="_blank">192.168.210.4</a> port 38401 connected with 172.16.0.1 port 5001</div>
<div>[ ID] Interval Transfer Bandwidth</div><div>[ 3] 0.0-10.0 sec <b><u> 324 MBytes 271 Mbits/sec</u></b></div></div><div>---<br></div><div><br></div><div>This latest test shows only 271 Mbits/s! I think it should be at least, 400~430 MBits/s... Right?!</div>
<div><br></div><div>---</div><div><br></div><div>TEST #3 - Two instances on the same hypervisor</div></div><div class="gmail_extra"><br></div><div class="gmail_extra"># iperf server</div><div class="gmail_extra"><br></div>
<div class="gmail_extra"><div class="gmail_extra">ubuntu@instance-2:~$ ip route</div><div class="gmail_extra">default via <a href="tel:192.168.210.1" value="+551921682101" target="_blank">192.168.210.1</a> dev eth0 metric 100 </div>
<div class="gmail_extra"><a href="http://192.168.210.0/24" target="_blank">192.168.210.0/24</a> dev eth0 proto kernel scope link src <a href="tel:192.168.210.4" value="+551921682104" target="_blank">192.168.210.4</a> </div>
<div class="gmail_extra"><br></div><div class="gmail_extra">ubuntu@instance-2:~$ iperf -s</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">Server listening on TCP port 5001</div>
<div class="gmail_extra">TCP window size: 85.3 KByte (default)</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">[ 4] local <a href="tel:192.168.210.4" value="+551921682104" target="_blank">192.168.210.4</a> port 5001 connected with 192.168.210.2 port 45800</div>
<div class="gmail_extra">[ ID] Interval Transfer Bandwidth</div><div class="gmail_extra">[ 4] 0.0-10.0 sec 4.61 GBytes 3.96 Gbits/sec</div><div><br></div><div># iperf client</div><div><br></div><div><div>ubuntu@instance-1:~$ iperf -c 192.168.210.4</div>
<div>------------------------------------------------------------</div><div>Client connecting to 192.168.210.4, TCP port 5001</div><div>TCP window size: 21.0 KByte (default)</div><div>------------------------------------------------------------</div>
<div>[ 3] local 192.168.210.2 port 45800 connected with 192.168.210.4 port 5001</div><div>[ ID] Interval Transfer Bandwidth</div><div>[ 3] 0.0-10.0 sec 4.61 GBytes 3.96 Gbits/sec</div></div></div><div class="gmail_extra">
---</div><div class="gmail_extra"><br></div><div class="gmail_extra">---</div><div class="gmail_extra"><br></div><div class="gmail_extra">TEST #4 - Two instances on different hypervisors - over GRE</div><div class="gmail_extra">
<br></div><div class="gmail_extra"><div class="gmail_extra">root@instance-2:~# iperf -s</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">Server listening on TCP port 5001</div>
<div class="gmail_extra">TCP window size: 85.3 KByte (default)</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">[ 4] local 192.168.210.4 port 5001 connected with 192.168.210.2 port 34640</div>
<div class="gmail_extra">[ ID] Interval Transfer Bandwidth</div><div class="gmail_extra">[ 4] 0.0-10.0 sec 237 MBytes 198 Mbits/sec</div><div><br></div></div><div class="gmail_extra"><br></div><div class="gmail_extra">
<div class="gmail_extra">root@instance-1:~# iperf -c 192.168.210.4</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">Client connecting to 192.168.210.4, TCP port 5001</div>
<div class="gmail_extra">TCP window size: 21.0 KByte (default)</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">[ 3] local 192.168.210.2 port 34640 connected with 192.168.210.4 port 5001</div>
<div class="gmail_extra">[ ID] Interval Transfer Bandwidth</div><div class="gmail_extra">[ 3] 0.0-10.0 sec 237 MBytes 198 Mbits/sec</div></div><div class="gmail_extra">---</div><div class="gmail_extra"><br>
</div><div class="gmail_extra">I just realized how slow is my intra-cloud (intra-VM) communication... :-/</div><div class="gmail_extra"><br></div><div class="gmail_extra">---</div><div class="gmail_extra"><br></div><div class="gmail_extra">
TEST #5 - Two hypervisors - "GRE TUNNEL LAN" - OVS local_ip / remote_ip</div><div class="gmail_extra"><br></div><div class="gmail_extra"># Same path of "TEST #4" but, testing the physical GRE path (where GRE traffic flows)</div>
<div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">root@hypervisor-2:~$ iperf -s</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">
Server listening on TCP port 5001</div><div class="gmail_extra">TCP window size: 85.3 KByte (default)</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">
n[ 4] local 10.20.2.57 port 5001 connected with 10.20.2.53 port 51694</div><div class="gmail_extra">[ ID] Interval Transfer Bandwidth</div><div class="gmail_extra">[ 4] 0.0-10.0 sec 1.09 GBytes 939 Mbits/sec</div>
<div><br></div></div><div class="gmail_extra"><div class="gmail_extra">root@hypervisor-1:~# iperf -c 10.20.2.57</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">
Client connecting to 10.20.2.57, TCP port 5001</div><div class="gmail_extra">TCP window size: 22.9 KByte (default)</div><div class="gmail_extra">------------------------------------------------------------</div><div class="gmail_extra">
[ 3] local 10.20.2.53 port 51694 connected with 10.20.2.57 port 5001</div><div class="gmail_extra">[ ID] Interval Transfer Bandwidth</div><div class="gmail_extra">[ 3] 0.0-10.0 sec 1.09 GBytes 939 Mbits/sec</div>
</div><div class="gmail_extra">---</div><div class="gmail_extra"><br></div><div class="gmail_extra">About Test #5, I don't know why the GRE traffic (Test #4) doesn't reach 1Gbit/sec (only ~200Mbit/s ?), since its physical path is much faster (GIGALan). Plus, Test #3 shows a pretty fast speed when traffic flows only within a hypervisor (3.96Gbit/sec).</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Tomorrow, I'll do this tests with netperf.</div><div class="gmail_extra"><br></div><div class="gmail_extra">NOTE: I'm using Open vSwitch 1.11.0, compiled for Ubuntu 12.04.3, via "dpkg-buildpackage" and installed via "Debian / Ubuntu way". If I downgrade to 1.10.2 from Havana Cloud Archive, same results... I can downgrade it, if you guys tell me to do so.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">BTW, I'll install another "Region", based on Havana on Ubuntu 13.10, with exactly the same configurations from my current Havana + Ubuntu 12.04.3, on top of the same hardware, to see if the problem still persist.<br>
</div><div class="gmail_extra"><br></div><div class="gmail_extra">Regards,</div><div class="gmail_extra">Thiago</div><div class="gmail_extra"><br><div class="gmail_quote">On 23 October 2013 22:40, Aaron Rosen <span dir="ltr"><<a href="mailto:arosen@nicira.com" target="_blank">arosen@nicira.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">
<div>On Mon, Oct 21, 2013 at 11:52 PM, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">James,<div><br></div><div>I think I'm hitting this problem.</div>
<div><br></div><div>I'm using "Per-Tenant Routers with Private Networks", GRE tunnels and L3+DHCP Network Node.</div>
<div><br>
</div><div>The connectivity from behind my Instances is very slow. It takes an eternity to finish "apt-get update".</div></div></blockquote><div><br></div><div><br></div></div><div>I'm curious if you can do the following tests to help pinpoint the bottle neck: </div>
<div><br></div><div>Run iperf or netperf between:</div><div>two instances on the same hypervisor - this will determine if it's a virtualization driver issue if the performance is bad. </div><div>two instances on different hypervisors.</div>
<div>one instance to the namespace of the l3 agent. </div><div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><br></div><div>If I run "apt-get update" from within tenant's Namespace, it goes fine.</div>
<div><br></div><div>If I enable "ovs_use_veth", Metadata (and/or DHCP) stops working and I and unable to start new Ubuntu Instances and login into them... Look:</div><div><br></div><div>--</div><div><div>cloud-init start running: Tue, 22 Oct 2013 05:57:39 +0000. up 4.01 seconds</div>
<div>2013-10-22 06:01:42,989 - util.py[WARNING]: '<a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>' failed [3/120s]: url error [[Errno 113] No route to host]</div>
<div>2013-10-22 06:01:45,988 - util.py[WARNING]: '<a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>' failed [6/120s]: url error [[Errno 113] No route to host]</div>
</div><div>--</div></div></blockquote><div><br></div><div><br></div></div><div>Do you see anything interesting in the neutron-metadata-agent log? Or it looks like your instance doesn't have a route to the default gw? </div>
<div><div>
<div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Is this problem still around?!</div>
<div><br></div><div>Should I stay away from GRE tunnels when with Havana + Ubuntu 12.04.3?</div>
<div><br></div><div>Is it possible to re-enable Metadata when ovs_use_veth = true ?</div>
<div><br></div><div>Thanks!</div><div>Thiago</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On 3 October 2013 06:27, James Page <span dir="ltr"><<a href="mailto:james.page@ubuntu.com" target="_blank">james.page@ubuntu.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA256<br>
<br>
</div><div>On 02/10/13 22:49, James Page wrote:<br>
>> sudo ip netns exec qrouter-d3baf1b1-55ee-42cb-a3f6-9629288e3221<br>
>>> traceroute -n 10.5.0.2 -p 44444 --mtu traceroute to 10.5.0.2<br>
>>> (10.5.0.2), 30 hops max, 65000 byte packets 1 10.5.0.2 0.950<br>
>>> ms F=1500 0.598 ms 0.566 ms<br>
>>><br>
>>> The PMTU from the l3 gateway to the instance looks OK to me.<br>
> I spent a bit more time debugging this; performance from within<br>
> the router netns on the L3 gateway node looks good in both<br>
> directions when accessing via the tenant network (10.5.0.2) over<br>
> the qr-XXXXX interface, but when accessing through the external<br>
> network from within the netns I see the same performance choke<br>
> upstream into the tenant network.<br>
><br>
> Which would indicate that my problem lies somewhere around the<br>
> qg-XXXXX interface in the router netns - just trying to figure out<br>
> exactly what - maybe iptables is doing something wonky?<br>
<br>
</div>OK - I found a fix but I'm not sure why this makes a difference;<br>
neither my l3-agent or dhcp-agent configuration had 'ovs_use_veth =<br>
True'; I switched this on, clearing everything down, rebooted and now<br>
I seem symmetric good performance across all neutron routers.<br>
<br>
This would point to some sort of underlying bug when ovs_use_veth = False.<br>
<div><br>
<br>
- --<br>
James Page<br>
Ubuntu and Debian Developer<br>
<a href="mailto:james.page@ubuntu.com" target="_blank">james.page@ubuntu.com</a><br>
<a href="mailto:jamespage@debian.org" target="_blank">jamespage@debian.org</a><br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1.4.14 (GNU/Linux)<br>
Comment: Using GnuPG with Thunderbird - <a href="http://www.enigmail.net/" target="_blank">http://www.enigmail.net/</a><br>
<br>
</div>iQIcBAEBCAAGBQJSTTh6AAoJEL/srsug59jDmpEP/jaB5/yn9+Xm12XrVu0Q3IV5<br>
fLGOuBboUgykVVsfkWccI/oygNlBaXIcDuak/E4jxPcoRhLAdY1zpX8MQ8wSsGKd<br>
CjSeuW8xxnXubdfzmsCKSs3FCIBhDkSYzyiJd/raLvCfflyy8Cl7KN2x22mGHJ6z<br>
qZ9APcYfm9qCVbEssA3BHcUL+st1iqMJ0YhVZBk03+QEXaWu3FFbjpjwx3X1ZvV5<br>
Vbac7enqy7Lr4DSAIJVldeVuRURfv3YE3iJZTIXjaoUCCVTQLm5OmP9TrwBNHLsA<br>
7W+LceQri+Vh0s4dHPKx5MiHsV3RCydcXkSQFYhx7390CXypMQ6WwXEY/a8Egssg<br>
SuxXByHwEcQFa+9sCwPQ+RXCmC0O6kUi8EPmwadjI5Gc1LoKw5Wov/SEen86fDUW<br>
P9pRXonseYyWN9I4MT4aG1ez8Dqq/SiZyWBHtcITxKI2smD92G9CwWGo4L9oGqJJ<br>
UcHRwQaTHgzy3yETPO25hjax8ZWZGNccHBixMCZKegr9p2dhR+7qF8G7mRtRQLxL<br>
0fgOAExn/SX59ZT4RaYi9fI6Gng13RtSyI87CJC/50vfTmqoraUUK1aoSjIY4Dt+<br>
DYEMMLp205uLEj2IyaNTzykR0yh3t6dvfpCCcRA/xPT9slfa0a7P8LafyiWa4/5c<br>
jkJM4Y1BUV+2L5Rrf3sc<br>
=4lO4<br>
<div><div>-----END PGP SIGNATURE-----<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
</div></div></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div>