<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif;font-size:12pt"><div><br></div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,Sans-Serif; background-color: transparent; font-style: normal;">ok, the tunnels look fine. One thing that looks funny on the network node are these untagged tap* devices. I guess you switched to using veths and then switched back to not using them. I don't know if they matter, but you should clean them up by stopping everthing, running neutron-ovs-cleanup (check bridges empty) and reboot.<br><span></span></div><pre> Bridge br-int
Port "tapa1376f61-05"
Interface "tapa1376f61-05"
...<br> Port "qr-a1376f61-05"
tag: 1
Interface "qr-a1376f61-05"
type: internal<br><br><span style="font-family: HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,Sans-Serif;">Re, Darragh.</span><br></pre><div style="display: block;" class="yahoo_quoted"> <br> <br> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif; font-size: 12pt;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> On Friday, 25 October 2013, 17:28, Martinx - ジェームズ <thiagocmartinsc@gmail.com> wrote:<br> </font> </div> <blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; margin-top: 5px; padding-left: 5px;"> <div class="y_msg_container"><div id="yiv5939486563"><div><div dir="ltr">Here we go:<div><br clear="none"></div><div>---</div><div><div>root@net-node-1:~# grep local_ip
/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini </div><div>local_ip = 10.20.2.52</div></div><div><br clear="none"></div><div><div>
root@net-node-1:~# ip r | grep 10.\20</div><div><a rel="nofollow" shape="rect" target="_blank" href="http://10.20.2.0/24">10.20.2.0/24</a> dev eth1 proto kernel scope link src 10.20.2.52 </div></div><div>---</div><div><br clear="none"></div><div>---</div><div><div>root@hypervisor-1:~# grep local_ip /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini</div>
<div>local_ip = 10.20.2.53</div><div><br clear="none"></div><div>root@hypervisor-1:~# ip r | grep 10.\20</div><div><a rel="nofollow" shape="rect" target="_blank" href="http://10.20.2.0/24">10.20.2.0/24</a> dev eth1 proto kernel scope link src 10.20.2.53 </div></div><div>---</div>
<div><br clear="none"></div><div>---</div><div><div>root@hypervisor-2:~# grep local_ip /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini</div><div>local_ip = 10.20.2.57</div><div><br clear="none"></div><div>root@hypervisor-2:~# ip r | grep 10.\20</div>
<div><a rel="nofollow" shape="rect" target="_blank" href="http://10.20.2.0/24">10.20.2.0/24</a> dev eth1 proto kernel scope link src 10.20.2.57</div></div><div>---</div><div><br clear="none"></div><div>Each "ovs-vsctl show":</div><div><br clear="none"></div><div>net-node-1: <a rel="nofollow" shape="rect" target="_blank" href="http://paste.openstack.org/show/49727/">http://paste.openstack.org/show/49727/</a></div>
<div><br clear="none"></div><div>hypervisor-1: <a rel="nofollow" shape="rect" target="_blank" href="http://paste.openstack.org/show/49728/">http://paste.openstack.org/show/49728/</a></div><div><br clear="none"></div><div>hypervisor-2: <a rel="nofollow" shape="rect" target="_blank" href="http://paste.openstack.org/show/49729/">http://paste.openstack.org/show/49729/</a><br clear="none">
</div><div><br clear="none"></div><div><br clear="none"></div><div>Best,</div><div>Thiago</div></div><div class="yiv5939486563yqt0872517347" id="yiv5939486563yqt56480"><div class="yiv5939486563gmail_extra"><br clear="none"><br clear="none"><div class="yiv5939486563gmail_quote">On 25 October 2013 14:11, Darragh O'Reilly <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:dara2002-openstack@yahoo.com" target="_blank" href="mailto:dara2002-openstack@yahoo.com">dara2002-openstack@yahoo.com</a>></span> wrote:<br clear="none">
<blockquote class="yiv5939486563gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div><div style="font-size:12pt;font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif;"><div>
<span><br clear="none"></span></div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif;"><span>the uneven ssh performance is strange - maybe learning on the tunnel mesh is not stablizing. It is easy to mess it up by giving a wrong local_ip in the ovs-plugin config file. Check the tunnels ports on br-tun with 'ovs-vsctl show'. Is each one using the correct IPs? Br-tun should have N-1 gre-x ports - no more! Maybe you can put 'ovs-vsctl show' from the nodes on paste.openstack if there are not to many?</span></div>
<div style="font-style:normal;font-size:16px;background-color:transparent;font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif;"><br clear="none"><span></span></div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif;">
<span>Re, Darragh.<br clear="none"></span></div><div><div class="yiv5939486563h5"><div style="display:block;"> <br clear="none"> <br clear="none"> <div style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif;font-size:12pt;"> <div style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif;font-size:12pt;">
<div dir="ltr"> <font face="Arial"> On Friday, 25 October 2013, 16:20, Martinx - ジェームズ <<a rel="nofollow" shape="rect" ymailto="mailto:thiagocmartinsc@gmail.com" target="_blank" href="mailto:thiagocmartinsc@gmail.com">thiagocmartinsc@gmail.com</a>> wrote:<br clear="none"> </font> </div> <blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;margin-top:5px;padding-left:5px;">
<div><div><div><div dir="ltr">I think can say... "YAY!!"
:-D<div><br clear="none"></div><div>With "LibvirtOpenVswitchDriver" my internal communication is the double now! It goes from ~200 (with LibvirtHybridOVSBridgeDriver) to <b><u>400Mbit/s</u></b> (with LibvirtOpenVswitchDriver)! Still far from 1Gbit/s (my physical path limit) but, more acceptable now.</div>
<div><br clear="none"></div><div>The command "ethtool -K eth1 gro off" still makes no difference.</div><div><br clear="none"></div><div>So, there is only 1 remain problem, when traffic pass trough L3 / Namespace, it is still useless. Even the SSH connection into my Instances, via its Floating IPs, is slow as hell, sometimes it just stops responding for a few seconds, and becomes online again "out-of-nothing"...</div>
<div><br clear="none"></div><div>I just detect a weird "behavior", when I run "apt-get update" from instance-1, it is slow as I said plus, its ssh connection (where I'm running apt-get update), stops responding right after I run "apt-get update" AND, <u>all my others ssh connections also stops working too!</u> For a few seconds... This means that when I run "apt-get update" from within instance-1, the SSH session of instance-2 is affected too!! There is something pretty bad going on at L3 / Namespace.</div>
<div><br clear="none"></div><div>BTW, do you think that a ~400MBit/sec intra-vm-communication (GRE tunnel) on top of a 1Gbit ethernet is acceptable?! It is still less than a half...</div><div><br clear="none"></div><div>
Thank you!</div><div>Thiago<br clear="none"><br clear="none"><div><div>On 25 October 2013 12:28, Darragh O'Reilly <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:dara2002-openstack@yahoo.com" target="_blank" href="mailto:dara2002-openstack@yahoo.com">dara2002-openstack@yahoo.com</a>></span> wrote:<br clear="none">
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex;"><div><div style="font-size:12pt;font-family:HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;">
<div>Hi Thiago,</div><div><br clear="none"></div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;">
for the VIF error: you will need to change qemu.conf as described here:</div>
<div style="font-style:normal;font-size:16px;background-color:transparent;font-family:HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;"><a rel="nofollow" shape="rect" target="_blank" href="http://openvswitch.org/openstack/documentation/">http://openvswitch.org/openstack/documentation/</a></div>
<div style="font-style:normal;font-size:16px;background-color:transparent;font-family:HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;"> <br clear="none"></div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;">
Re, Darragh.<br clear="none"></div><div><div><div style="display:block;"> <br clear="none"> <br clear="none"> <div style="font-family:HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;font-size:12pt;">
<div style="font-family:HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;font-size:12pt;">
<div dir="ltr"> <font face="Arial"> On Friday, 25 October 2013, 15:14, Martinx - ジェームズ <<a rel="nofollow" shape="rect" ymailto="mailto:thiagocmartinsc@gmail.com" target="_blank" href="mailto:thiagocmartinsc@gmail.com">thiagocmartinsc@gmail.com</a>> wrote:<br clear="none">
</font> </div> <blockquote style="border-left-width:2px;border-left-style:solid;border-left-color:rgb(16,16,255);margin-left:5px;margin-top:5px;padding-left:5px;">
<div><div><div><div dir="ltr">Hi Darragh,<div><br clear="none"></div><div>Yes, Instances are getting MTU 1400.</div><div><br clear="none"></div><div>I'm using LibvirtHybridOVSBridgeDriver at my Compute Nodes. I'll check BG 1223267 right now! </div>
<div><br clear="none">
</div><div><br clear="none"></div><div>The <span style="font-family:arial, sans-serif;font-size:12.800000190734863px;">LibvirtOpenVswitchDriver doesn't work, look:</span></div><div><span style="font-family:arial, sans-serif;font-size:12.800000190734863px;"><br clear="none">
</span></div><div><a rel="nofollow" shape="rect" target="_blank" href="http://paste.openstack.org/show/49709/">http://paste.openstack.org/show/49709/</a><span style="font-family:arial, sans-serif;font-size:12.800000190734863px;"><br clear="none">
</span></div><div><br clear="none"></div><div>
<a rel="nofollow" shape="rect" target="_blank" href="http://paste.openstack.org/show/49710/">http://paste.openstack.org/show/49710/</a><br clear="none">
</div><div><span style="font-family:arial, sans-serif;font-size:12.800000190734863px;"><br clear="none"></span></div><div><span style="font-family:arial, sans-serif;font-size:12.800000190734863px;"><br clear="none"></span></div>
<div><span style="font-family:arial, sans-serif;font-size:12.800000190734863px;">My NICs are "</span><font face="arial, sans-serif">RTL8111/8168/8411 PCI Express Gigabit Ethernet", Hypervisors motherboard are MSI-890FXA-GD70.</font></div>
<div><font face="arial, sans-serif"><br clear="none"></font></div><div><font face="arial, sans-serif">The command "ethtool -K eth1 gro off" did not had any effect on the communication between instances on different hypervisors, still poor, around 248Mbit/sec, when its physical path reach 1Gbit/s (where GRE is built).</font></div>
<div><font face="arial, sans-serif"><br clear="none"></font></div><div><font face="arial, sans-serif">My Linux version is "Linux hypervisor-1 3.8.0-32-generic #47~precise1-Ubuntu", same kernel on Network Node" and others nodes too (Ubuntu 12.04.3 installed from scratch for this Havana deployment).</font></div>
<div><font face="arial, sans-serif"><br clear="none"></font></div><div><font face="arial, sans-serif">The only difference I can see right now, between my two hypervisors, is that my second is just a spare machine, with a slow CPU but, I don't think it will have a negative impact at the network throughput, since I have only 1 Instance running into it (plus a qemu-nbd process eating 90% of its CPU). I'll replace this CPU tomorrow, to redo this tests again but, I don't think that this is the source of my problem. The MOBOs of two hypervisors are identical, 1 3Com (manageable) switch connecting the two.</font></div>
<div><font face="arial, sans-serif"><br clear="none"></font></div><div><font face="arial, sans-serif">Thanks!</font></div><div><font face="arial, sans-serif">Thiago</font></div></div><div><br clear="none"><br clear="none">
<div><div>
On 25 October 2013 07:15, Darragh O'Reilly <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:dara2002-openstack@yahoo.com" target="_blank" href="mailto:dara2002-openstack@yahoo.com">dara2002-openstack@yahoo.com</a>></span> wrote:<br clear="none">
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex;">
Hi Thiago,<br clear="none">
<br clear="none">
you have configured DHCP to push out a MTU of 1400. Can you confirm that the 1400 MTU is actually getting out to the instances by running 'ip link' on them?<br clear="none">
<br clear="none">
There is an open problem where the veth used to connect the OVS and Linux bridges causes a performance drop on some kernels - <a rel="nofollow" shape="rect" target="_blank" href="https://bugs.launchpad.net/nova-project/+bug/1223267">https://bugs.launchpad.net/nova-project/+bug/1223267</a> . If you are using the LibvirtHybridOVSBridgeDriver VIF driver, can you try changing to LibvirtOpenVswitchDriver and repeat the iperf test between instances on different compute-nodes.<br clear="none">
<br clear="none">
What NICs (maker+model) are you using? You could try disabling any off-load functionality - 'ethtool -k <iface-used-for-gre>'.<br clear="none">
<br clear="none">
What kernal are you using: 'uname -a'?<br clear="none">
<br clear="none">
Re, Darragh.<br clear="none">
<div><br clear="none">
> Hi Daniel,<br clear="none">
<br clear="none">
><br clear="none">
> I followed that page, my Instances MTU is lowered by DHCP Agent but, same<br clear="none">
> result: poor network performance (internal between Instances and when<br clear="none">
> trying to reach the Internet).<br clear="none">
><br clear="none">
> No matter if I use "dnsmasq_config_file=/etc/neutron/dnsmasq-neutron.conf +<br clear="none">
> "dhcp-option-force=26,1400"" for my Neutron DHCP agent, or not (i.e. MTU =<br clear="none">
> 1500), the result is almost the same.<br clear="none">
><br clear="none">
> I'll try VXLAN (or just VLANs) this weekend to see if I can get better<br clear="none">
> results...<br clear="none">
><br clear="none">
> Thanks!<br clear="none">
> Thiago<br clear="none">
<br clear="none">
</div><div><div>_______________________________________________<br clear="none">
Mailing list: <a rel="nofollow" shape="rect" target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br clear="none">
Post to : <a rel="nofollow" shape="rect" ymailto="mailto:openstack@lists.openstack.org" target="_blank" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br clear="none">
Unsubscribe : <a rel="nofollow" shape="rect" target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br clear="none">
</div></div></blockquote></div><br clear="none"></div></div></div></div><br clear="none"><br clear="none"></div> </blockquote> </div> </div> </div> </div></div></div></div></blockquote></div></div><br clear="none"></div>
</div>
</div></div><br clear="none"><br clear="none"></div> </blockquote> </div> </div> </div> </div></div></div></div></blockquote></div><br clear="none"></div></div>
</div></div><br><br></div> </blockquote> </div> </div> </div> </div></body></html>