<div dir="ltr"><div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0)">Well, i have news.</div><div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0)">
<br></div><div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0)">On the compute nodes with 128GB of RAM, 2x1Gb bonded interfaces, 200Mb/s bandwidth and 10k packages/s going through each ethernet.</div>
<div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0)">Leaving this sysctl setting on the host :</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace"><br></font></div><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_max_tw_buckets = 3600000</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_max_syn_backlog = 30000</font></div></div><div class="gmail_default"><div class="gmail_default">
<font color="#000000" face="courier new, monospace">net.core.netdev_max_backlog = 50000</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.core.somaxconn = 16384</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.core.rmem_max = 16777216</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.core.wmem_max = 16777216</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_rmem = 4096 87380 16777216</font></div></div><div class="gmail_default"><div class="gmail_default">
<font color="#000000" face="courier new, monospace">net.ipv4.tcp_wmem = 4096 65536 16777216</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.core.rmem_default = 16777216</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.core.wmem_default = 16777216</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_congestion_control = cubic</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.ip_local_port_range = 1024 65000</font></div></div><div class="gmail_default"><div class="gmail_default">
<font color="#000000" face="courier new, monospace">net.ipv4.tcp_fin_timeout = 5</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_keepalive_time = 5</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_tw_recycle = 1</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_tw_reuse = 1</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">vm.swappiness = 0</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_syncookies = 1</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_timestamps = 1</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_max_orphans = 60000</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_synack_retries = 3</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_ecn=1</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_sack=1</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_dsack=1</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.route.flush = 1</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv6.route.flush = 1</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_udp_timeout = 30</font></div></div><div class="gmail_default"><div class="gmail_default">
<font color="#000000" face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60</font></div></div><div class="gmail_default"><div class="gmail_default">
<font color="#000000" face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_max = 1200000</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000</font></div>
</div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60</font></div></div><div class="gmail_default"><div class="gmail_default">
<font color="#000000" face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120</font></div></div><div class="gmail_default"><div class="gmail_default"><font color="#000000" face="courier new, monospace">net.ipv4.tcp_keepalive_time = 90</font></div>
</div></blockquote><font color="#000000" face="courier new, monospace"><div class="gmail_default" style="display:inline"><div style="color:rgb(0,0,0);font-family:'courier new',monospace;font-size:small"><font color="#000000" face="courier new, monospace"><div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0);display:inline">
<br></div></font></div>But setting both interfaces offload off (ethtool -k) and getting the ring higher from 256 (ethtool -G eth[0-1] rx 1024 tx 1024) the 20ms delays dissapeared from the HOST at least, we are now getting down to the VM to see if we can see the same result.</div>
<br></font><div><font color="#000000" face="courier new, monospace"><div class="gmail_default" style="display:inline"><br></div></font></div><div><font color="#000000" face="courier new, monospace"><div class="gmail_default" style="display:inline">
BUT!!!! </div></font></div><div><div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0)">On the compute nodes with 256GB of RAM, 2x1Gb bonded interfaces, 500/700Mb/s bandwidth and 30k packages/s going through each ethernet.</div>
<div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0)">Leaving the same sysctl settings as above, offload turned off on both interfaces, and tx/rx ring tested on 1024/2048, has showed no possitive result as the datanodes with less RAM/throughput.</div>
<div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0)">
Maybe some advice about sysctl kernel buffers ? maybe something that worth growing ?</div><div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0)">Thank you all.</div>
<div class="gmail_default" style="font-family:'courier new',monospace;font-size:small;color:rgb(0,0,0)"></div><br></div></div><div class="gmail_extra"><br clear="all"><div><div><font><b><br></b></font></div><div>
<font><b><img src="http://s14.postimage.org/sg1lztqep/cloudbuilders_Logo_last_small.png" width="96" height="58"><br></b></font></div><font><b>Alejandro Comisario <br>
#melicloud CloudBuilders</b></font><br>
<font color="#666666"><span style="font-size:6pt;color:gray" lang="ES">Arias 3751, Piso 7 (C1430CRG) <br>
Ciudad de Buenos Aires -
Argentina<br>
Cel: +549(11) 15-3770-1857<br>
Tel : +54(11) 4640-8443</span></font></div>
<br><br><div class="gmail_quote">On Thu, Jan 16, 2014 at 7:06 AM, George Shuklin <span dir="ltr"><<a href="mailto:george.shuklin@gmail.com" target="_blank">george.shuklin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Upgrade OVS to version >= 1.11. <br>
<br>
I don't know it will work with neutron or not, but OVS 1.10 (and
1.4, and any version <1.11) is just not production ready. <br>
<br>
Way to reproduce problem:<br>
<br>
hping3 --flood --rand-source ANY_FLOATING_IP.<br>
<br>
It kills any hosts with older OVS up to the level 'connection
timeout'. <br><div><div class="h5">
<br>
On 11.01.2014 21:12, Alejandro Comisario wrote:<br>
</div></div></div>
<blockquote type="cite"><div><div class="h5">
<div dir="ltr">
<div class="gmail_default">Well, its
been a long time since we use nova with KVM, we got over the
many thousand vms, and still, something doesnt feel right.</div>
<div class="gmail_default">We are using
ubuntu 12.04 kernel 3.2.0-[40-48], tuned sysctl with lots of
parameters, and everything ... works, you can say, quite well.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">
But here's the deal, we have an special networking scenario
that is, EVERYTHING IS APIS, everything is throughput, no
bandwidth.</div>
<div class="gmail_default">
Every 2x1Gb bonded compute node, doesnt get over the [200Mb/s
- 400Mb/s] but its handling hundreds of thousands requests per
minute to the vms.</div>
<div class="gmail_default">
<br>
</div>
<div class="gmail_default">And once in a
while, gives you the sensation that everything goes to hell,
timeouts from aplications over there, response times from apis
going from 10ms to 200ms over there, 20ms delays happening
between the vm ETH0 and the VNET interface, etc.</div>
<div class="gmail_default">So, since its
a massive scenario to tune, we never kinda, nailedon WHERE TO
give this 1, 2 or 3 final buffer/ring/affinity tune to make
everything work from the compute side.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">
I know its a little awkward, but im craving, and jaunting for
community real life examples regarding "HIGH THROUGHPUT"
tuning with KVM scenarios, dark linux or if someone can help
me go through configurations that might sound weird /
unnecesary / incorrect.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">
For those who are wondering, well ... i dont know what you
have, lets start with this.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">
COMPUTE NODES (99% of them, different vendors, but ...)</div>
<div class="gmail_default">* 128/256 GB
of ram</div>
<div class="gmail_default">
* 2 hexacores with HT enabled</div>
<div class="gmail_default">* 2x1Gb
bonded interfaces (want to know the more than 20 models we are
using, just ask for it)</div>
<div class="gmail_default">* Multi queue
interfaces, pined via irq to different cores</div>
<div class="gmail_default">
* ubuntu 12.04 kernel 3.2.0-[40-48]</div>
<div class="gmail_default">* Linux
bridges, no VLAN, no open-vswitch</div>
<div class="gmail_default">
<br>
</div>
<div class="gmail_default">I want to try
to keep the networking appliances ( TOR's, AGGR, CORES ) as
out of the picture as possible.</div>
<div class="gmail_default">im thinking
"i hope this thread gets great, in time"</div>
<div class="gmail_default">
<br>
</div>
<div class="gmail_default">So, ready to
learn as much as i can.</div>
<div class="gmail_default">
Thank you openstack community, as allways.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">
alejandrito</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><div class="im"><pre>_______________________________________________
OpenStack-operators mailing list
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a>
</pre>
</div></blockquote>
<br>
</div>
<br>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br></div>