<div dir="ltr"><div class="gmail_default" style="font-size:small;font-family:courier new,monospace">Like i said, while 12 vms are runing on this hosts, im conducting this test only from the hosts, not the vms, since i can see the delay on the host side already.</div>
<div class="gmail_default" style="font-size:small;font-family:courier new,monospace">I really want to make the delay go away on the host first, and once solved, going down the vnet -> vm:eth0.</div><div class="gmail_default" style="font-size:small;font-family:courier new,monospace">
<br></div><div class="gmail_default" style="font-size:small;font-family:courier new,monospace">Obviously the ethX tuning (ethtool are done on the host) but the sysctl tuning is done on the host and the guest also.</div>
<div class="gmail_default" style="font-size:small;font-family:courier new,monospace">I was thinking about the bonding being one culprit, but i want to test buffer settings first (queues and ring parameters).</div><div class="gmail_default" style="font-size:small;font-family:courier new,monospace">
regarding bonding, what kind of problems you had ? are you now running direct eth0 -> br100 settings ? </div><div class="gmail_default" style="font-size:small;font-family:courier new,monospace"><br></div><div class="gmail_default" style="font-size:small;font-family:courier new,monospace">
Is it vhost_net any good ? We are just using virtio.</div><div class="gmail_default" style="font-size:small;font-family:courier new,monospace"><br></div><div class="gmail_default" style="font-size:small;font-family:courier new,monospace">
thanks!</div><div class="gmail_extra"><br clear="all"><div><div><font><b><br></b></font></div><div><br></div></div>
<br><br><div class="gmail_quote">On Wed, Jan 15, 2014 at 1:32 PM, Narayan Desai <span dir="ltr"><<a href="mailto:narayan.desai@gmail.com" target="_blank">narayan.desai@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Are you using virtio, and vhost_net?<div><br></div><div>Also, where are you tuning those parameters, host or guest? The ethernet level ones will definitely need to be done in the host, but the TCP and socket buffer ones need to be in the guest. </div>
<div><br></div><div>Also, these buffers may be too large for 2x1ge. You might also check if the link aggregation is messing you up here. I've generally had problems with it. </div><div><br></div><div>One last thing: how does the app run from the hypervisor? You can rule a lot of things out by testing that.</div>
<div> -nld</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 15, 2014 at 9:58 AM, Alejandro Comisario <span dir="ltr"><<a href="mailto:alejandro.comisario@mercadolibre.com" target="_blank">alejandro.comisario@mercadolibre.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div style="font-size:small;font-family:'courier new',monospace">Hi Narayan, thanks fir the prompted response, let me try to give you some insights.</div>
<div style="font-size:small;font-family:'courier new',monospace">
Since like you said, our setup can reach the maximum bandwidth in some tests, but we cant achieve the THROUGHPUT we want, we run avg of 14vms on 128GB of ram compute nodes, and while all those vms are runing, we run a test between two compute nodes with a c++ aplication that sends 50 packets per second (lower than our 1500bytes MTU) and waits from the response from the target server lower than 10ms.</div>
<div style="font-size:small;font-family:'courier new',monospace">This test runs on the br100 interface on both compute nodes (passign through eth0-eth1, bond0 and br100) and while all vms are running (using high throughput low bandwidth applications) we see this simple tests showing thenths of thousands responses higher than 10ms, actually 99% of this slow responses are taking 20/21ms, i dont seem to find whats that magic delay value means, we are starting to look traversing what interfaces are adding the delay.</div>
<div style="font-size:small;font-family:'courier new',monospace"><br></div><div style="font-size:small;font-family:'courier new',monospace">Let me show you what our settings look like regarding networking (i will take the vms out of the picture)</div>
<div style="font-size:small;font-family:'courier new',monospace"><br></div><div style="font-size:small;font-family:'courier new',monospace">COMPUTE HOST</div>
<div style="font-size:small;font-family:'courier new',monospace">------------</div><div style="font-size:small;font-family:'courier new',monospace">2x1Gb bonded interfaces (no jumbo frames, 1500MTU since jumbo frames are a separate project)</div>
<div style="font-size:small;font-family:'courier new',monospace"><br></div><div style="font-size:small;font-family:'courier new',monospace">Ethernet ring settings on both interfaces:</div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div style="font-size:small;font-family:'courier new',monospace">RX 256</div><div style="font-size:small;font-family:'courier new',monospace">
TX 256</div></blockquote><div style="font-size:small;font-family:'courier new',monospace"><br></div><div style="font-size:small;font-family:'courier new',monospace">
Ethernet txqueuelen on both interfaces:</div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div style="font-size:small;font-family:'courier new',monospace">txqueuelen 1000</div>
</blockquote><div style="font-size:small;font-family:'courier new',monospace"><br></div><div style="font-size:small;font-family:'courier new',monospace">sysctl settings:</div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font face="courier new, monospace">net.ipv4.tcp_max_tw_buckets = 3600000</font></div></div><div>
<div><font face="courier new, monospace">net.ipv4.tcp_max_syn_backlog = 30000</font></div></div><div><div><font face="courier new, monospace">net.core.netdev_max_backlog = 50000</font></div>
</div><div><div><font face="courier new, monospace">net.core.somaxconn = 16384</font></div></div><div><div><font face="courier new, monospace">net.core.rmem_max = 16777216</font></div>
</div><div><div><font face="courier new, monospace">net.core.wmem_max = 16777216</font></div></div><div><div><font face="courier new, monospace">net.ipv4.tcp_rmem = 4096 87380 16777216</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.tcp_wmem = 4096 65536 16777216</font></div></div><div><div><font face="courier new, monospace">net.core.rmem_default = 16777216</font></div>
</div><div><div><font face="courier new, monospace">net.core.wmem_default = 16777216</font></div></div><div><div><font face="courier new, monospace">net.ipv4.tcp_congestion_control = cubic</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.ip_local_port_range = 1024 65000</font></div></div><div><div><font face="courier new, monospace">net.ipv4.tcp_fin_timeout = 5</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.tcp_keepalive_time = 5</font></div></div><div><div><font face="courier new, monospace">net.ipv4.tcp_tw_recycle = 1</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.tcp_tw_reuse = 1</font></div></div><div><div><font face="courier new, monospace">vm.swappiness = 0</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.tcp_syncookies = 1</font></div></div><div><div><font face="courier new, monospace">net.ipv4.tcp_timestamps = 1</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.tcp_max_orphans = 60000</font></div></div><div><div><font face="courier new, monospace">net.ipv4.tcp_synack_retries = 3</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.tcp_ecn=1</font></div></div><div><div><font face="courier new, monospace">net.ipv4.tcp_sack=1</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.tcp_dsack=1</font></div></div><div><div><font face="courier new, monospace">net.ipv4.route.flush = 1</font></div>
</div><div><div><font face="courier new, monospace">net.ipv6.route.flush = 1</font></div></div><div><div><font face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_udp_timeout = 30</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10</font></div></div><div><div>
<font face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120</font></div></div><div><div><font face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_max = 1200000</font></div></div><div><div><font face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000</font></div>
</div><div><div><font face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60</font></div></div><div><div>
<font face="courier new, monospace">net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120</font></div></div><div><div><font face="courier new, monospace">net.ipv4.tcp_keepalive_time = 90</font></div>
<div><font face="courier new, monospace"><br></font></div></div></blockquote><font face="courier new, monospace"><div style="font-size:small;display:inline;font-family:'courier new',monospace">
One other tip i can add is that allways the delay is on the RX side, this means, the server responding.</div></font><div><font face="courier new, monospace"><font color="#000000"><div style="font-size:small;display:inline;font-family:'courier new',monospace">
So, we were thinking about going upper with ring or txqueuelen settings.</div></font></font></div><div><font face="courier new, monospace"><font color="#000000"><div style="font-size:small;display:inline;font-family:'courier new',monospace">
<br></div></font></font></div><div><font face="courier new, monospace"><font color="#000000"><div style="font-size:small;display:inline;font-family:'courier new',monospace">
Any idea ?</div>
<br></font></font><div><div style="font-family:'courier new',monospace;font-size:small"><br></div></div>
</div></div><div class="gmail_extra"><br clear="all"><div><div><font><b><br></b></font></div><div><font><b><img src="http://s14.postimage.org/sg1lztqep/cloudbuilders_Logo_last_small.png" width="96" height="58"><span><font color="#888888"><br>
</font></span></b></font></div><span><font color="#888888">
<font><b>Alejandro Comisario <br>
#melicloud CloudBuilders</b></font><br>
<font color="#666666"><span style="font-size:6pt;color:gray" lang="ES">Arias 3751, Piso 7 (C1430CRG) <br>
Ciudad de Buenos Aires -
Argentina<br>
Cel: +549(11) 15-3770-1857<br>
Tel : <a href="tel:%2B54%2811%29%204640-8443" value="+541146408443" target="_blank">+54(11) 4640-8443</a></span></font></font></span></div><div><div>
<br><br><div class="gmail_quote">On Wed, Jan 15, 2014 at 12:32 AM, Narayan Desai <span dir="ltr"><<a href="mailto:narayan.desai@gmail.com" target="_blank">narayan.desai@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">We don't have a workload remotely like that (generally, we have a lot more demand for bandwidth, but we also generally run faster networks than that as well), but 1k pps sounds awfully low. Like low by several orders of magnitude.<div>
<br></div><div>I didn't measure pps in our benchmarking, but did manage to saturate a 10GE link from a VM (actually we did this on 10 nodes at a time to saturate a 100GE wide area link), and all of those settings are here:</div>
<div><a href="http://buriedlede.blogspot.com/2012/11/driving-100-gigabit-network-with.html" target="_blank">http://buriedlede.blogspot.com/2012/11/driving-100-gigabit-network-with.html</a></div><div><br></div><div>I'd start trying to do some fault isolation; see if you can get NAT out of the mix, for example, or see if it is a network stack tuning problem. You probably need to crank up some of your buffer sizes, even if you don't need to mess with your TCP windows. </div>
<div><br></div><div>Can you actually saturate your 2x1ge lag with bandwidth? (single or ganged flows?)</div><div> -nld</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Tue, Jan 14, 2014 at 3:52 PM, Alejandro Comisario <span dir="ltr"><<a href="mailto:alejandro.comisario@mercadolibre.com" target="_blank">alejandro.comisario@mercadolibre.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div style="font-size:small;font-family:courier new,monospace">
Wow, its kinda hard to imagine we are the only ones that have only 100Mb/s bandwidth but 50.000 requests per minute on each compute, i mean, lots of throughput, almost none bandwith.</div>
<div style="font-size:small;font-family:courier new,monospace"><br></div><div style="font-size:small;font-family:courier new,monospace">Everyone has their networking performance figured out ?</div>
<div style="font-size:small;font-family:courier new,monospace">No one to share some "SUPER THROUGHPUT" sysctl / ethtool / power / etc settings on the compute side ?</div><div style="font-size:small;font-family:courier new,monospace">
<br></div><div style="font-size:small;font-family:courier new,monospace">Best regards.</div><div class="gmail_extra"><br clear="all"><div><div><font><b><div style="font-size:small;display:inline;font-family:'courier new',monospace">
alejandrito</div></b></font></div></div><div><div><br><div class="gmail_quote">On Sat, Jan 11, 2014 at 4:12 PM, Alejandro Comisario <span dir="ltr"><<a href="mailto:alejandro.comisario@mercadolibre.com" target="_blank">alejandro.comisario@mercadolibre.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div style="font-size:small;font-family:'courier new',monospace">Well, its been a long time since we use nova with KVM, we got over the many thousand vms, and still, something doesnt feel right.</div>
<div style="font-size:small;font-family:'courier new',monospace">We are using ubuntu 12.04 kernel 3.2.0-[40-48], tuned sysctl with lots of parameters, and everything ... works, you can say, quite well.</div>
<div style="font-size:small;font-family:'courier new',monospace"><br></div><div style="font-size:small;font-family:'courier new',monospace">
But here's the deal, we have an special networking scenario that is, EVERYTHING IS APIS, everything is throughput, no bandwidth.</div><div style="font-size:small;font-family:'courier new',monospace">
Every 2x1Gb bonded compute node, doesnt get over the [200Mb/s - 400Mb/s] but its handling hundreds of thousands requests per minute to the vms.</div><div style="font-size:small;font-family:'courier new',monospace">
<br></div><div style="font-size:small;font-family:'courier new',monospace">And once in a while, gives you the sensation that everything goes to hell, timeouts from aplications over there, response times from apis going from 10ms to 200ms over there, 20ms delays happening between the vm ETH0 and the VNET interface, etc.</div>
<div style="font-size:small;font-family:'courier new',monospace">So, since its a massive scenario to tune, we never kinda, nailedon WHERE TO give this 1, 2 or 3 final buffer/ring/affinity tune to make everything work from the compute side.</div>
<div style="font-size:small;font-family:'courier new',monospace"><br></div><div style="font-size:small;font-family:'courier new',monospace">
I know its a little awkward, but im craving, and jaunting for community real life examples regarding "HIGH THROUGHPUT" tuning with KVM scenarios, dark linux or if someone can help me go through configurations that might sound weird / unnecesary / incorrect.</div>
<div style="font-size:small;font-family:'courier new',monospace"><br></div><div style="font-size:small;font-family:'courier new',monospace">
For those who are wondering, well ... i dont know what you have, lets start with this.</div><div style="font-size:small;font-family:'courier new',monospace"><br></div><div style="font-size:small;font-family:'courier new',monospace">
COMPUTE NODES (99% of them, different vendors, but ...)</div><div style="font-size:small;font-family:'courier new',monospace">* 128/256 GB of ram</div><div style="font-size:small;font-family:'courier new',monospace">
* 2 hexacores with HT enabled</div><div style="font-size:small;font-family:'courier new',monospace">* 2x1Gb bonded interfaces (want to know the more than 20 models we are using, just ask for it)</div>
<div style="font-size:small;font-family:'courier new',monospace">* Multi queue interfaces, pined via irq to different cores</div><div style="font-size:small;font-family:'courier new',monospace">
* ubuntu 12.04 kernel 3.2.0-[40-48]</div><div style="font-size:small;font-family:'courier new',monospace">* Linux bridges, no VLAN, no open-vswitch</div><div style="font-size:small;font-family:'courier new',monospace">
<br></div><div style="font-size:small;font-family:'courier new',monospace">I want to try to keep the networking appliances ( TOR's, AGGR, CORES ) as out of the picture as possible.</div>
<div style="font-size:small;font-family:'courier new',monospace">im thinking "i hope this thread gets great, in time"</div><div style="font-size:small;font-family:'courier new',monospace">
<br></div><div style="font-size:small;font-family:'courier new',monospace">So, ready to learn as much as i can.</div><div style="font-size:small;font-family:'courier new',monospace">
Thank you openstack community, as allways.</div><div style="font-size:small;font-family:'courier new',monospace"><br></div><div style="font-size:small;font-family:'courier new',monospace">
alejandrito</div><div><br></div>
</div>
</blockquote></div><br></div></div></div></div>
<br></div></div><div>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></div></blockquote></div><br></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>