<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 07/13/2012 06:55 AM, Leandro Reox
wrote:<br>
</div>
<blockquote
cite="mid:CAC9jWxPmg=nEptdEbxwxi0khWHLE=fqSeqHATiUBQG1BQQGLew@mail.gmail.com"
type="cite">
<div class="km" style="margin-left: 1em; color: rgb(34, 34, 34);
font-family: arial,sans-serif; font-size: 13px;
background-color: rgb(255, 255, 255);">
<div class="kk" style="margin-bottom:0.2em">Ok, here is the
story, we deployed some inhouse APIs in our Openstack privade
cloud, and we were stressing them up, we realize that some
packages were taking so long, to discard the behavior of the
api, we installed apache, lighttpd and event tried with
netcat, of course on the guest systems running ubuntu 10.10
w/virtio, after getting nuts modifing sysctl parameters to
change the guest behavior, we realized that if we installed
apache, or lighttpd on the PHYSICAL host the behavior was the
same ...., that surprised us, when we try the same benchmark
on a node without bonding, bridging and without any KVM
packages or nova installed, with the same HW specs, the
benchmark passes OK, but if we run the same tests on a spare
nova node with everything installed + bonding + bridging that
never run a virtual guest machine, the test fails too, so, so
far:</div>
<div class="kk" style="margin-bottom:0.2em"><br>
</div>
<div class="kk" style="margin-bottom:0.2em">Tested on hosts with
Ubuntu 10.10, 11.10 and 12.04</div>
<div class="kk" style="margin-bottom:0.2em"><br>
</div>
<div class="kk" style="margin-bottom:0.2em">
- Clean node without bonding + briding or KVM - just the eth0
configured - PASS</div>
<div class="kk" style="margin-bottom:0.2em">- Spare node with
bridging - PASS</div>
<div class="kk" style="margin-bottom:0.2em">- Spare node with
just bonding (dynamic link aggr mode4) - PASS</div>
<div class="kk" style="margin-bottom:0.2em">- Spare node with
nova + kvm + bonding + bridging - FAILS</div>
<div class="kk" style="margin-bottom:0.2em">- Spare node with
nova + kvm - PASS</div>
<div class="kk" style="margin-bottom:0.2em">
<br>
</div>
<div class="kk" style="margin-bottom:0.2em">Is there a chance
that working with bridging + bonding + nova some module get
screwed, ill attach the tests , you can see that a small
amount of packages takes TOO LONG, like 3secs, and the
overhead time is on the "CONNECT" phase</div>
</div>
</blockquote>
<br>
If I recall correctly, 3 seconds is the default, initial TCP
retransmission timeout (at least in older kernels - what is your
load generator running?). Between that, and your mentioning connect
phase, my first guess (it is only a guess) would be that something
is causing TCP SYNchronize segments to be dropped. If that is the
case, it should show-up in netstat -s statistics. Snap them on both
client and server before the test is started, and after the test is
completed, and then run them through something like beforeafter (
<a class="moz-txt-link-freetext" href="ftp://ftp.cup.hp.com/dist/networking/tools">ftp://ftp.cup.hp.com/dist/networking/tools</a> )<br>
<br>
netstat -s > before.server<br>
# run benchmark<br>
netstat -s > after.server<br>
beforeafter before.server after.server > delta.server<br>
less delta.server<br>
<br>
(As a sanity check, make certain that before.server and after.server
have the same number of lines. The habit of Linux's netstat to avoid
printing a statistic with a value of zero can, sometimes, confuse
beforeafter if a stat appears in after that was not present in
before.)<br>
<br>
It might not be a bad idea to include ethtool -S statistics from
each of the interfaces in that procedure as well.<br>
<br>
rick jones<br>
probably a good idea to mention the bonding mode you are using<br>
<br>
<blockquote
cite="mid:CAC9jWxPmg=nEptdEbxwxi0khWHLE=fqSeqHATiUBQG1BQQGLew@mail.gmail.com"
type="cite">
<div class="km"
style="margin-left:1em;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<div class="kk" style="margin-bottom:0.2em"><span dir="ltr"
id=":zu">This is ApacheBench, Version 2.3 <$Revision:
655654 $><br>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, <a
moz-do-not-send="true" href="http://www.zeustech.net/"
style="color:rgb(0,101,204)">http://www.zeustech.net/</a><br>
Licensed to The Apache Software Foundation, <a
moz-do-not-send="true" href="http://www.apache.org/"
style="color:rgb(0,101,204)">http://www.apache.org/</a><br>
<br>
Benchmarking 172.16.161.25 (be patient)<br>
Completed 2500 requests<br>
Completed 5000 requests<br>
Completed 7500 requests<br>
Completed 10000 requests<br>
Completed 12500 requests<br>
Completed 15000 requests<br>
Completed 17500 requests<br>
Completed 20000 requests<br>
Completed 22500 requests<br>
Completed 25000 requests<br>
Finished 25000 requests<br>
<br>
<br>
Server Software: Apache/2.2.16<br>
Server Hostname: 172.16.161.25<br>
Server Port: 80<br>
<br>
Document Path: /<br>
Document Length: 177 bytes<br>
<br>
Concurrency Level: 5<br>
Time taken for tests: 7.493 seconds<br>
Complete requests: 25000<br>
Failed requests: 0<br>
Write errors: 0<br>
Total transferred: 11350000 bytes<br>
HTML transferred: 4425000 bytes<br>
Requests per second: 3336.53 [#/sec] (mean)<br>
Time per request: 1.499 [ms] (mean)<br>
Time per request: 0.300 [ms] (mean, across all
concurrent requests)<br>
Transfer rate: 1479.28 [Kbytes/sec] received<br>
<br>
Connection Times (ms)<br>
min mean[+/-sd] median max<br>
Connect: 0 1 46.6 0 3009<br>
Processing: 0 1 5.7 0 277<br>
Waiting: 0 0 4.6 0 277<br>
Total: 0 1 46.9 1 3010<br>
<br>
Percentage of the requests served within a certain time (ms)<br>
50% 1<br>
66% 1<br>
75% 1<br>
80% 1<br>
90% 1<br>
95% 1<br>
98% 1<br>
99% 1<br>
100% 3010 (longest request)<br>
</span></div>
<div class="kk" style="margin-bottom:0.2em"><span dir="ltr"><br>
</span></div>
<div class="kk" style="margin-bottom:0.2em">
Regards!</div>
<div><span dir="ltr"><br>
</span></div>
<div><span dir="ltr"><br>
</span></div>
<div><span dir="ltr"><br>
</span></div>
</div>
<div class="kq"
style="margin-bottom:0.2em;color:rgb(119,119,119);margin-left:1em;font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)"></div>
</blockquote>
<br>
<br>
</body>
</html>