<div dir="ltr"><div>Three things.</div><div><br></div><div>1) make your runs last longer than a few seconds, a sustained load over a longer period will average out some of jitter..</div><div><br></div>2) You don't have enough log lines. Is this a single replica cluster? How does that work?<div>
<br></div><div>3) If you're benchmarking with the background daemons on, you might try turning them off as they're going to introduce some unpredictability between runs - it may be more realistic, but you'll need ever *longer* runs to smooth out the spotty load over time.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Aug 9, 2013 at 7:17 AM, Snider, Tim <span dir="ltr"><<a href="mailto:Tim.Snider@netapp.com" target="_blank">Tim.Snider@netapp.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal">When running ssbench I’ve consistently gotten inconsistent results – 67% difference between the fastest and slowest CREATE times. Nothing (scenario, parameters, cluster, etc…) was varied between the runs. I’m using tempAuth and not Keystone
authentication.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Ssbench was run on a node using the internal (10GbE) network that isn’t currently part of the Swift cluster.<u></u><u></u></p>
<p class="MsoNormal">Is this commonly seen with Swift? <u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">CREATE Count: 406 ( 0 error; 0 retries: 0.00%) Average requests per second: 79.6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">CREATE Count: 389 ( 0 error; 0 retries: 0.00%) Average requests per second: 99.8<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">CREATE Count: 412 ( 0 error; 0 retries: 0.00%) Average requests per second: 65.2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">CREATE Count: 416 ( 0 error; 0 retries: 0.00%) Average requests per second: 58.6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">CREATE Count: 406 ( 0 error; 0 retries: 0.00%) Average requests per second: 41.6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">CREATE Count: 409 ( 0 error; 0 retries: 0.00%) Average requests per second: 71.3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">CREATE Count: 395 ( 0 error; 0 retries: 0.00%) Average requests per second: 32.3<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">/var/log/swift entries for a PUT with a large latency doesn’t seem to reveal much:<u></u><u></u></p>
<p class="MsoNormal">root@swift21:/home/swift/bin# grep txeaf22b65bbc442d091373399a931340e /var/log/swift/*<u></u><u></u></p>
<p class="MsoNormal">/var/log/swift/container.log:Aug 9 06:47:40 swift21 container-server 192.168.10.208 - - [09/Aug/2013:13:47:40 +0000] "PUT /sdc/8738/AUTH_test/ssbench_000074/100k.1_000180" 201 - "txeaf22b65bbc442d091373399a931340e" "-" "-" 0.0005<u></u><u></u></p>
<p class="MsoNormal">/var/log/swift/object.log:Aug 9 06:47:40 swift21 object-server 192.168.10.209 - - [09/Aug/2013:13:47:40 +0000] "PUT /sdd/12336/AUTH_test/ssbench_000074/100k.1_000180" 201 - "-" "txeaf22b65bbc442d091373399a931340e" "-" 5.3650<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’d like to investigate the possible source of the large result variation and requests with large latencies. Suggestions on where / what to look at/for are appreciated.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Details in: <a href="http://paste.openstack.org/show/43703" target="_blank">
http://paste.openstack.org/show/43703</a> <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Tim<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>