Have you monitored the cpu utilization of the proxy server and the storage nodes? I did similar tests with Swift and the proxy server exhausted its capacity with a only few concurrent requests for very small objects.<div><br>
</div><div>If you notice object servers are not overloaded, but proxy is overloaded, a solution might be to have more proxy servers if you hav<br><div><br></div><div>It seems a problem of overload, since there are only 4 servers in the system and a large level of concurrency. Have you tried slowly increasing the number of concurrency to find the point where the problem starts? This point may be the capacity of your system.</div>
<div><br></div><div>Also, are you using persistent connections to the proxy server to send the object? If so, maybe try to renew them once in a while. </div><div><br></div><div>Cheers,</div><div><br>Paulo</div><div><br><div class="gmail_quote">
2012/7/20 Kuo Hugo <span dir="ltr"><<a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Sam , and all openstacker<div><br></div><div>This is Hugo . I'm facing an issue about the performance
<em style="line-height:16px;color:rgb(221,75,57);font-style:normal;font-family:arial,sans-serif">degradation</em> of swift . </div><div>I tried to figure out the problem of the issue which I faced in recent days. </div>
<div><br></div><div>Environment :</div><div>Swift version : master branch . latest code.</div><div>Tried on Ubuntu 12.04/11.10 </div><div>1 Swift-proxy : 32GB-ram / CPU 4*2 / 1Gb NIC*2</div><div>3 Storage-nodes : each for 32GB-ram / CPU 4*2 / 2TB*7 / 1Gb NIC*2</div>
<div><br></div><div>storage nodes runs only main workers(object-server , container-server , account-server)</div><div><br></div><div>I'm in testing with 4K size objects by swift-bench.</div><div><br></div><div>Per round bench.conf</div>
<div>object_size = 4096</div><div>Concurrency : 200</div><div>Object number: 200000</div><div>Containers : 200</div><div>no delete objects ..</div><div><br></div><div>At beginning , everything works fine in my environment. The average speed of PUT is reached to 1200/s . </div>
<div>After several rounds test . I found that the performance is down to 300~400/s </div><div>And after more rounds , failures appeared , and ERROR in proxy's log as followed </div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><div><font color="#999999">Jul 20 18:44:54 angryman-proxy-01 proxy-server ERROR with Object server <a href="http://192.168.100.101:36000/DISK5" target="_blank">192.168.100.101:36000/DISK5</a> re: Trying to get final status of PUT to /v1/AUTH_admin/9cbb3f9336b34019a6e7651adfc06a86_51/87b48a3474c7485c95aeef95c6911afb: </font><font color="#cc0000">Timeout (10s)</font><font color="#999999"> (txn: txb4465d895c9345be95d81632db9729af) (client_ip: 172.168.1.2)</font></div>
</div><div><div><font color="#999999">Jul 20 18:44:54 angryman-proxy-01 proxy-server ERROR with Object server <a href="http://192.168.100.101:36000/DISK4" target="_blank">192.168.100.101:36000/DISK4</a> re: Trying to get final status of PUT to /v1/AUTH_admin/9cbb3f9336b34019a6e7651adfc06a86_50/7405e5824cff411f8bb3ecc7c52ffd5a: </font><font color="#cc0000">Timeout (10s)</font><font color="#999999"> (txn: txe0efab51f99945a7a09fa664b821777f) (client_ip: 172.168.1.2)</font></div>
</div><div><div><font color="#999999">Jul 20 18:44:55 angryman-proxy-01 proxy-server ERROR with Object server <a href="http://192.168.100.101:36000/DISK5" target="_blank">192.168.100.101:36000/DISK5</a> re: Trying to get final status of PUT to /v1/AUTH_admin/9cbb3f9336b34019a6e7651adfc06a86_33/f322f4c08b124666bf7903812f4799fe: </font><font color="#cc0000">Timeout (10s)</font><font color="#999999"> (txn: tx8282ecb118434f828b9fb269f0fb6bd0) (client_ip: 172.168.1.2)</font></div>
</div></blockquote><div><br></div><div>After trace the code of object-server swift/obj/server.py and insert a timer on <a href="https://github.com/openstack/swift/blob/master/swift/obj/server.py#L591" target="_blank">https://github.com/openstack/swift/blob/master/swift/obj/server.py#L591</a></div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><span style="margin:0px;padding:0px;border:0px;font-weight:bold;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap"><br>
</span></div><div><span style="margin:0px;padding:0px;border:0px;font-weight:bold;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">for</span><span style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap;background-color:rgb(255,255,204)"> </span><span style="margin:0px;padding:0px;border:0px;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">chunk</span><span style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap;background-color:rgb(255,255,204)"> </span><span style="margin:0px;padding:0px;border:0px;font-weight:bold;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">in</span><span style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap;background-color:rgb(255,255,204)"> </span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">iter</span><span style="margin:0px;padding:0px;border:0px;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">(</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">lambda</span><span style="margin:0px;padding:0px;border:0px;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">:</span><span style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap;background-color:rgb(255,255,204)"> </span><span style="margin:0px;padding:0px;border:0px;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">reader</span><span style="margin:0px;padding:0px;border:0px;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">(</span><span style="margin:0px;padding:0px;border:0px;color:rgb(153,153,153);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">self</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">.</span><span style="margin:0px;padding:0px;border:0px;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">network_chunk_size</span><span style="margin:0px;padding:0px;border:0px;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">),</span><span style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap;background-color:rgb(255,255,204)"> </span><span style="margin:0px;padding:0px;border:0px;color:rgb(221,17,68);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">''</span><span style="margin:0px;padding:0px;border:0px;color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:13.333333969116211px;line-height:17.77777862548828px;white-space:pre-wrap">):</span></div>
</blockquote><div><br></div><div>Seems that the reader sometimes took a lot of time for receiving data from wsgi.input. Not every request , it looks like has a time of periods. </div><div><br></div><div>So that I check the history of Swift , I saw your commit <a href="https://github.com/openstack/swift/commit/783f16035a8e251d2138eb5bbaa459e9e4486d90" target="_blank">https://github.com/openstack/swift/commit/783f16035a8e251d2138eb5bbaa459e9e4486d90</a></div>
<div> . That's the only one which close to my issue. So that I hope that there's some suggestions for me. </div><div><br></div><div>My considerations :</div><div><br></div><div>1. Does it possible caused by greenio switch ?</div>
<div><br></div><div>2. Does it related to the number of objects existing on storage disks ? </div><div><br></div><div>3. Did someone play with swift by small size + fast client request ? </div><div><br></div><div>4. I found that the performance would never back to 1200/s . The only way to do is flush all data from disk. Once disk cleaned , the performance get back to the best one. </div>
<div><br></div><div>5. I re-read entire workflow of object server to handle a PUT request , I don't understand the reason why that the number of objects will affect reading wsgi.input data. With 4K size objects. no need to be chunked as I know. </div>
<div><br></div><div><br></div><div>The time consumed by <font color="#cc0000"><b><span style="font-size:13.333333969116211px;margin:0px;padding:0px;border:0px;font-family:Consolas,'Liberation Mono',Courier,monospace;line-height:17.77777862548828px;white-space:pre-wrap">reader</span><span style="font-size:13.333333969116211px;margin:0px;padding:0px;border:0px;font-family:Consolas,'Liberation Mono',Courier,monospace;line-height:17.77777862548828px;white-space:pre-wrap">(</span><span style="font-size:13.333333969116211px;margin:0px;padding:0px;border:0px;font-family:Consolas,'Liberation Mono',Courier,monospace;line-height:17.77777862548828px;white-space:pre-wrap">self</span><span style="font-size:13.333333969116211px;margin:0px;padding:0px;border:0px;font-family:Consolas,'Liberation Mono',Courier,monospace;line-height:17.77777862548828px;white-space:pre-wrap">.</span><span style="font-size:13.333333969116211px;margin:0px;padding:0px;border:0px;font-family:Consolas,'Liberation Mono',Courier,monospace;line-height:17.77777862548828px;white-space:pre-wrap">network_chunk_size</span><span style="font-size:13.333333969116211px;margin:0px;padding:0px;border:0px;font-family:Consolas,'Liberation Mono',Courier,monospace;line-height:17.77777862548828px;white-space:pre-wrap">)</span></b></font></div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font color="#999999">Jul 20 17:09:36 angryman-storage-01 object-server Reader: 0.001391</font></div>
</blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font color="#999999">Jul 20 17:09:36 angryman-storage-01 object-server Reader: 0.001839</font></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><font color="#999999">Jul 20 17:09:36 angryman-storage-01 object-server Reader: 0.00164</font></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font color="#999999">Jul 20 17:09:36 angryman-storage-01 object-server Reader: 0.002786</font></div>
</blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font color="#cc0000">Jul 20 17:09:36 angryman-storage-01 object-server Reader: 2.716707</font></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><font color="#cc0000">Jul 20 17:09:36 angryman-storage-01 object-server Reader: 1.005659</font></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font color="#999999">Jul 20 17:09:36 angryman-storage-01 object-server Reader: 0.055982</font></div>
</blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font color="#999999">Jul 20 17:09:36 angryman-storage-01 object-server Reader: 0.002205</font></div></blockquote></blockquote><div><br></div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font color="#999999">Jul 20 18:39:14 angryman-storage-01 object-server WTF: 0.000968</font></div>
</div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font color="#999999">Jul 20 18:39:14 angryman-storage-01 object-server WTF: 0.001328</font></div></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><div><font color="#cc0000">Jul 20 18:39:14 angryman-storage-01 object-server WTF: 10.003368</font></div></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font color="#999999">Jul 20 18:39:14 angryman-storage-01 object-server WTF: 0.001243</font></div>
</div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font color="#999999">Jul 20 18:39:14 angryman-storage-01 object-server WTF: 0.001562</font></div></div></blockquote></blockquote>
<div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font color="#999999">Jul 20 17:52:41 angryman-storage-01 object-server WTF: 0.001067</font></div>
</div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font color="#cc0000">Jul 20 17:52:41 angryman-storage-01 object-server WTF: 13.804413</font></div></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><div><font color="#cc0000">Jul 20 17:52:41 angryman-storage-01 object-server WTF: 5.301166</font></div></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font color="#999999">Jul 20 17:52:41 angryman-storage-01 object-server WTF: 0.001167</font></div>
</div></blockquote></blockquote><div><br></div><div><br></div><div> </div><div>Would it be a bug of eventlet or SWIFT ? Please feel free to let me know that should I file a bug for Swift . </div><div><br></div><div>Appreciate ~</div>
<span class="HOEnZb"><font color="#888888">
<div><div><br></div>-- <br><div>+Hugo Kuo+</div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com<br></a></div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">+</a>886 935004793</div><br>
</div>
</font></span><br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Paulo Ricardo</div><div><br></div>-- <br><span>European Master in Distributed Computing<i></i></span><span style="font-family:arial,sans-serif;line-height:15px"><i style="font-style:normal"><br>
Royal Institute of Technology - <i style="font-style:normal">KTH</i><br></i></span><div><span style="font-family:arial,sans-serif;line-height:15px"><i style="font-style:normal"><i style="font-style:normal">Instituto Superior Técnico - IST</i></i></span></div>
<div><span style="font-family:arial,sans-serif;line-height:15px"><i style="font-style:normal"><i style="font-style:normal"><a href="http://paulormg.com" target="_blank">http://paulormg.com</a></i></i></span></div><br>
</div></div>