<div dir="ltr">Hi Pangj,<div><br></div><div>First, make sure you have an updated version of swift-bench. There was a bug where it was relying on python-swiftclient to setup eventlet, but when eventlet was removed from swiftclient, that caused swift-bench to not run requests concurrently.</div>
<div><br></div><div>There are quite a few things that should be checked after that:</div><div><br></div><div>1. Check the logs to make sure there are no major errors. Swift is designed to work around failures, but failures will add some latency to requests. This can be anything from configuration issues, ring issues, connectivity issues, etc.</div>
<div><br></div><div>2. Once you have validated that the system is running correctly, observe the system while under load. I would start with the proxies and watch the CPU usage of the proxy processes to see if they are maxing out the CPU. Increase the workers if they are, or add more proxies if you are running out of total CPU. Do the same for the other services if needed.</div>
<div><br></div><div>3. I noticed that you have one server for accounts/containers. IO for container services can be a bit of a bottleneck for PUTs, since every object PUT is going to send an update to the container listing. If you start seeing timeouts in the object-server logs trying to do container updates, then some tweaking here might need to be done.</div>
<div><br></div><div>Those are some good starting points, and hope that helps you get further along.</div><div><br></div><div>--</div><div>Chuck</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Sep 2, 2013 at 6:51 AM, pangj <span dir="ltr"><<a href="mailto:pangj@laposte.net" target="_blank">pangj@laposte.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
We have benched the swift, the cluster is with one proxy server, one combined account/container server, five object servers. All are separated physical servers.<br>
<br>
We benched from 12 clients (each with 200 threads), got the results as below:<br>
<br>
| node | PUT | GET | DELETE |<br>
|--------+-------+-------+----<u></u>----|<br>
| sb2 A | 35.6 | 57.1 | 44 |<br>
| sb2 B | 35.6 | 56.1 | 43.4 |<br>
| sb3 A | 35.4 | 58.1 | 43.4 |<br>
| sb3 B | 35.6 | 56.1 | 43.4 |<br>
| ad61 A | 35.5 | 57.2 | 43.8 |<br>
| ad61 B | 35.5 | 55.2 | 43.5 |<br>
| sb64 A | 35.4 | 53.4 | 43.7 |<br>
| sb64 B | 35.5 | 54.9 | 43.5 |<br>
| sb65 A | 35.7 | 53.3 | 43.6 |<br>
| sb65 B | 35.6 | 52.5 | 44.1 |<br>
| sb27 A | 35.4 | 49.4 | 44.1 |<br>
| sb27 B | 35.4 | 51.1 | 43.9 |<br>
|--------+-------+-------+----<u></u>----|<br>
| total | 426.2 | 654.4 | 524.4 |<br>
<br>
<br>
The total IOPS for PUT is 426, means 426 PUT requests per second.<br>
But our application design requires 3000 PUT requests per second.<br>
So how to improve swift's performance to match the requirement?<br>
<br>
Thanks.<br>
<br>
______________________________<u></u>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
</blockquote></div><br></div>