Open Stack

That helped -- using tempauth instead of keystone improved performance significantly on the openStack cluster.
However performance is still slower compared to my Swift only setup where commands are sent directly to the swift node instead of going thru the controller node in the openstack  cluster.
How does controller and swift node communications affect swift performance?
I've also noticed that all objects are stored on the 2nd swift node and not the 1st on the openstack cluster. I'm wondering if that could also be a factor in slow performance.

Keystone:
TOTAL
       Count:    50  Average requests per second:   9.2
                            min       max      avg      std_dev  95%-ile                   Worst latency TX ID
       First-byte latency:  0.067 -   2.513    0.390  (  0.604)    1.948  (all obj sizes)  txae75691d37d544b4ac0cfe3b8cba7f38
       Last-byte  latency:  0.067 -   3.337    0.430  (  0.695)    1.997  (all obj sizes)  txdcedb82227654b338daa85751f6d1232
       First-byte latency:  0.070 -   2.513    0.542  (  0.749)    2.255  (    tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
       Last-byte  latency:  0.070 -   2.514    0.468  (  0.659)    1.997  (    tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
       First-byte latency:  0.067 -   1.884    0.251  (  0.382)    0.695  (   small objs)  tx2ceec827f3304530b01a0d5993eea2e8
       Last-byte  latency:  0.067 -   3.337    0.385  (  0.732)    1.884  (   small objs)  txdcedb82227654b338daa85751f6d1232

Tempauth:
       Count:    50  Average requests per second:  65.7
                            min       max      avg      std_dev  95%-ile                   Worst latency TX ID
       First-byte latency:  0.006 -   0.073    0.014  (  0.015)    0.055  (all obj sizes)  tx69bf033a246645808b2c6a280e334f15
       Last-byte  latency:  0.006 -   0.248    0.047  (  0.070)    0.198  (all obj sizes)  txb8cf5dc0ce264eb08a1e05edbbf5a40f
       First-byte latency:  0.006 -   0.073    0.017  (  0.020)    0.072  (    tiny objs)  tx69bf033a246645808b2c6a280e334f15
       Last-byte  latency:  0.006 -   0.248    0.053  (  0.072)    0.195  (    tiny objs)  txb8cf5dc0ce264eb08a1e05edbbf5a40f
       First-byte latency:  0.006 -   0.026    0.010  (  0.005)    0.026  (   small objs)  tx65d1fd4b6ae049bb902442ac4c28ffe9
       Last-byte  latency:  0.006 -   0.218    0.040  (  0.066)    0.198  (   small objs)  txbfd6ebc74ed04068affd17c123572a44

Swift Only:
TOTAL
       Count:    50  Average requests per second: 397.0
                            min       max      avg      std_dev  95%-ile                   Worst latency TX ID
       First-byte latency:  0.003 -   0.007    0.005  (  0.001)    0.006  (all obj sizes)  None
       Last-byte  latency:  0.003 -   0.046    0.008  (  0.009)    0.029  (all obj sizes)  None
       First-byte latency:  0.003 -   0.007    0.005  (  0.001)    0.007  (    tiny objs)  None
       Last-byte  latency:  0.003 -   0.046    0.008  (  0.010)    0.027  (    tiny objs)  None
       First-byte latency:  0.004 -   0.006    0.005  (  0.001)    0.006  (   small objs)  None
       Last-byte  latency:  0.004 -   0.043    0.008  (  0.009)    0.029  (   small objs)  None

From: Chmouel Boudjnah [mailto:chmouel at enovance.com]
Sent: Tuesday, July 09, 2013 8:22 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] Swift debugging / performance - large latencies seen.

On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim <Tim.Snider at netapp.com<mailto:Tim.Snider at netapp.com>> wrote:
I have 2 openstack clusters running the Folsom release with multiple Swift nodes. I also have a small setup that is running only Swift with a single node.  I'm noticing very large Swift I/O latencies (seconds long) on the openstack clusters - ssbench output snippet is below. Performance is approximately identical on the openstack clusters. The Swift only cluster performs much better.

Keystone performance can be pretty awful unless you are using something else than the default WSGI container configuration (single process eventlet I think). I would suggest you try to run it under apache with multiple process.

See the dicussion at last summit about Keystone performance here :

https://etherpad.openstack.org/havana-keystone-performance

Chmouel.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130709/4cbc4a11/attachment-0001.html>