Open Stack

I have 2 openstack clusters running the Folsom release with multiple Swift nodes. I also have a small setup that is running only Swift with a single node.  I'm noticing very large Swift I/O latencies (seconds long) on the openstack clusters - ssbench output snippet is below. Performance is approximately identical on the openstack clusters. The Swift only cluster performs much better.
Setup differences:
Openstack clusters are using Keystone authentication - Swift only setup uses temp auth.
Multiple Swift nodes on openstack clusters - Swift only has single node.

I also see that all object I/Os are being sent to the 2nd Swift node, there are no objects on the 1st swift node. Both nodes are running swift-proxy services.
I have LOG_LOCAL0 set in the environment and also logging enabled in /etc/swift/swift.conf, but haven't seen any log entries made.

What can I look at to debug the cause of the excessive latencies on both of these stacks?
I'd also like to determine why all objects are on the 2nd swift node and none on the 1st node.

I'm supposed to do performance evaluations but need to fix the latency problem first.

Ssbench output snippet:
TOTAL
       Count:    50  Average requests per second:   9.2
                            min       max      avg      std_dev  95%-ile                   Worst latency TX ID
       First-byte latency:  0.067 -   2.513    0.390  (  0.604)    1.948  (all obj sizes)  txae75691d37d544b4ac0cfe3b8cba7f38
       Last-byte  latency:  0.067 -   3.337    0.430  (  0.695)    1.997  (all obj sizes)  txdcedb82227654b338daa85751f6d1232
       First-byte latency:  0.070 -   2.513    0.542  (  0.749)    2.255  (    tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
       Last-byte  latency:  0.070 -   2.514    0.468  (  0.659)    1.997  (    tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
       First-byte latency:  0.067 -   1.884    0.251  (  0.382)    0.695  (   small objs)  tx2ceec827f3304530b01a0d5993eea2e8
       Last-byte  latency:  0.067 -   3.337    0.385  (  0.732)    1.884  (   small objs)  txdcedb82227654b338daa85751f6d1232

Thanks,
Tim

Timothy Snider
Strategic Planning & Architecture - Advanced Development
NetApp
316-636-8736 Direct Phone
316-213-0223 Mobile Phone
tim.snider at netapp.com
netapp.com<http://www.netapp.com/?ref_source=eSig>
 [Description: http://media.netapp.com/images/netapp-logo-sig-5.gif]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130709/f42907b2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 2568 bytes
Desc: image001.gif
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130709/f42907b2/attachment.gif>