Open Stack

I've been doing some detailed analysis of the server/proxy logs to see how unusually long transactions spend their time when writing 1KB objects noting most of the time they take < 0.1 seconds.  When I do see longer latencies, as expected, that time is usually attributed to a single object server.

However, in some cases I'm seeing three servers take the same, extended amount of time.  I've seen this happen with latencies of all different durations even as high as 18 seconds.  I should also point out that the container latencies are constant at about 0.001 seconds.

My question is I thought the 3 object servers are more-of-less autonomous, focused on getting the object onto storage?  The numbers I'm seeing suggest there is there's more than that going on and I'm wondering what it is and more importantly what can be done about it.  Could some of the background processing be blocking and doing it across all 3 servers?  Is this known behavior?

There are other oddities I've seen as well but thought I'd focus on this for now.  Would this also be better discussed on IRC?

-mark

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140302/9c6d1d69/attachment.html>