Open Stack

Hi,

Confused about the right mailing list to ask this question. So including
both openstack and openstack-dev in the CC list.

I'm running a swift cluster with 4 nodes.
All 4 nodes are symmetrical. i.e. proxy, object, container, and account
servers running on each with similar storage configuration and conf files.
The I/O traffic to this cluster is mainly to upload dynamic large objects
(typically 1GB chunks (sub-objects) and around 5-6 chunks under each large
object).

The setup is running and serving data; but I've begun to see a few perf
issues, as the traffic increases. I want to understand the reason behind
some of these issues, and make sure that there is nothing wrong with the
setup configuration.

1. High CPU utilization from rsync. I have set replica count in each of
account, container, and object rings to 2. From what I've read, this
assigns 2 devices for each partition in the storage cluster. And for each
PUT, the 2 replicas should be written synchronously. And for GET, the I/O
is through one of the object servers. So nothing here should be
asynchronous in nature. Then what is causing the rsync traffic here?

I recently ran a ring rebalance command after adding a node recently. Could
this be causing the issue?

2. High CPU utilization from swift-account-server threads. All my frontend
traffic use 1 account and 1 container on the servers. There are hundreds of
such objects in the same container. I don't understand what's keeping the
account servers busy.

3. I've started noticing that the 1GB object transfers of the frontend
traffic are taking significantly more time than they used to (more than
double the time). Could this be because i'm using the same subnet for both
the internal and the frontend traffic.

4. Can someone provide me some pointers/tips to improving perf for my
cluster configuration? (I guess I've given out most details above. Feel
free to ask if you need more details)

As always, thanks in advance for your replies. Appreciate the support. :)
-- 
-Shyam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140528/e9bbd0b7/attachment.html>