<div dir="ltr">Hi , <div><br></div><div>1. Correct ! Once you adding new devices and rebalance rings, portion of partitions will be reassigned to new devices. If those partitions were used by some objects, object-replicator is going to move data to new devices. You should see logs of object-replicator to transfer objects from one device to another by invoking rsync. </div>
<div><br></div><div>2. Regarding to busy swift-account-server, that's pretty abnormal tho. Is there any log indicating account-server doing any jobs? A possibility is the ring which includes wrong port number of other workers to account-server. Perhaps you can paste all your rings layout to <a href="http://paste.openstack.org/">http://paste.openstack.org/</a> . To use strace on account-server process may help to track the exercise. </div>
<div><br></div><div>3. In kind of deployment that outward-facing interface shares same network resource with cluster-facing interface, it definitely causes some race on network utilization. Hence the frontend traffic is under impact by replication traffic now. </div>
<div><br></div><div>4. To have a detail network topology diagram will help. </div><div><br></div><div>Hugo Kuo </div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-05-29 1:06 GMT+08:00 Shyam Prasad N <span dir="ltr"><<a href="mailto:nspmangalore@gmail.com" target="_blank">nspmangalore@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi,<br><br></div><div>Confused about the right mailing list to ask this question. So including both openstack and openstack-dev in the CC list.<br>
<br></div>I'm running a swift cluster with 4 nodes. <br>
All 4 nodes are symmetrical. i.e. proxy, object, container, and account servers running on each with similar storage configuration and conf files.<br>The I/O traffic to this cluster is mainly to upload dynamic large objects (typically 1GB chunks (sub-objects) and around 5-6 chunks under each large object). <br>
<br>The setup is running and serving data; but I've begun to see a few perf issues, as the traffic increases. I want to understand the reason behind some of these issues, and make sure that there is nothing wrong with the setup configuration.<br clear="all">
<div><div><br></div><div>1. High CPU utilization from rsync. I have set replica count in each of account, container, and object rings to 2. From what I've read, this assigns 2 devices for each partition in the storage cluster. And for each PUT, the 2 replicas should be written synchronously. And for GET, the I/O is through one of the object servers. So nothing here should be asynchronous in nature. Then what is causing the rsync traffic here? <br>
</div><div><br></div><div>I recently ran a ring rebalance command after adding a node recently. Could this be causing the issue?<br><br></div><div>2. High CPU utilization from swift-account-server threads. All my frontend traffic use 1 account and 1 container on the servers. There are hundreds of such objects in the same container. I don't understand what's keeping the account servers busy.<br>
<br></div><div>3. I've started noticing that the 1GB object transfers of the frontend traffic are taking significantly more time than they used to (more than double the time). Could this be because i'm using the same subnet for both the internal and the frontend traffic.<br>
<br></div><div>4. Can someone provide me some pointers/tips to improving perf for my cluster configuration? (I guess I've given out most details above. Feel free to ask if you need more details)<br><br></div><div>As always, thanks in advance for your replies. Appreciate the support. :)<span class="HOEnZb"><font color="#888888"><br>
</font></span></div><span class="HOEnZb"><font color="#888888"><div>-- <br>-Shyam
</div></font></span></div></div>
<br>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br></div>