<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi clay</div><div><br></div><div>I just using the command like `swift-ring-builder object.builder set_weight d234 2220`.</div><div>Then execute `swift-ring-builder object.builder rebalance` and push the rings all object node.</div><div><br></div><div>Here is more detail information.</div><div>I/O scheduler : deadline</div><div>OS : CentOS 7.4</div><div>Kernel version : 3.10.0-693.11.1.el7.x86_64</div><div><br></div><div>I will try decrease the node_timeout from 3 to 1. (Do you have any recommend value?)</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Oct 30, 2018 at 1:42 AM Clay Gerrard <<a href="mailto:clay.gerrard@gmail.com">clay.gerrard@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Obviously a re-balance will cost some IO, but it's normally perceptible to the client unless you were already on a razor thin line. <br></div><div><br></div><div>Two config options seem obvious to think about experimenting with:</div><div><br></div><div>You could decrease the node_timeout and let the proxy try to write more to handoffs</div><div>You could try to use some of the ionice options (or other tuning options) to make the replicators hammer the disks a little less</div><div><br></div><div>It might having something to do with how you're organizing your ring weight changes - could you describe how you're managing rings?</div><div>Could also be a io scheduling issue - are you using noop/deadline or cfq? What kernel version?<br></div><div><br></div><div>-Clay</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Oct 29, 2018 at 9:55 AM SeongSoo Cho <<a href="mailto:ppiyakk2@printf.kr" target="_blank">ppiyakk2@printf.kr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">For more information, I use ocata version.<br><br><div class="gmail_quote"><div dir="ltr">2018년 10월 29일 (월) 오후 10:07, SeongSoo Cho <<a href="mailto:ppiyakk2@printf.kr" target="_blank">ppiyakk2@printf.kr</a>>님이 작성:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hello, All<div><br></div><div>I have a terrible problem with object server.</div><div>Here is the case.</div><div>1. User upload an object to proxy-server</div><div>2. Proxy server try to connect with object-server</div><div>3. If one of object-server is slow to respond, proxy-server is waiting for response.</div><div> 3.1 While waiting for response, proxy-server can't do anything</div><div>4. So, The response of client request will be delayed.</div><div><br></div><div>In my opinion, this code seems to be a problem</div><div>(<a href="https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734" target="_blank">https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734</a>)<br></div><div><br></div><div>```</div><div><div>with ResponseTimeout(node_timeout):</div><div> resp = conn.getexpect()</div></div><div>```</div><div><br></div><div>If node_timeout's value is 3 and object-server respond after 2 seconds, proxy-server wait 2 seconds.</div><div><br></div><div>Because proxy-server wait for the above response, the execution of the following code is delayed.</div><div>(<a href="https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627" target="_blank">https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627</a>)<br></div><div><br></div><div>```</div><div><div>for node in nodes:</div><div> try:</div><div> putter = self._make_putter(node, part, req, headers)</div><div> self.app.set_node_timing(node, putter.connect_duration)</div><div> return putter</div></div><div>```</div><div><br></div><div>This problem occurs when i do a ring rebalance.</div><div>When object-replicator delete a partition directory that are no longer mine, the disk becomes very busy (Because of xfsaild daemon)</div><div>Because the disk are busy, object-server can't create diskfile during PUT operation.</div><div><br></div><div>Is there anyone who is having problems like me?<br></div><div>How can I solve this problem?</div><div><br></div><div>I need everyone's help.</div><div>Thanks.</div><div><br></div><div>Best Regards</div><div>SeongSoo Cho</div><div><br></div><div>------</div><div>SeongSoo Cho (South Korea)</div><div><br></div></div></div></div></div></div></div>
</blockquote></div>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
</blockquote></div>
</blockquote></div></div></div>