Open Stack

Hi clay

I just using the command like `swift-ring-builder object.builder
set_weight  d234 2220`.
Then execute `swift-ring-builder object.builder rebalance` and push the
rings all object node.

Here is more detail information.
I/O scheduler : deadline
OS : CentOS 7.4
Kernel version : 3.10.0-693.11.1.el7.x86_64

I will try decrease the node_timeout from 3 to 1.  (Do you have any
recommend value?)

On Tue, Oct 30, 2018 at 1:42 AM Clay Gerrard <clay.gerrard at gmail.com> wrote:

> Obviously a re-balance will cost some IO, but it's normally perceptible to
> the client unless you were already on a razor thin line.
>
> Two config options seem obvious to think about experimenting with:
>
> You could decrease the node_timeout and let the proxy try to write more to
> handoffs
> You could try to use some of the ionice options (or other tuning options)
> to make the replicators hammer the disks a little less
>
> It might having something to do with how you're organizing your ring
> weight changes - could you describe how you're managing rings?
> Could also be a io scheduling issue - are you using noop/deadline or cfq?
> What kernel version?
>
> -Clay
>
>
> On Mon, Oct 29, 2018 at 9:55 AM SeongSoo Cho <ppiyakk2 at printf.kr> wrote:
>
>> For more information, I use ocata version.
>>
>> 2018년 10월 29일 (월) 오후 10:07, SeongSoo Cho <ppiyakk2 at printf.kr>님이 작성:
>>
>>> Hello, All
>>>
>>> I have a terrible problem with object server.
>>> Here is the case.
>>> 1. User upload an object to proxy-server
>>> 2. Proxy server try to connect with object-server
>>> 3. If one of object-server is slow to respond, proxy-server is waiting
>>> for response.
>>>   3.1 While waiting for response, proxy-server can't do anything
>>> 4. So, The response of client request will be delayed.
>>>
>>> In my opinion, this code seems to be a problem
>>> (
>>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734
>>> )
>>>
>>> ```
>>> with ResponseTimeout(node_timeout):
>>>             resp = conn.getexpect()
>>> ```
>>>
>>> If node_timeout's value is 3 and object-server respond after 2 seconds,
>>> proxy-server wait 2 seconds.
>>>
>>> Because proxy-server wait for the above response, the execution of the
>>> following code is delayed.
>>> (
>>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627
>>> )
>>>
>>> ```
>>> for node in nodes:
>>>             try:
>>>                 putter = self._make_putter(node, part, req, headers)
>>>                 self.app.set_node_timing(node, putter.connect_duration)
>>>                 return putter
>>> ```
>>>
>>> This problem occurs when i do a ring rebalance.
>>> When object-replicator delete a partition directory that are no longer
>>> mine, the disk becomes very busy (Because of xfsaild daemon)
>>> Because the disk are busy, object-server can't create diskfile during
>>> PUT operation.
>>>
>>> Is there anyone who is having problems like me?
>>> How can I solve this problem?
>>>
>>> I need everyone's help.
>>> Thanks.
>>>
>>> Best Regards
>>> SeongSoo Cho
>>>
>>> ------
>>> SeongSoo Cho (South Korea)
>>>
>>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20181030/ceb8c089/attachment.html>