[Openstack] [Swift] PUT Object performance problem

Clay Gerrard clay.gerrard at gmail.com
Mon Oct 29 16:42:17 UTC 2018


Obviously a re-balance will cost some IO, but it's normally perceptible to
the client unless you were already on a razor thin line.

Two config options seem obvious to think about experimenting with:

You could decrease the node_timeout and let the proxy try to write more to
handoffs
You could try to use some of the ionice options (or other tuning options)
to make the replicators hammer the disks a little less

It might having something to do with how you're organizing your ring weight
changes - could you describe how you're managing rings?
Could also be a io scheduling issue - are you using noop/deadline or cfq?
What kernel version?

-Clay


On Mon, Oct 29, 2018 at 9:55 AM SeongSoo Cho <ppiyakk2 at printf.kr> wrote:

> For more information, I use ocata version.
>
> 2018년 10월 29일 (월) 오후 10:07, SeongSoo Cho <ppiyakk2 at printf.kr>님이 작성:
>
>> Hello, All
>>
>> I have a terrible problem with object server.
>> Here is the case.
>> 1. User upload an object to proxy-server
>> 2. Proxy server try to connect with object-server
>> 3. If one of object-server is slow to respond, proxy-server is waiting
>> for response.
>>   3.1 While waiting for response, proxy-server can't do anything
>> 4. So, The response of client request will be delayed.
>>
>> In my opinion, this code seems to be a problem
>> (
>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734
>> )
>>
>> ```
>> with ResponseTimeout(node_timeout):
>>             resp = conn.getexpect()
>> ```
>>
>> If node_timeout's value is 3 and object-server respond after 2 seconds,
>> proxy-server wait 2 seconds.
>>
>> Because proxy-server wait for the above response, the execution of the
>> following code is delayed.
>> (
>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627
>> )
>>
>> ```
>> for node in nodes:
>>             try:
>>                 putter = self._make_putter(node, part, req, headers)
>>                 self.app.set_node_timing(node, putter.connect_duration)
>>                 return putter
>> ```
>>
>> This problem occurs when i do a ring rebalance.
>> When object-replicator delete a partition directory that are no longer
>> mine, the disk becomes very busy (Because of xfsaild daemon)
>> Because the disk are busy, object-server can't create diskfile during PUT
>> operation.
>>
>> Is there anyone who is having problems like me?
>> How can I solve this problem?
>>
>> I need everyone's help.
>> Thanks.
>>
>> Best Regards
>> SeongSoo Cho
>>
>> ------
>> SeongSoo Cho (South Korea)
>>
>> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20181029/8f4a7884/attachment.html>


More information about the Openstack mailing list