Open Stack

Hi Clay,

Thanks for the reply. And sorry for the late reply.
I should have added a few more details in the email.

The devices are mounted correctly and object-server is configured as
you advised. I'm not seeing these errors all the time. I was seeing it
on one of every ten to fifteen writes though. At the moment, I don't
see the errors though.
The permissions look fine to me. The swift processes are all running
with UID swiftuser, and swiftuser is the owner of the mount point and
has necessary permissions.

I'll jump into the IRC channel if I start seeing the errors again.



On Tue, Aug 8, 2017 at 9:54 PM, Clay Gerrard <clay.gerrard at gmail.com> wrote:
> Probably the "devices" option in the object server is misconfigured?
>
> On my lab and production servers I configure the object-server.conf with
>
> [DEFAULT]
> devices = /srv/node
>
> And then I make sure my mounted devices appear at:
>
> /srv/node/d1
> /srv/node/d2
> /srv/node/d3
>
> etc
>
> The path in the error message:
>
> /srv/xvdb1/node/xvdb1/
>
> Looks like the object-server.conf is configured with:
>
> devices = /srv/xvdb1/node
>
> And the ring has devices like "xvdb1"
>
> But as the error states: "No such file or directory at"
>
> devices + device => /srv/xvdb1/node/xvdb1/...
>
> And I trust the error that the path doesn't exist (or if it does maybe the
> swift processes don't have permissions?)
>
> Hope you can get it squared.  You might jump in IRC and join
> #openstack-swift on Freenode for some more iterative feedback (I'd recommend
> irccloud.com if you're new to IRC).
>
> GL,
>
> -Clay
>
>
> On Tue, Aug 8, 2017 at 2:54 AM, Shyam Prasad N <nspmangalore at gmail.com>
> wrote:
>>
>> Hi,
>>
>> In my openstack swift cluster, I'm seeing a lot of 503 errors as a
>> result of tracebacks in swift logs with "No such file or directory"
>> exceptions...
>> # grep -Rnw txdaba05e70c6b4dfaa5884-0059895aca /var/log/swift/*
>> /var/log/swift/proxy.error:31030:Aug  7 23:31:39
>> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: ERROR 500
>> Traceback (most recent call last):#012  File
>> "/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 1032, in
>> __call__#012    res = method(req)#012  File
>> "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1412,
>> in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File
>> "/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 751, in
>> PUT#012    writer.put(metadata)#012  File
>> "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2451,
>> in put#012    super(DiskFileWriter, self)._put(metadata, True)#012
>> File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line
>> 1476, in _put#012    self._finalize_put, metadata, target_path,
>> cleanup)#012  File
>> "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3342,
>> in force_run_in_thread#012    return self._run_in_eventlet_tpool(func,
>> *args, **kwargs)#012  File
>> "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3322,
>> in _run_in_eventlet_tpool#012    raise result#012OSError: [Errno 2] No
>> such file or directory#012 From Object Server re:
>>
>> /v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
>> 10.3.60.8:6010/xvdb1 (txn: txdaba05e70c6b4dfaa5884-0059895aca)
>> (client_ip: 10.3.60.11)
>> /var/log/swift/proxy.error:31031:Aug  7 23:31:39
>> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: Object
>> PUT returning 503 for [500, 201] (txn:
>> txdaba05e70c6b4dfaa5884-0059895aca) (client_ip: 10.3.60.11)
>> /var/log/swift/proxy.error:31032:Aug  7 23:31:39
>> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: STDERR:
>> 10.3.60.11 - - [08/Aug/2017 06:31:39] "PUT
>>
>> /v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
>> HTTP/1.1" 503 346 1.553481 (txn: txdaba05e70c6b4dfaa5884-0059895aca)
>> /var/log/swift/proxy.log:27701:Aug  7 23:31:39
>> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server:
>> 10.3.60.11 10.3.60.11 08/Aug/2017/06/31/39 PUT
>>
>> /v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
>> HTTP/1.0 503 - - AUTH_tke6014ecd5... 16777216 118 -
>> txdaba05e70c6b4dfaa5884-0059895aca - 1.5526 - - 1502173898.383203983
>> 1502173899.935844898 0
>> /var/log/swift/storage1.log:41634:Aug  7 23:31:39
>> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c object-server:
>> 10.3.60.8 - - [08/Aug/2017:06:31:39 +0000] "PUT
>>
>> /xvdb1/118/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1"
>> 500 981 "PUT
>> http://10.3.60.8:8080/v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1"
>> "txdaba05e70c6b4dfaa5884-0059895aca" "proxy-server 2117" 1.0534 "-"
>> 2127 0
>> /var/log/swift/storage2.log:128852:Aug  7 23:31:39
>> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c container-server:
>> 10.3.60.9 - - [08/Aug/2017:06:31:39 +0000] "PUT
>>
>> /xvdb2/972/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1"
>> 201 - "PUT
>> http://10.3.60.8:8080/xvdb2/118/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1"
>> "txdaba05e70c6b4dfaa5884-0059895aca" "object-server 1728" 0.0006 "-"
>> 2099 0
>>
>> I'm also seeing some errors removing tempfile errors in storage logs
>> also...
>> Aug  8 02:28:15 BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c
>> object-server: Error removing tempfile:
>> /srv/xvdb1/node/xvdb1/tmp/tmpFouKzU: #012Traceback (most recent call
>> last):#012  File
>> "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2396,
>> in create#012    os.unlink(tmppath)#012OSError: [Errno 2] No such file
>> or directory: '/srv/xvdb1/node/xvdb1/tmp/tmpFouKzU' (txn:
>> tx860a415e4c454baeab4fc-005989842e)
>>
>> Can someone tell me what's going on?
>> Thanks in advance for any help you can give me here.
>>
>> Regards,
>> Shyam
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>



-- 
-Shyam