Open Stack

Probably the "devices" option in the object server is misconfigured?

On my lab and production servers I configure the object-server.conf with

[DEFAULT]
devices = /srv/node

And then I make sure my mounted devices appear at:

/srv/node/d1
/srv/node/d2
/srv/node/d3

etc

The path in the error message:

/srv/xvdb1/node/xvdb1/

Looks like the object-server.conf is configured with:

devices = /srv/xvdb1/node

And the ring has devices like "xvdb1"

But as the error states: "No such file or directory at"

devices + device => /srv/xvdb1/node/xvdb1/...

And I trust the error that the path doesn't exist (or if it does maybe the
swift processes don't have permissions?)

Hope you can get it squared.  You might jump in IRC and join
#openstack-swift on Freenode for some more iterative feedback (I'd
recommend irccloud.com if you're new to IRC).

GL,

-Clay


On Tue, Aug 8, 2017 at 2:54 AM, Shyam Prasad N <nspmangalore at gmail.com>
wrote:
>
> Hi,
>
> In my openstack swift cluster, I'm seeing a lot of 503 errors as a
> result of tracebacks in swift logs with "No such file or directory"
> exceptions...
> # grep -Rnw txdaba05e70c6b4dfaa5884-0059895aca /var/log/swift/*
> /var/log/swift/proxy.error:31030:Aug  7 23:31:39
> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: ERROR 500
> Traceback (most recent call last):#012  File
> "/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 1032, in
> __call__#012    res = method(req)#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1412,
> in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File
> "/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 751, in
> PUT#012    writer.put(metadata)#012  File
> "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2451,
> in put#012    super(DiskFileWriter, self)._put(metadata, True)#012
> File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line
> 1476, in _put#012    self._finalize_put, metadata, target_path,
> cleanup)#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3342,
> in force_run_in_thread#012    return self._run_in_eventlet_tpool(func,
> *args, **kwargs)#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3322,
> in _run_in_eventlet_tpool#012    raise result#012OSError: [Errno 2] No
> such file or directory#012 From Object Server re:
>
/v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
> 10.3.60.8:6010/xvdb1 (txn: txdaba05e70c6b4dfaa5884-0059895aca)
> (client_ip: 10.3.60.11)
> /var/log/swift/proxy.error:31031:Aug  7 23:31:39
> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: Object
> PUT returning 503 for [500, 201] (txn:
> txdaba05e70c6b4dfaa5884-0059895aca) (client_ip: 10.3.60.11)
> /var/log/swift/proxy.error:31032:Aug  7 23:31:39
> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: STDERR:
> 10.3.60.11 - - [08/Aug/2017 06:31:39] "PUT
>
/v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
> HTTP/1.1" 503 346 1.553481 (txn: txdaba05e70c6b4dfaa5884-0059895aca)
> /var/log/swift/proxy.log:27701:Aug  7 23:31:39
> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server:
> 10.3.60.11 10.3.60.11 08/Aug/2017/06/31/39 PUT
>
/v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
> HTTP/1.0 503 - - AUTH_tke6014ecd5... 16777216 118 -
> txdaba05e70c6b4dfaa5884-0059895aca - 1.5526 - - 1502173898.383203983
> 1502173899.935844898 0
> /var/log/swift/storage1.log:41634:Aug  7 23:31:39
> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c object-server:
> 10.3.60.8 - - [08/Aug/2017:06:31:39 +0000] "PUT
>
/xvdb1/118/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1"
> 500 981 "PUT
http://10.3.60.8:8080/v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
"
> "txdaba05e70c6b4dfaa5884-0059895aca" "proxy-server 2117" 1.0534 "-"
> 2127 0
> /var/log/swift/storage2.log:128852:Aug  7 23:31:39
> BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c container-server:
> 10.3.60.9 - - [08/Aug/2017:06:31:39 +0000] "PUT
>
/xvdb2/972/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1"
> 201 - "PUT
http://10.3.60.8:8080/xvdb2/118/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
"
> "txdaba05e70c6b4dfaa5884-0059895aca" "object-server 1728" 0.0006 "-"
> 2099 0
>
> I'm also seeing some errors removing tempfile errors in storage logs
also...
> Aug  8 02:28:15 BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c
> object-server: Error removing tempfile:
> /srv/xvdb1/node/xvdb1/tmp/tmpFouKzU: #012Traceback (most recent call
> last):#012  File
> "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2396,
> in create#012    os.unlink(tmppath)#012OSError: [Errno 2] No such file
> or directory: '/srv/xvdb1/node/xvdb1/tmp/tmpFouKzU' (txn:
> tx860a415e4c454baeab4fc-005989842e)
>
> Can someone tell me what's going on?
> Thanks in advance for any help you can give me here.
>
> Regards,
> Shyam
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170808/08298ae4/attachment.html>