Open Stack

Hi,

In my openstack swift cluster, I'm seeing a lot of 503 errors as a
result of tracebacks in swift logs with "No such file or directory"
exceptions...
# grep -Rnw txdaba05e70c6b4dfaa5884-0059895aca /var/log/swift/*
/var/log/swift/proxy.error:31030:Aug  7 23:31:39
BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: ERROR 500
Traceback (most recent call last):#012  File
"/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 1032, in
__call__#012    res = method(req)#012  File
"/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1412,
in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File
"/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 751, in
PUT#012    writer.put(metadata)#012  File
"/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2451,
in put#012    super(DiskFileWriter, self)._put(metadata, True)#012
File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line
1476, in _put#012    self._finalize_put, metadata, target_path,
cleanup)#012  File
"/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3342,
in force_run_in_thread#012    return self._run_in_eventlet_tpool(func,
*args, **kwargs)#012  File
"/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3322,
in _run_in_eventlet_tpool#012    raise result#012OSError: [Errno 2] No
such file or directory#012 From Object Server re:
/v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
10.3.60.8:6010/xvdb1 (txn: txdaba05e70c6b4dfaa5884-0059895aca)
(client_ip: 10.3.60.11)
/var/log/swift/proxy.error:31031:Aug  7 23:31:39
BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: Object
PUT returning 503 for [500, 201] (txn:
txdaba05e70c6b4dfaa5884-0059895aca) (client_ip: 10.3.60.11)
/var/log/swift/proxy.error:31032:Aug  7 23:31:39
BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: STDERR:
10.3.60.11 - - [08/Aug/2017 06:31:39] "PUT
/v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
HTTP/1.1" 503 346 1.553481 (txn: txdaba05e70c6b4dfaa5884-0059895aca)
/var/log/swift/proxy.log:27701:Aug  7 23:31:39
BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server:
10.3.60.11 10.3.60.11 08/Aug/2017/06/31/39 PUT
/v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1
HTTP/1.0 503 - - AUTH_tke6014ecd5... 16777216 118 -
txdaba05e70c6b4dfaa5884-0059895aca - 1.5526 - - 1502173898.383203983
1502173899.935844898 0
/var/log/swift/storage1.log:41634:Aug  7 23:31:39
BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c object-server:
10.3.60.8 - - [08/Aug/2017:06:31:39 +0000] "PUT
/xvdb1/118/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1"
500 981 "PUT http://10.3.60.8:8080/v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1"
"txdaba05e70c6b4dfaa5884-0059895aca" "proxy-server 2117" 1.0534 "-"
2127 0
/var/log/swift/storage2.log:128852:Aug  7 23:31:39
BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c container-server:
10.3.60.9 - - [08/Aug/2017:06:31:39 +0000] "PUT
/xvdb2/972/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1"
201 - "PUT http://10.3.60.8:8080/xvdb2/118/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1"
"txdaba05e70c6b4dfaa5884-0059895aca" "object-server 1728" 0.0006 "-"
2099 0

I'm also seeing some errors removing tempfile errors in storage logs also...
Aug  8 02:28:15 BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c
object-server: Error removing tempfile:
/srv/xvdb1/node/xvdb1/tmp/tmpFouKzU: #012Traceback (most recent call
last):#012  File
"/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2396,
in create#012    os.unlink(tmppath)#012OSError: [Errno 2] No such file
or directory: '/srv/xvdb1/node/xvdb1/tmp/tmpFouKzU' (txn:
tx860a415e4c454baeab4fc-005989842e)

Can someone tell me what's going on?
Thanks in advance for any help you can give me here.

Regards,
Shyam