Open Stack

Hi Clay, Hi All,

Configured my cluster from ground up with one proxy node and one storage
node.


​
Now I am getting two types of errors.

Large files:

Feb  3 18:12:33 centos7-swift-proxy1 swift-proxy-server: ERROR with Object
server 192.168.47.128:6201/sda re: Trying to write to /AUTH_admin/ara1/abc:
#012Traceback (most recent call last):#012  File
"/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
1617, in _send_file#012    self.conn.send(to_send)#012  File
"/usr/lib64/python2.7/httplib.py", line 840, in send#012
self.sock.sendall(data)#012  File
"/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 393, in
sendall#012    tail += self.send(data[tail:], flags)#012  File
"/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in
send#012    return self._send_loop(self.fd.send, data, flags)#012  File
"/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in
_send_loop#012    return send_method(data, *args)#012error: [Errno 32]
Broken pipe
Feb  3 18:12:33 centos7-swift-proxy1 swift-proxy-server: Object 1 PUT
exceptions during send, 0/1 required connections (txn:
tx0b01a226078f47b4b3593-005a7641e1) (client_ip: 192.168.47.132)

Large file error on storage node:

Feb  4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35624)
Feb  4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:51 +0000] "PUT /sda/89/AUTH_admin/ara1" 202 - "PUT
http://192.168.47.132:8080/v1/AUTH_admin/ara1"
"tx158875c238214209bdfc3-005a7646ab" "proxy-server 44255" 0.0063 "-" 112666
0
Feb  4 19:15:51 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:51] "PUT /sda/89/AUTH_admin/ara1
HTTP/1.1" 202 252 0.006706 (txn: tx158875c238214209bdfc3-005a7646ab)
Feb  4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35630)
Feb  4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:51 +0000] "HEAD /sda/89/AUTH_admin/ara1" 204 - "HEAD
http://192.168.47.132:8080/v1/AUTH_admin/ara1"
"txc23a59c70e8940bf85101-005a7646ab" "proxy-server 44253" 0.0011 "-" 112666
0
Feb  4 19:15:51 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:51] "HEAD /sda/89/AUTH_admin/ara1
HTTP/1.1" 204 521 0.001422 (txn: txc23a59c70e8940bf85101-005a7646ab)
Feb  4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35632)
Feb  4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:51 +0000] "HEAD /sda/75/AUTH_admin/ara1/abc" 404 - "HEAD
http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc"
"txc23a59c70e8940bf85101-005a7646ab" "proxy-server 44253" 0.0003 "-" 112666
0
Feb  4 19:15:51 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:51] "HEAD /sda/75/AUTH_admin/ara1/abc
HTTP/1.1" 404 351 0.000549 (txn: txc23a59c70e8940bf85101-005a7646ab)
Feb  4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35634)
Feb  4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:51 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc"
"tx5138df5cf1f840808d2b2-005a7646ab" "proxy-server 44253" 0.0002 "-" 112666
0
Feb  4 19:15:51 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:51] "PUT /sda/75/AUTH_admin/ara1/abc
HTTP/1.1" 404 212 0.000506 (txn: tx5138df5cf1f840808d2b2-005a7646ab)
Feb  4 19:15:52 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35636)
Feb  4 19:15:52 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:52 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc"
"txeb1c76b896854e4885009-005a7646ac" "proxy-server 44253" 0.0002 "-" 112666
0
Feb  4 19:15:52 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:52] "PUT /sda/75/AUTH_admin/ara1/abc
HTTP/1.1" 404 212 0.000589 (txn: txeb1c76b896854e4885009-005a7646ac)
Feb  4 19:15:54 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35638)
Feb  4 19:15:54 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:15:54 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc"
"tx1a011b125608447ca36a8-005a7646ae" "proxy-server 44253" 0.0002 "-" 112666
0
Feb  4 19:15:54 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:15:54] "PUT /sda/75/AUTH_admin/ara1/abc
HTTP/1.1" 404 212 0.000569 (txn: tx1a011b125608447ca36a8-005a7646ae)

or

Smaller files:

Feb  3 18:30:40 centos7-swift-proxy1 swift-proxy-server: ERROR Unhandled
exception in request: #012Traceback (most recent call last):#012  File
"/usr/lib/python2.7/site-packages/swift/proxy/server.py", line 521, in
handle_request#012    return handler(req)#012  File
"/usr/lib/python2.7/site-packages/swift/proxy/controllers/base.py", line
283, in wrapped#012    return func(*a, **kw)#012  File
"/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
745, in PUT#012    req, data_source, nodes, partition,
outgoing_headers)#012  File
"/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
949, in _store_object#012    self._get_put_responses(req, putters,
len(nodes))#012  File
"/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
406, in _get_put_responses#012    _handle_response(putter, response)#012
File "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py",
line 402, in _handle_response#012
etags.add(response.getheader('etag').strip('"'))#012AttributeError:
'NoneType' object has no attribute 'strip' (txn:
tx029f271d16a84de09d4fb-005a764620) (client_ip: 192.168.47.132)

Smaller file error on storage node:

Feb  4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35696)
Feb  4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:34 +0000] "PUT /sda/89/AUTH_admin/ara1" 202 - "PUT
http://192.168.47.132:8080/v1/AUTH_admin/ara1"
"txb8d792ec40844162978cc-005a764712" "proxy-server 44253" 0.0077 "-" 112666
0
Feb  4 19:17:34 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:34] "PUT /sda/89/AUTH_admin/ara1
HTTP/1.1" 202 252 0.008124 (txn: txb8d792ec40844162978cc-005a764712)
Feb  4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35702)
Feb  4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:34 +0000] "HEAD /sda/89/AUTH_admin/ara1" 204 - "HEAD
http://192.168.47.132:8080/v1/AUTH_admin/ara1"
"tx672029e079574e78b1c1b-005a764712" "proxy-server 44254" 0.0011 "-" 112666
0
Feb  4 19:17:34 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:34] "HEAD /sda/89/AUTH_admin/ara1
HTTP/1.1" 204 521 0.001465 (txn: tx672029e079574e78b1c1b-005a764712)
Feb  4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35704)
Feb  4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:34 +0000] "HEAD /sda/26/AUTH_admin/ara1/swift.conf" 404
- "HEAD http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf"
"tx672029e079574e78b1c1b-005a764712" "proxy-server 44254" 0.0003 "-" 112666
0
Feb  4 19:17:34 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:34] "HEAD
/sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 351 0.000547 (txn:
tx672029e079574e78b1c1b-005a764712)
Feb  4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35706)
Feb  4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:34 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404 -
"PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf"
"tx0ec94217b5584b7ea4e21-005a764712" "proxy-server 44254" 0.0002 "-" 112666
0
Feb  4 19:17:34 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:34] "PUT
/sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000513 (txn:
tx0ec94217b5584b7ea4e21-005a764712)
Feb  4 19:17:35 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35708)
Feb  4 19:17:35 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:35 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404 -
"PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf"
"tx8e93f98453804972b39d8-005a764713" "proxy-server 44254" 0.0004 "-" 112666
0
Feb  4 19:17:35 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:35] "PUT
/sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000791 (txn:
tx8e93f98453804972b39d8-005a764713)
Feb  4 19:17:37 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35710)
Feb  4 19:17:37 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:37 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404 -
"PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf"
"txd44faaca5b5b408da4fea-005a764715" "proxy-server 44254" 0.0004 "-" 112666
0
Feb  4 19:17:37 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:37] "PUT
/sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000716 (txn:
txd44faaca5b5b408da4fea-005a764715)
Feb  4 19:17:41 centos7-swift-node1 container-server: STDERR: (112666)
accepted ('192.168.47.132', 35712)
Feb  4 19:17:41 centos7-swift-node1 container-server: 192.168.47.132 - -
[05/Feb/2018:00:17:41 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404 -
"PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf"
"tx6e5c49ba6d54489aa1de5-005a764719" "proxy-server 44254" 0.0002 "-" 112666
0
Feb  4 19:17:41 centos7-swift-node1 container-server: STDERR:
192.168.47.132 - - [05/Feb/2018 00:17:41] "PUT
/sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000590 (txn:
tx6e5c49ba6d54489aa1de5-005a764719)

Directories created on storage node are



We are really stuck on the issue and not able to use swift in our
environment. Any help will be really appreciated.​

Aravind M D

On Thu, Feb 8, 2018 at 2:43 AM, Clay Gerrard <clay.gerrard at gmail.com> wrote:

> One replica is a little strange.  Do the uploads *always* fail - in the
> same way?  Or is this just one example of a PUT that returned 503?  Are you
> doing a lot of concurrent PUTs to the same object/name/disk?
>
> The error from the log (EPIPE) means the object-server closed the
> connection as the proxy was writing to it... which is a little strange.
> There should be a corresponding exception/error from the object-server
> service - you should make sure the object-servers are running and find
> where they are logging - then grep all the logs for the transaction-id to
> get a better picture of the whole distributed transaction.
>
> If you keep digging I know you can find the problem.  Let us know what you
> find.
>
> Good luck,
>
> -Clay
>
>
> On Wed, Feb 7, 2018 at 1:58 AM, aRaviNd <ambadiaravind at gmail.com> wrote:
>
>> Hi All,
>>
>> We have created an openstack cluster with one proxy server and three
>> storage nodes. Configuration consist of two regions and three zones.
>>
>> [image: enter image description here]
>> <https://i.stack.imgur.com/w9EWG.png>
>>
>> We are able to create containers
>>
>> [image: enter image description here]
>> <https://i.stack.imgur.com/QgQQM.png>
>>
>> But while trying to upload files we are getting 503 service unavailable
>> and seeing below logs in swift.log
>>
>> [image: enter image description here]
>> <https://i.stack.imgur.com/7ROMm.png>
>> - Aravind
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20180212/82f61044/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2018-02-12 at 4.41.21 PM.png
Type: image/png
Size: 13242 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20180212/82f61044/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2018-02-12 at 4.34.07 PM.png
Type: image/png
Size: 32289 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20180212/82f61044/attachment-0001.png>