[Openstack] Not able to upload files to openstack swift
Matthew Oliver
matt at oliver.net.au
Mon Feb 12 22:29:22 UTC 2018
Hi Aravind,
There only seems to be container-server logs in your reply. So you have any
from the object server?
Also what ports are your object-server and container-servers listening on?
Your ring is saying the object-servers are listening on 6201. Just making
sure the port numbers aren't confused and the proxy isn't tryng to send the
object PUTs to the container servers. Because it's interesting that there
is no objects subfolder.
Matt
On Mon, Feb 12, 2018 at 10:12 PM, aRaviNd <ambadiaravind at gmail.com> wrote:
> Hi Clay, Hi All,
>
> Configured my cluster from ground up with one proxy node and one storage
> node.
>
>
>
> Now I am getting two types of errors.
>
> Large files:
>
> Feb 3 18:12:33 centos7-swift-proxy1 swift-proxy-server: ERROR with Object
> server 192.168.47.128:6201/sda re: Trying to write to
> /AUTH_admin/ara1/abc: #012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
> 1617, in _send_file#012 self.conn.send(to_send)#012 File
> "/usr/lib64/python2.7/httplib.py", line 840, in send#012
> self.sock.sendall(data)#012 File "/usr/lib/python2.7/site-
> packages/eventlet/greenio/base.py", line 393, in sendall#012 tail +=
> self.send(data[tail:], flags)#012 File "/usr/lib/python2.7/site-
> packages/eventlet/greenio/base.py", line 384, in send#012 return
> self._send_loop(self.fd.send, data, flags)#012 File
> "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in
> _send_loop#012 return send_method(data, *args)#012error: [Errno 32]
> Broken pipe
> Feb 3 18:12:33 centos7-swift-proxy1 swift-proxy-server: Object 1 PUT
> exceptions during send, 0/1 required connections (txn:
> tx0b01a226078f47b4b3593-005a7641e1) (client_ip: 192.168.47.132)
>
> Large file error on storage node:
>
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35624)
> Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:51 +0000] "PUT /sda/89/AUTH_admin/ara1" 202 - "PUT
> http://192.168.47.132:8080/v1/AUTH_admin/ara1" "tx158875c238214209bdfc3-005a7646ab"
> "proxy-server 44255" 0.0063 "-" 112666 0
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:51] "PUT /sda/89/AUTH_admin/ara1
> HTTP/1.1" 202 252 0.006706 (txn: tx158875c238214209bdfc3-005a7646ab)
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35630)
> Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:51 +0000] "HEAD /sda/89/AUTH_admin/ara1" 204 - "HEAD
> http://192.168.47.132:8080/v1/AUTH_admin/ara1" "txc23a59c70e8940bf85101-005a7646ab"
> "proxy-server 44253" 0.0011 "-" 112666 0
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:51] "HEAD /sda/89/AUTH_admin/ara1
> HTTP/1.1" 204 521 0.001422 (txn: txc23a59c70e8940bf85101-005a7646ab)
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35632)
> Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:51 +0000] "HEAD /sda/75/AUTH_admin/ara1/abc" 404 - "HEAD
> http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc"
> "txc23a59c70e8940bf85101-005a7646ab" "proxy-server 44253" 0.0003 "-"
> 112666 0
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:51] "HEAD /sda/75/AUTH_admin/ara1/abc
> HTTP/1.1" 404 351 0.000549 (txn: txc23a59c70e8940bf85101-005a7646ab)
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35634)
> Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:51 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
> http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc"
> "tx5138df5cf1f840808d2b2-005a7646ab" "proxy-server 44253" 0.0002 "-"
> 112666 0
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:51] "PUT /sda/75/AUTH_admin/ara1/abc
> HTTP/1.1" 404 212 0.000506 (txn: tx5138df5cf1f840808d2b2-005a7646ab)
> Feb 4 19:15:52 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35636)
> Feb 4 19:15:52 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:52 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
> http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc"
> "txeb1c76b896854e4885009-005a7646ac" "proxy-server 44253" 0.0002 "-"
> 112666 0
> Feb 4 19:15:52 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:52] "PUT /sda/75/AUTH_admin/ara1/abc
> HTTP/1.1" 404 212 0.000589 (txn: txeb1c76b896854e4885009-005a7646ac)
> Feb 4 19:15:54 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35638)
> Feb 4 19:15:54 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:54 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
> http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc"
> "tx1a011b125608447ca36a8-005a7646ae" "proxy-server 44253" 0.0002 "-"
> 112666 0
> Feb 4 19:15:54 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:54] "PUT /sda/75/AUTH_admin/ara1/abc
> HTTP/1.1" 404 212 0.000569 (txn: tx1a011b125608447ca36a8-005a7646ae)
>
> or
>
> Smaller files:
>
> Feb 3 18:30:40 centos7-swift-proxy1 swift-proxy-server: ERROR Unhandled
> exception in request: #012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/server.py", line 521, in
> handle_request#012 return handler(req)#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/controllers/base.py", line
> 283, in wrapped#012 return func(*a, **kw)#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
> 745, in PUT#012 req, data_source, nodes, partition,
> outgoing_headers)#012 File "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py",
> line 949, in _store_object#012 self._get_put_responses(req, putters,
> len(nodes))#012 File "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py",
> line 406, in _get_put_responses#012 _handle_response(putter,
> response)#012 File "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py",
> line 402, in _handle_response#012 etags.add(response.getheader('
> etag').strip('"'))#012AttributeError: 'NoneType' object has no attribute
> 'strip' (txn: tx029f271d16a84de09d4fb-005a764620) (client_ip:
> 192.168.47.132)
>
> Smaller file error on storage node:
>
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35696)
> Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:34 +0000] "PUT /sda/89/AUTH_admin/ara1" 202 - "PUT
> http://192.168.47.132:8080/v1/AUTH_admin/ara1" "txb8d792ec40844162978cc-005a764712"
> "proxy-server 44253" 0.0077 "-" 112666 0
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:34] "PUT /sda/89/AUTH_admin/ara1
> HTTP/1.1" 202 252 0.008124 (txn: txb8d792ec40844162978cc-005a764712)
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35702)
> Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:34 +0000] "HEAD /sda/89/AUTH_admin/ara1" 204 - "HEAD
> http://192.168.47.132:8080/v1/AUTH_admin/ara1" "tx672029e079574e78b1c1b-005a764712"
> "proxy-server 44254" 0.0011 "-" 112666 0
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:34] "HEAD /sda/89/AUTH_admin/ara1
> HTTP/1.1" 204 521 0.001465 (txn: tx672029e079574e78b1c1b-005a764712)
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35704)
> Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:34 +0000] "HEAD /sda/26/AUTH_admin/ara1/swift.conf"
> 404 - "HEAD http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf"
> "tx672029e079574e78b1c1b-005a764712" "proxy-server 44254" 0.0003 "-"
> 112666 0
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:34] "HEAD
> /sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 351 0.000547 (txn:
> tx672029e079574e78b1c1b-005a764712)
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35706)
> Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:34 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404
> - "PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf"
> "tx0ec94217b5584b7ea4e21-005a764712" "proxy-server 44254" 0.0002 "-"
> 112666 0
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:34] "PUT
> /sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000513 (txn:
> tx0ec94217b5584b7ea4e21-005a764712)
> Feb 4 19:17:35 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35708)
> Feb 4 19:17:35 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:35 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404
> - "PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf"
> "tx8e93f98453804972b39d8-005a764713" "proxy-server 44254" 0.0004 "-"
> 112666 0
> Feb 4 19:17:35 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:35] "PUT
> /sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000791 (txn:
> tx8e93f98453804972b39d8-005a764713)
> Feb 4 19:17:37 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35710)
> Feb 4 19:17:37 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:37 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404
> - "PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf"
> "txd44faaca5b5b408da4fea-005a764715" "proxy-server 44254" 0.0004 "-"
> 112666 0
> Feb 4 19:17:37 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:37] "PUT
> /sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000716 (txn:
> txd44faaca5b5b408da4fea-005a764715)
> Feb 4 19:17:41 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35712)
> Feb 4 19:17:41 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:41 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404
> - "PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf"
> "tx6e5c49ba6d54489aa1de5-005a764719" "proxy-server 44254" 0.0002 "-"
> 112666 0
> Feb 4 19:17:41 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:41] "PUT
> /sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000590 (txn:
> tx6e5c49ba6d54489aa1de5-005a764719)
>
> Directories created on storage node are
>
>
>
> We are really stuck on the issue and not able to use swift in our
> environment. Any help will be really appreciated.
>
> Aravind M D
>
> On Thu, Feb 8, 2018 at 2:43 AM, Clay Gerrard <clay.gerrard at gmail.com>
> wrote:
>
>> One replica is a little strange. Do the uploads *always* fail - in the
>> same way? Or is this just one example of a PUT that returned 503? Are you
>> doing a lot of concurrent PUTs to the same object/name/disk?
>>
>> The error from the log (EPIPE) means the object-server closed the
>> connection as the proxy was writing to it... which is a little strange.
>> There should be a corresponding exception/error from the object-server
>> service - you should make sure the object-servers are running and find
>> where they are logging - then grep all the logs for the transaction-id to
>> get a better picture of the whole distributed transaction.
>>
>> If you keep digging I know you can find the problem. Let us know what
>> you find.
>>
>> Good luck,
>>
>> -Clay
>>
>>
>> On Wed, Feb 7, 2018 at 1:58 AM, aRaviNd <ambadiaravind at gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> We have created an openstack cluster with one proxy server and three
>>> storage nodes. Configuration consist of two regions and three zones.
>>>
>>> [image: enter image description here]
>>> <https://i.stack.imgur.com/w9EWG.png>
>>>
>>> We are able to create containers
>>>
>>> [image: enter image description here]
>>> <https://i.stack.imgur.com/QgQQM.png>
>>>
>>> But while trying to upload files we are getting 503 service unavailable
>>> and seeing below logs in swift.log
>>>
>>> [image: enter image description here]
>>> <https://i.stack.imgur.com/7ROMm.png>
>>> - Aravind
>>>
>>> _______________________________________________
>>> Mailing list: http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>> Post to : openstack at lists.openstack.org
>>> Unsubscribe : http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>>
>>>
>>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20180213/160fb83f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2018-02-12 at 4.41.21 PM.png
Type: image/png
Size: 13242 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20180213/160fb83f/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2018-02-12 at 4.34.07 PM.png
Type: image/png
Size: 32289 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20180213/160fb83f/attachment-0001.png>
More information about the Openstack
mailing list
