[Openstack] [Swift] 404 re-reading just created container
Mark Kirkwood
mark.kirkwood at catalyst.net.nz
Fri Jun 3 04:09:06 UTC 2016
Hi,
I did previously mention that I would start a new thread if I could
reproduce 'unexplained 404s'. Well, I can - however not for the exact
case I mentioned in the previous thread.
I have some python code using python-swiftclient that does:
for a number of containers:
create the container
create a number of (e.g 1000) objects in it
I'm seeing:
container con3
..object con3/obj0
Traceback (most recent call last):
File "./swiftclient-creatread.py", line 72, in <module>
content_type='text/plain')
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line
1709, in put_object
response_dict=response_dict)
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line
1565, in _retry
service_token=self.service_token, **kwargs)
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line
1281, in put_object
http_response_content=body)
swiftclient.exceptions.ClientException: Object PUT failed:
https://api.ostst.wgtn.cat-it.co.nz:8843/v1/AUTH_5efabf2fdf4a41509c8ac941ae09ee36/con3/obj0
404 Not Found [first 60 chars of response] <html><h1>Not
Found</h1><p>The resource could not be found.<
However the container has in fact been created ok - rerunning the
program works fine (for the container where it failed prev, subsequent
iterations *may* fail adding objects to just created containers).
This is a Swift 2.7.0.1 system with 2 proxies and 3 storage nodes (each
the latter with 6 devices).
The proxies are load balanced behind Haproxy (which I'm guessing is
causing the 404 - see below)
The variables recheck_account_existence, recheck_container_existence,
request_node_count are all default.
Looking at the proxy logs, it seems that the request to create the
container went to proxy number 1, and the request to create the 1st
object in the container went to proxy number 2:
Jun 3 15:06:04 tst-wgtn-opxy1 proxy-server: 10.16.0.41 127.0.0.1
03/Jun/2016/03/06/04 PUT /v1/AUTH_5efabf2fdf4a41509c8ac941ae09ee36/con3
HTTP/1.0 201 - python-swiftclient-3.0.0 af45fc08a94d4097... - - -
tx6aaf7d3208724079b1b92-005750f41c - 0.0444 - - 1464923164.878182888
1464923164.922573090 0
Jun 3 15:06:04 tst-wgtn-opxy2 proxy-server: 10.16.0.41 127.0.0.1
03/Jun/2016/03/06/04 PUT
/v1/AUTH_5efabf2fdf4a41509c8ac941ae09ee36/con3/obj0 HTTP/1.0 404 -
python-swiftclient-3.0.0 af45fc08a94d4097... - 70 -
tx3ed4012d74e94eafa6ca1-005750f41c - 0.0007 - - 1464923164.943866968
1464923164.944570065 0
I'll dig up the Haproxy config and post. However any thoughts in the
meantime?
regards
Mark
More information about the Openstack
mailing list