[Openstack] [Swift] 404 re-reading just created container

Mark Kirkwood mark.kirkwood at catalyst.net.nz
Fri Jun 3 04:09:06 UTC 2016


Hi,

I did previously mention that I would start a new thread if I could 
reproduce 'unexplained 404s'. Well, I can - however not for the exact 
case I mentioned in the previous thread.

I have some python code using python-swiftclient that does:

for a number of containers:
     create the container
     create a number of (e.g 1000) objects in it

I'm seeing:

container con3
..object con3/obj0
Traceback (most recent call last):
   File "./swiftclient-creatread.py", line 72, in <module>
     content_type='text/plain')
   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 
1709, in put_object
     response_dict=response_dict)
   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 
1565, in _retry
     service_token=self.service_token, **kwargs)
   File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 
1281, in put_object
     http_response_content=body)
swiftclient.exceptions.ClientException: Object PUT failed: 
https://api.ostst.wgtn.cat-it.co.nz:8843/v1/AUTH_5efabf2fdf4a41509c8ac941ae09ee36/con3/obj0 
404 Not Found  [first 60 chars of response] <html><h1>Not 
Found</h1><p>The resource could not be found.<


However the container has in fact been created ok - rerunning the 
program works fine (for the container where it failed prev, subsequent 
iterations *may* fail adding objects to just created containers).

This is a Swift 2.7.0.1 system with 2 proxies and 3 storage nodes (each 
the latter with 6 devices).

The proxies are load balanced behind Haproxy (which I'm guessing is 
causing the 404 - see below)

The variables recheck_account_existence, recheck_container_existence, 
request_node_count are all default.

Looking at the proxy logs, it seems that the request to create the 
container went to proxy number 1, and the request to create the 1st 
object in the container went to proxy number 2:

Jun  3 15:06:04 tst-wgtn-opxy1 proxy-server: 10.16.0.41 127.0.0.1 
03/Jun/2016/03/06/04 PUT /v1/AUTH_5efabf2fdf4a41509c8ac941ae09ee36/con3 
HTTP/1.0 201 - python-swiftclient-3.0.0 af45fc08a94d4097... - - - 
tx6aaf7d3208724079b1b92-005750f41c - 0.0444 - - 1464923164.878182888 
1464923164.922573090 0


Jun  3 15:06:04 tst-wgtn-opxy2 proxy-server: 10.16.0.41 127.0.0.1 
03/Jun/2016/03/06/04 PUT 
/v1/AUTH_5efabf2fdf4a41509c8ac941ae09ee36/con3/obj0 HTTP/1.0 404 - 
python-swiftclient-3.0.0 af45fc08a94d4097... - 70 - 
tx3ed4012d74e94eafa6ca1-005750f41c - 0.0007 - - 1464923164.943866968 
1464923164.944570065 0

I'll dig up the Haproxy config and post. However any thoughts in the 
meantime?

regards

Mark




More information about the Openstack mailing list