[Openstack] [SWIFT] 404 error during uploads

Stephen Wood smwood4 at gmail.com
Mon Dec 23 20:13:13 UTC 2013 

I've run into a very annoying problem with my swift cluster and I'm hoping
somebody can help me out. During any given 100 uploads, I'll usually see
between 1 to 4% of the calls give a 404. The swift client writes the
following:

Object PUT failed: http://swift:8080/v1/AUTH_swift/test1/042 404 Not Found
 [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could
not be found.<

I'm uploading everything to a single bucket and there's only 100 files in
there that are less than 10kb each. For auth I'm using tempauth. The 404 is
totally random. Sometimes I don't see it, sometimes I see it 5% of the time.

On the proxies, the error message looks like this:

Dec 23 09:00:08 proxy01 proxy-server 10.xx.xx.xx 10.xxx.xxx.xxx
23/Dec/2013/09/00/08 PUT /v1/AUTH_swift/test1/002 HTTP/1.0 404 - -
AUTH_tk275e94194c2447a787e57ec2574789f3 - 70 -
txf2066f1fa5ec403db9791-0052b7fb98 - 0.0367 - -

On the storage servers I see this:

Dec 23 08:49:11 storage1 container-server 10.xx.xx.xxx - -
[23/Dec/2013:08:49:11 +0000] "HEAD /slot-8/11126/AUTH_swift/test1" 404 -
"tx92750eeef61640a3929f4-0052b7f907" "HEAD
http://swift:8080/v1/AUTH_swift/test1" "proxy-server 10178" 0.0002

Note that these are not necessarily from the same request, but it's the
messages that pop up.

Here's what I have for my proxy-server.conf:

[DEFAULT]
bind_port = 8080
workers = 8
user = swift
log_statsd_host = statsd
log_statsd_port = 8125
log_statsd_default_sample_rate = 1
log_statsd_metric_prefix = proxy01

[pipeline:main]
pipeline = healthcheck proxy-logging cache swift3 tempauth proxy-logging
proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:proxy-logging]
use = egg:swift#proxy_logging

[filter:swift3]
use = egg:swift3#swift3

[filter:tempauth]
use = egg:swift#tempauth
[some user] .reseller_admin

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
memcache_servers = 10.xx.xx.xx:11211,10.xx.xx.xx:11211,10.xx.xx.xx:11211,
10.xx.xx.xx:11211,10.xx.xx.xx:11211

On the storage backends I'm using the following:

obect-server.conf

[DEFAULT]
bind_ip = 10.xx.xx.xxx
workers = 16
log_facility = LOG_LOCAL4

pipeline:main]
pipeline = object-server

app:object-server]
use = egg:swift#object

[object-replicator]

[object-updater]

[object-auditor]

The container and account servers look identical to this.

Any ideas?


-- 
Stephen Wood
www.heystephenwood.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131223/799434f6/attachment.html>

More information about the Openstack mailing list