<div dir="ltr">I've run into a very annoying problem with my swift cluster and I'm hoping somebody can help me out. During any given 100 uploads, I'll usually see between 1 to 4% of the calls give a 404. The swift client writes the following:<div>
<br></div><div><font face="courier new, monospace">Object PUT failed: <a href="http://swift:8080/v1/AUTH_swift/test1/042">http://swift:8080/v1/AUTH_swift/test1/042</a> 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<</font></div>
<div><br></div><div>I'm uploading everything to a single bucket and there's only 100 files in there that are less than 10kb each. For auth I'm using tempauth. The 404 is totally random. Sometimes I don't see it, sometimes I see it 5% of the time.</div>
<div><br></div><div>On the proxies, the error message looks like this:</div><div><br></div><div><font face="courier new, monospace">Dec 23 09:00:08 proxy01 proxy-server 10.xx.xx.xx 10.xxx.xxx.xxx 23/Dec/2013/09/00/08 PUT /v1/AUTH_swift/test1/002 HTTP/1.0 404 - - AUTH_tk275e94194c2447a787e57ec2574789f3 - 70 - txf2066f1fa5ec403db9791-0052b7fb98 - 0.0367 - -</font></div>
<div><br></div><div>On the storage servers I see this:</div><div><br></div><div><font face="courier new, monospace">Dec 23 08:49:11 storage1 container-server 10.xx.xx.xxx - - [23/Dec/2013:08:49:11 +0000] "HEAD /slot-8/11126/AUTH_swift/test1" 404 - "tx92750eeef61640a3929f4-0052b7f907" "HEAD <a href="http://swift:8080/v1/AUTH_swift/test1">http://swift:8080/v1/AUTH_swift/test1</a>" "proxy-server 10178" 0.0002</font></div>
<div><br></div><div>Note that these are not necessarily from the same request, but it's the messages that pop up.<div><br></div><div>Here's what I have for my proxy-server.conf:</div><div><br></div><div><div><font face="courier new, monospace">[DEFAULT]</font></div>
<div><font face="courier new, monospace">bind_port = 8080</font></div><div><font face="courier new, monospace">workers = 8</font></div><div><font face="courier new, monospace">user = swift</font></div><div><font face="courier new, monospace">log_statsd_host = statsd</font></div>
<div><font face="courier new, monospace">log_statsd_port = 8125</font></div><div><font face="courier new, monospace">log_statsd_default_sample_rate = 1</font></div><div><font face="courier new, monospace">log_statsd_metric_prefix = proxy01</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[pipeline:main]</font></div><div><font face="courier new, monospace">pipeline = healthcheck proxy-logging cache swift3 tempauth proxy-logging proxy-server</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[app:proxy-server]</font></div><div><font face="courier new, monospace">use = egg:swift#proxy</font></div><div><font face="courier new, monospace">allow_account_management = true</font></div>
<div><font face="courier new, monospace">account_autocreate = true</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[filter:proxy-logging]</font></div><div><font face="courier new, monospace">use = egg:swift#proxy_logging</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[filter:swift3]</font></div><div><font face="courier new, monospace">use = egg:swift3#swift3</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">[filter:tempauth]</font></div></div><div><div><font face="courier new, monospace">use = egg:swift#tempauth</font></div><div><font face="courier new, monospace">[some user] .reseller_admin<br>
</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[filter:healthcheck]</font></div><div><font face="courier new, monospace">use = egg:swift#healthcheck</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[filter:cache]</font></div><div><font face="courier new, monospace">use = egg:swift#memcache</font></div><div><font face="courier new, monospace">memcache_servers = 10.xx.xx.xx:11211,</font><span style="font-family:'courier new',monospace">10.xx.xx.xx:11211,</span><span style="font-family:'courier new',monospace">10.xx.xx.xx:11211,</span><span style="font-family:'courier new',monospace">10.xx.xx.xx:11211,</span><span style="font-family:'courier new',monospace">10.xx.xx.xx:11211</span></div>
</div><div><font face="courier new, monospace"><br></font></div>On the storage backends I'm using the following:</div><div><br></div><div>obect-server.conf</div><div><br><div><font face="courier new, monospace"><div>
[DEFAULT]</div>
<div>bind_ip = 10.xx.xx.xxx</div><div>workers = 16</div><div>log_facility = LOG_LOCAL4</div><div><br></div><div>pipeline:main]</div><div>pipeline = object-server</div><div><br></div><div>app:object-server]</div><div>use = egg:swift#object</div>
<div><br></div><div>[object-replicator]</div><div><br></div><div>[object-updater]</div><div><br></div><div>[object-auditor]</div></font></div><div><font face="courier new, monospace"><br></font></div>The container and account servers look identical to this.</div>
<div><br></div><div>Any ideas?</div><div><br></div><div><br></div><div>-- <br><div dir="ltr">Stephen Wood<div><a href="http://www.heystephenwood.com" target="_blank">www.heystephenwood.com</a></div></div>
</div></div>