<div dir="ltr">I'm also seeing this in the object-server logs:<div><br></div><div><div><font face="courier new, monospace">12OperationalError: database is locked#012 (txn: tx37d4e1807957447cac403-0052b89cbb)</font></div>
<div><font face="courier new, monospace">Dec 23 20:27:39 store01 container-server ERROR __call__ error with PUT /slot-3/11126/AUTH_swift/test1 : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/container/server.py", line 486, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1870, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 686, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/container/server.py", line 262, in PUT#012 created = broker.is_deleted()#012 File "/usr/lib/python2.7/dist-packages/swift/container/backend.py", line 246, in is_deleted#012 with self.get() as conn:#012 File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__#012 return self.gen.next()#012 File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 325, in get#012 self.possibly_quarantine(*sys.exc_info())#012 File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 323, in get#012 self.conn = get_db_connection(self.db_file, self.timeout)#012 File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 167, in get_db_connection#012 timeout=timeout)#012DatabaseConnectionError: DB connection error (/srv/node/slot-3/containers/11126/5ab/56edb8dfe26326806d33c3c73aeb65ab/56edb8dfe26326806d33c3c73aeb65ab.db, 25):#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 159, in get_db_connection#012 cur.execute('PRAGMA synchronous = NORMAL')#012OperationalError: database is locked#012 (txn: tx0171c51f618f4dbbbc712-0052b89cbb)</font></div>
</div><div><font face="courier new, monospace"><br></font></div>I'll see this error even during successful uploads, which has me wondering if something is borked on my container service for these hosts, or if locks aren't properly getting released.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Dec 23, 2013 at 12:13 PM, Stephen Wood <span dir="ltr"><<a href="mailto:smwood4@gmail.com" target="_blank">smwood4@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I've run into a very annoying problem with my swift cluster and I'm hoping somebody can help me out. During any given 100 uploads, I'll usually see between 1 to 4% of the calls give a 404. The swift client writes the following:<div>
<br></div><div><font face="courier new, monospace">Object PUT failed: <a href="http://swift:8080/v1/AUTH_swift/test1/042" target="_blank">http://swift:8080/v1/AUTH_swift/test1/042</a> 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<</font></div>
<div><br></div><div>I'm uploading everything to a single bucket and there's only 100 files in there that are less than 10kb each. For auth I'm using tempauth. The 404 is totally random. Sometimes I don't see it, sometimes I see it 5% of the time.</div>
<div><br></div><div>On the proxies, the error message looks like this:</div><div><br></div><div><font face="courier new, monospace">Dec 23 09:00:08 proxy01 proxy-server 10.xx.xx.xx 10.xxx.xxx.xxx 23/Dec/2013/09/00/08 PUT /v1/AUTH_swift/test1/002 HTTP/1.0 404 - - AUTH_tk275e94194c2447a787e57ec2574789f3 - 70 - txf2066f1fa5ec403db9791-0052b7fb98 - 0.0367 - -</font></div>
<div><br></div><div>On the storage servers I see this:</div><div><br></div><div><font face="courier new, monospace">Dec 23 08:49:11 storage1 container-server 10.xx.xx.xxx - - [23/Dec/2013:08:49:11 +0000] "HEAD /slot-8/11126/AUTH_swift/test1" 404 - "tx92750eeef61640a3929f4-0052b7f907" "HEAD <a href="http://swift:8080/v1/AUTH_swift/test1" target="_blank">http://swift:8080/v1/AUTH_swift/test1</a>" "proxy-server 10178" 0.0002</font></div>
<div><br></div><div>Note that these are not necessarily from the same request, but it's the messages that pop up.<div><br></div><div>Here's what I have for my proxy-server.conf:</div><div><br></div><div><div><font face="courier new, monospace">[DEFAULT]</font></div>
<div><font face="courier new, monospace">bind_port = 8080</font></div><div><font face="courier new, monospace">workers = 8</font></div><div><font face="courier new, monospace">user = swift</font></div><div><font face="courier new, monospace">log_statsd_host = statsd</font></div>
<div><font face="courier new, monospace">log_statsd_port = 8125</font></div><div><font face="courier new, monospace">log_statsd_default_sample_rate = 1</font></div><div><font face="courier new, monospace">log_statsd_metric_prefix = proxy01</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[pipeline:main]</font></div><div><font face="courier new, monospace">pipeline = healthcheck proxy-logging cache swift3 tempauth proxy-logging proxy-server</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[app:proxy-server]</font></div><div><font face="courier new, monospace">use = egg:swift#proxy</font></div><div><font face="courier new, monospace">allow_account_management = true</font></div>
<div><font face="courier new, monospace">account_autocreate = true</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[filter:proxy-logging]</font></div><div><font face="courier new, monospace">use = egg:swift#proxy_logging</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[filter:swift3]</font></div><div><font face="courier new, monospace">use = egg:swift3#swift3</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">[filter:tempauth]</font></div></div><div><div><font face="courier new, monospace">use = egg:swift#tempauth</font></div><div><font face="courier new, monospace">[some user] .reseller_admin<br>
</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[filter:healthcheck]</font></div><div><font face="courier new, monospace">use = egg:swift#healthcheck</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[filter:cache]</font></div><div><font face="courier new, monospace">use = egg:swift#memcache</font></div><div><font face="courier new, monospace">memcache_servers = 10.xx.xx.xx:11211,</font><span style="font-family:'courier new',monospace">10.xx.xx.xx:11211,</span><span style="font-family:'courier new',monospace">10.xx.xx.xx:11211,</span><span style="font-family:'courier new',monospace">10.xx.xx.xx:11211,</span><span style="font-family:'courier new',monospace">10.xx.xx.xx:11211</span></div>
</div><div><font face="courier new, monospace"><br></font></div>On the storage backends I'm using the following:</div><div><br></div><div>obect-server.conf</div><div><br><div><font face="courier new, monospace"><div>
[DEFAULT]</div>
<div>bind_ip = 10.xx.xx.xxx</div><div>workers = 16</div><div>log_facility = LOG_LOCAL4</div><div><br></div><div>pipeline:main]</div><div>pipeline = object-server</div><div><br></div><div>app:object-server]</div><div>use = egg:swift#object</div>
<div><br></div><div>[object-replicator]</div><div><br></div><div>[object-updater]</div><div><br></div><div>[object-auditor]</div></font></div><div><font face="courier new, monospace"><br></font></div>The container and account servers look identical to this.</div>
<div><br></div><div>Any ideas?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div><br></div><div>-- <br><div dir="ltr">Stephen Wood<div><a href="http://www.heystephenwood.com" target="_blank">www.heystephenwood.com</a></div>
</div>
</div></font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Stephen Wood<div><a href="http://www.heystephenwood.com" target="_blank">www.heystephenwood.com</a></div></div>
</div>