Open Stack

Howdy folks,

While populating an empty Swift test cluster with
swift-dispersion-populate some container creation requests failed,
leaving 'db.pending' files.

This is standard operation I think
(http://docs.openstack.org/developer/swift/overview_architecture.html#updaters)
however the pending files are never removed (the updaters aren't
picking up those changes?) and I can see periodic errors in the
storage node error log:

<pre>
root at swift-002:/srv/node# find|grep pending
./cd6b760c-bf47-4733-b7bb-23659dd8ee1d/accounts/523880/adc/e2cfc6be58be71d5ad6111364fff0adc/e2cfc6be58be71d5ad6111364fff0adc.db.pending
./7e0cfcfc-7c6e-41a4-adc8-e4173147bd2a/accounts/523880/adc/e2cfc6be58be71d5ad6111364fff0adc/e2cfc6be58be71d5ad6111364fff0adc.db.pending
./7cf651d7-e2f7-4dc4-ada0-7cd0dc832449/containers/498724/d1d/bd66b837c7b9a95dec68b3ca05a75d1d/bd66b837c7b9a95dec68b3ca05a75d1d.db.pending
</pre>


<pre>
Apr 23 11:06:26 swift-001 account-server ERROR __call__ error with PUT
/d3829495-9f10-4075-a558-f99fb665cfe2/464510/AUTH_aeedb7bdcb8846599e2b6b87bb8a947f/dispersion_916bc3a9cc1548aca41be6633c1bd194
: #012Traceback (most recent call last):#012  File
"/usr/lib/python2.7/dist-packages/swift/account/server.py", line 333,
in __call__#012    res = method(req)#012  File
"/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1558,
in wrapped#012    return func(*a, **kw)#012  File
"/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 520, in
_timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File
"/usr/lib/python2.7/dist-packages/swift/account/server.py", line 112,
in PUT#012    req.headers['x-bytes-used'])#012  File
"/usr/lib/python2.7/dist-packages/swift/common/db.py", line 1431, in
put_container#012    raise DatabaseConnectionError(self.db_file, "DB
doesn't exist")#012DatabaseConnectionError: DB connection error
(/srv/node/d3829495-9f10-4075-a558-f99fb665cfe2/accounts/464510/adc/e2cfc6be58be71d5ad6111364fff0adc/e2cfc6be58be71d5ad6111364fff0adc.db,
0):#012DB doesn't exist
</pre>

The test cluster has two storage nodes with 12 drives each, packaged
Swift 1.8.0 from Ubuntu Cloud Archive and the account/container
configuration is the following:
https://gist.github.com/rubiojr/6ea3d0ea0c4d00949d33

The cluster is fully operational and there are no other known issues
ATM. I can easily reproduce the problem by emptying the cluster and
populating it again with swift-dispersion-populate.

I've exhausted all the possibilities (bug reports, mailing lists,
google, etc) so I decided to ask before start digging the source code
trying to gather evidence to open a bug report.

If anyone can shed some light on the issue that would be great.

Thanks!