Open Stack

For the sake of completion, the whole thing was a false alarm, has
nothing to do with db.pending files and it has been documented here:

https://bugs.launchpad.net/swift/+bug/1172358


On Tue, Apr 23, 2013 at 11:39 AM, Sergio Rubio <rubiojr at frameos.org> wrote:
> Howdy folks,
>
> While populating an empty Swift test cluster with
> swift-dispersion-populate some container creation requests failed,
> leaving 'db.pending' files.
>
> This is standard operation I think
> (http://docs.openstack.org/developer/swift/overview_architecture.html#updaters)
> however the pending files are never removed (the updaters aren't
> picking up those changes?) and I can see periodic errors in the
> storage node error log:
>
> <pre>
> root at swift-002:/srv/node# find|grep pending
> ./cd6b760c-bf47-4733-b7bb-23659dd8ee1d/accounts/523880/adc/e2cfc6be58be71d5ad6111364fff0adc/e2cfc6be58be71d5ad6111364fff0adc.db.pending
> ./7e0cfcfc-7c6e-41a4-adc8-e4173147bd2a/accounts/523880/adc/e2cfc6be58be71d5ad6111364fff0adc/e2cfc6be58be71d5ad6111364fff0adc.db.pending
> ./7cf651d7-e2f7-4dc4-ada0-7cd0dc832449/containers/498724/d1d/bd66b837c7b9a95dec68b3ca05a75d1d/bd66b837c7b9a95dec68b3ca05a75d1d.db.pending
> </pre>
>
>
> <pre>
> Apr 23 11:06:26 swift-001 account-server ERROR __call__ error with PUT
> /d3829495-9f10-4075-a558-f99fb665cfe2/464510/AUTH_aeedb7bdcb8846599e2b6b87bb8a947f/dispersion_916bc3a9cc1548aca41be6633c1bd194
> : #012Traceback (most recent call last):#012  File
> "/usr/lib/python2.7/dist-packages/swift/account/server.py", line 333,
> in __call__#012    res = method(req)#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1558,
> in wrapped#012    return func(*a, **kw)#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 520, in
> _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File
> "/usr/lib/python2.7/dist-packages/swift/account/server.py", line 112,
> in PUT#012    req.headers['x-bytes-used'])#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 1431, in
> put_container#012    raise DatabaseConnectionError(self.db_file, "DB
> doesn't exist")#012DatabaseConnectionError: DB connection error
> (/srv/node/d3829495-9f10-4075-a558-f99fb665cfe2/accounts/464510/adc/e2cfc6be58be71d5ad6111364fff0adc/e2cfc6be58be71d5ad6111364fff0adc.db,
> 0):#012DB doesn't exist
> </pre>
>
> The test cluster has two storage nodes with 12 drives each, packaged
> Swift 1.8.0 from Ubuntu Cloud Archive and the account/container
> configuration is the following:
> https://gist.github.com/rubiojr/6ea3d0ea0c4d00949d33
>
> The cluster is fully operational and there are no other known issues
> ATM. I can easily reproduce the problem by emptying the cluster and
> populating it again with swift-dispersion-populate.
>
> I've exhausted all the possibilities (bug reports, mailing lists,
> google, etc) so I decided to ask before start digging the source code
> trying to gather evidence to open a bug report.
>
> If anyone can shed some light on the issue that would be great.
>
> Thanks!