Open Stack

Greetings:

This is most likely to be an administration problem, but I am trying to
use it as a hook to gain understanding into workings of Swift.

I have a test cluster with 2 VMs and 4 "nodes". At some point, I reinstalled
one of the nodes, and now this 404 happens from time to time:

Dec  6 19:08:19 kvm-ni account-server 192.168.129.18 - - [07/Dec/2011:02:08:19 +0000] "HEAD /vdb/18239/AUTH_1" 404 - "-" "-" "-" 0.0010 ""

The root cause, apparently, is that this account does not exist.
On the system which shows no such symptom, we have:

[root at kvm-rei log]# ls -l /srv/node/vdb/accounts/
total 12
drwxrwxrwx. 3 swift swift 4096 Nov 19 00:22 124217
drwxrwxrwx. 3 swift swift 4096 Nov 30 15:36 18239
drwxrwxrwx. 3 swift swift 4096 Nov 18 18:44 236930

On the problem node, we have:

[root at kvm-ni log]# ls -l /srv/node/vdb/accounts/
total 8
drwxrwxrwx. 3 swift swift 4096 Nov 18 22:22 124217
drwxrwxrwx. 3 swift swift 4096 Nov 18 16:50 236930

All I know at present is this blurb in the Administration Guide:

  Replication updates are push based. For object replication,
  updating is just a matter of rsyncing files to the peer.
  Account and container replication push missing records over
  HTTP or rsync whole database files.

So, the suspicion is that the account updater is not running.

Why is it that the accounts are not synchronized? Or in a more
interesting way, what part of code, if any, is supposed to be
responsible for keeping the cluster consistent?

-- Pete