[openstack-dev] [puppet] [swift] Storage service startup should await ring creation
mark.kirkwood at catalyst.net.nz
Fri Jul 10 00:43:43 UTC 2015
I am using puppet-swift to deploy a swift multi node cluster (Icehouse),
following the setup in supplied tests/site.pp. I am running into two
issues that seem to be related to the subject above:
1/ Errors when the storage replication services try to start before the
ring files exist. e.g:
Error: Could not start Service[swift-object-replicator]: Execution of
'/sbin/start swift-object-replicator' returned 1: start: Job failed to start
Execution of '/sbin/start swift-object-replicator' returned 1: start:
Job failed to start
change from stopped to running failed: Could not start
Service[swift-object-replicator]: Execution of '/sbin/start
swift-object-replicator' returned 1: start: Job failed to start
Now these will be fixed the *next* time I do a puppet run (provided I've
performed a run on the appropriate proxy/ringmaster). However the
failing services make scripted testing difficult as we have to put in
logic to the effect "don't worry about errors the 1st time".
2/ Container and object stats not updated without full restart of services
This one is a bit more subtle - everything works but 'swift stat' always
shows zero objects and bytes for every container. The only way to fix
this is to stop and start all services on each storage node.
Again this complicates scripted builds as there is the need to go and
stop + start all the swift storage services! Not to mention an extra
little quirk for ops to remember at zero dark 30 oclock...
I've made a patch that prevents these services starting until the ring
files exist (actually for now it just checks the object ring) - see
Now while I'm not entirely sure that this is the best way to solve the
issue (custom fact that changes the service start flag)...I *do* think
that making the storage services await the ring existence *is* a needed
change, so any thoughts on better ways to do this are appreciated.
Also note that this change *does* require one more puppet run on the
- one to create the storage servers config and drives
- one to get the ring from the proxy/ringmaster
- one to start the services
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 6654 bytes
Desc: not available
More information about the OpenStack-dev