Open Stack

I've been looking into the startup constraints involved when launching
Nova services with systemd using Type=notify (which causes systemd to
wait for an explicit notification from the service before considering
it to be "started".  Some services (e.g., nova-conductor) will happily
"start" even if the backing database is currently unavailable (and
will enter a retry loop waiting for the database).

Other services -- specifically, nova-scheduler -- will block waiting
for the database *before* providing systemd with the necessary
notification.

nova-scheduler blocks because it wants to initialize a list of
available aggregates (in scheduler.host_manager.HostManager.__init__),
which it gets by calling objects.AggregateList.get_all.

Does it make sense to block service startup at this stage?  The
database disappearing during runtime isn't a hard error -- we will
retry and reconnect when it comes back -- so should the same situation
at startup be a hard error?  As an operator, I am more interested in
"did my configuration files parse correctly?" at startup, and would
generally prefer the service to start (and permit any dependent
services to start) even when the database isn't up (because that's
probably a situation of which I am already aware).

It would be relatively easy to have the scheduler lazy-load the list
of aggregates on first references, rather than at __init__.  I'm not
familiar enough with the nova code to know if there would be any
undesirable implications of this behavior.  We're already punting
initializing the list of instances to an asynchronous task in order to
avoid blocking service startup.

Does it make sense to permit nova-scheduler to complete service
startup in the absence of the database (and then retry the connection
in the background)?

-- 
Lars Kellogg-Stedman <lars at redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack          | http://blog.oddbit.com/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151223/daf1f59a/attachment.pgp>