Open Stack

Excerpts from Zane Bitter's message of 2015-05-28 11:24:33 -0700:
> On 28/05/15 03:35, Jan Provaznik wrote:
> > On 05/28/2015 01:10 AM, Steve Baker wrote:
> >> On 28/05/15 10:54, Richard Raseley wrote:
> >>> Zane Bitter wrote:
> >>>> Steve is working on a patch to allow package-based updates of overcloud
> >>>> nodes[1] using the distro's package manager (yum in the case of RDO,
> >>>> but
> >>>> conceivable apt in others). Note we're talking exclusively about minor
> >>>> updates, not version-to-version upgrades here.
> >>>>
> >>>> Dan mentioned at the summit that this approach fails to take into
> >>>> account the complex ballet of service restarts required to update
> >>>> OpenStack services. (/me shakes fist at OpenStack services.) And
> >>>> furthermore, that the Puppet manifests already encode the necessary
> >>>> relationships to do this properly. (Thanks Puppeteers!) Indeed we'd be
> >>>> doing the Wrong Thing by Puppet if we changed this stuff from under it.
> >>>>
> >>>> The problem of course is that neither Puppet nor yum/apt has a view of
> >>>> the entire system. Yum doesn't know about the relationships between
> >>>> services and Puppet doesn't know about all of the _other_ packages that
> >>>> they depend on.
> >>>>
> >>>> One solution proposed was to do a yum update first but specifically
> >>>> exclude any packages that Puppet knows about (the --excludes flag
> >>>> appears sufficient for this); then follow that up with another Puppet
> >>>> run using ensure -> latest.
> >>>>
> >> My only concern with this approach is how do we collect and maintain the
> >> excludes list. Other than that it sounds reasonable.
> >
> > Why not swap the order? First run puppet using ensure=>latest which
> > updates/restarts everything Openstack depends on, then run yum/apt
> > update to update any remaining bits. You wouldn't need exclude list then.
> 
> Will ensure=>latest update all packages that the given one is dependent 
> on, even if it doesn't require new versions? I assumed that it wouldn't, 
> so by doing Puppet first we would just ensure that we are even less 
> likely to pick up library changes by restarting services after the 
> libraries are updated.
> 

No.

> >>>> A problem with that approach is that it still fails to restart services
> >>>> which have had libraries updated but have not themselves been updated.
> >>>> That's no worse than the pure yum approach though. We might need an
> >>>> additional way to just manually trigger a restart of services.
> >>
> >> Maybe this could be handled at the packaging stage by reving the package
> >> version when there is a known fix in a low-level library, thus
> >> triggering a service restart in the puppet phase.
> >>
> >
> > My concern is that then e.g. libc update would mean repackaging (bumping
> > version) of zillion of other packages, also zillion of packages would be
> > downloaded/upgraded on each system because of a new package version.
> 
> Yes, no distro ever created works like this - for good reason - and it 
> is waaaaaaaaay out of scope for us to try to change that :)
>

There are a number of clever things that can detect C shared libraries
that need to be reloaded. One such thing is checkrestart from
debian-goodies. It combs through lsof to do that. Not possible for
python libs and dlopened plugins though.

This is one of the arguments for containers -- at least you know if you
changed something, it's time for a new container.

> > I think that providing user a manual (script) way to restart services
> > after update would be sufficient solution (though not so sophisticated).
> 
> Maybe there's a way we can poke Puppet to do it.

This is one big flaw with the "declare the world and converge on it"
model. Often there is a single diminutive workflow that could be used to
achieve the end-goal and one would be better off expending their energy
on tests to ensure that the end state is reached than trying to figure
out how to express all the things as DSL that is actually achievable.