[openstack-dev] [grenade] future direction on partial upgrade support
joe.gordon0 at gmail.com
Wed Jun 24 18:01:05 UTC 2015
On Wed, Jun 24, 2015 at 10:45 AM, Sean Dague <sean at dague.net> wrote:
> On 06/24/2015 01:31 PM, Joe Gordon wrote:
> > On Tue, Jun 16, 2015 at 9:58 AM, Sean Dague <sean at dague.net
> > <mailto:sean at dague.net>> wrote:
> > Back when Nova first wanted to test partial upgrade, we did a bunch
> > slightly odd conditionals inside of grenade and devstack to make it
> > that if you were very careful, you could just not stop some of the
> > services on a single node, upgrade everything else, and as long as
> > old services didn't stop, they'd be running cached code in memory,
> > it would look a bit like a 2 node worker not upgraded model. It
> > but it was weird.
> > There has been some interest by the Nova team to expand what's not
> > touched, as well as the Neutron team to add partial upgrade testing
> > support. Both are great initiatives, but I think going about it the
> > way is going to add a lot of complexity in weird places, and not be
> > good of a test as we really want.
> > Nodepool now supports allocating multiple nodes. We have a multinode
> > in Nova regularly testing live migration using this.
> > If we slice this problem differently, I think we get a better
> > architecture, a much easier way to add new configs, and a much more
> > realistic end test.
> > Conceptually, use devstack-gate multinode support to set up 2 nodes,
> > all in one, and a worker. Let grenade upgrade the all in one, leave
> > worker alone.
> > I think the only complexity here is the fact that grenade.sh
> > drives stack.sh. Which means one of:
> > 1) devstack-gate could build the worker first, then run grenade.sh
> > 2) we make it so grenade.sh can execute in parts more easily, so it
> > hand something else running stack.sh for it.'
> > 3) we make grenade understand the subnode for partial upgrade, so it
> > will run the stack phase on the subnode itself (given credentials).
> > This kind of approach means deciding which services you don't want to
> > upgrade doesn't require devstack changes, it's just a change of the
> > services on the worker.
> > We need a volunteer for taking this on, but I think all the follow on
> > partial upgrade support will be much much easier to do after we have
> > this kind of mechanism in place.
> > I think this is a great approach for the future of partial upgrade
> > support in grenade. I would like to point out step 0 here, is to get
> > tempest passing consistently in multinode.
> > Currently the neutron job is failing consistently, and nova-network
> > fails roughly 10% of the time due
> > to https://bugs.launchpad.net/nova/+bug/1462305
> > and https://bugs.launchpad.net/nova/+bug/1445569
> Grenade is only running tempest smoke, which is a quite small number of
> tests (and not the shelve/unshelve one for instance). I would expect
> it's pass rate to be much higher.
One way to find out. Want to get a multinode tempest smoke job running and
see how it looks after running for a few days.
> Sean Dague
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OpenStack-dev