Open Stack

On 06/24/2015 01:31 PM, Joe Gordon wrote:
> 
> 
> On Tue, Jun 16, 2015 at 9:58 AM, Sean Dague <sean at dague.net
> <mailto:sean at dague.net>> wrote:
> 
>     Back when Nova first wanted to test partial upgrade, we did a bunch of
>     slightly odd conditionals inside of grenade and devstack to make it so
>     that if you were very careful, you could just not stop some of the old
>     services on a single node, upgrade everything else, and as long as the
>     old services didn't stop, they'd be running cached code in memory, and
>     it would look a bit like a 2 node worker not upgraded model. It worked,
>     but it was weird.
> 
>     There has been some interest by the Nova team to expand what's not being
>     touched, as well as the Neutron team to add partial upgrade testing
>     support. Both are great initiatives, but I think going about it the old
>     way is going to add a lot of complexity in weird places, and not be as
>     good of a test as we really want.
> 
>     Nodepool now supports allocating multiple nodes. We have a multinode job
>     in Nova regularly testing live migration using this.
> 
>     If we slice this problem differently, I think we get a better
>     architecture, a much easier way to add new configs, and a much more
>     realistic end test.
> 
>     Conceptually, use devstack-gate multinode support to set up 2 nodes, an
>     all in one, and a worker. Let grenade upgrade the all in one, leave the
>     worker alone.
> 
>     I think the only complexity here is the fact that grenade.sh implicitly
>     drives stack.sh. Which means one of:
> 
>     1) devstack-gate could build the worker first, then run grenade.sh
> 
>     2) we make it so grenade.sh can execute in parts more easily, so it can
>     hand something else running stack.sh for it.'
> 
>     3) we make grenade understand the subnode for partial upgrade, so it
>     will run the stack phase on the subnode itself (given credentials).
> 
>     This kind of approach means deciding which services you don't want to
>     upgrade doesn't require devstack changes, it's just a change of the
>     services on the worker.
> 
>     We need a volunteer for taking this on, but I think all the follow on
>     partial upgrade support will be much much easier to do after we have
>     this kind of mechanism in place.
> 
> 
> I think this is a great approach for the future of partial upgrade
> support in grenade. I would like to point out step 0 here, is to get
> tempest passing consistently in multinode.
> 
> Currently the neutron job is failing consistently, and nova-network
> fails roughly 10% of the time due
> to https://bugs.launchpad.net/nova/+bug/1462305
> and https://bugs.launchpad.net/nova/+bug/1445569

Grenade is only running tempest smoke, which is a quite small number of
tests (and not the shelve/unshelve one for instance). I would expect
it's pass rate to be much higher.

	-Sean

-- 
Sean Dague
http://dague.net