Open Stack

On 06/24/2015 01:31 PM, Joe Gordon wrote:
> 
> 
> On Tue, Jun 16, 2015 at 9:58 AM, Sean Dague <sean at dague.net
> <mailto:sean at dague.net>> wrote:
> 
>     Back when Nova first wanted to test partial upgrade, we did a bunch of
>     slightly odd conditionals inside of grenade and devstack to make it so
>     that if you were very careful, you could just not stop some of the old
>     services on a single node, upgrade everything else, and as long as the
>     old services didn't stop, they'd be running cached code in memory, and
>     it would look a bit like a 2 node worker not upgraded model. It worked,
>     but it was weird.
> 
>     There has been some interest by the Nova team to expand what's not being
>     touched, as well as the Neutron team to add partial upgrade testing
>     support. Both are great initiatives, but I think going about it the old
>     way is going to add a lot of complexity in weird places, and not be as
>     good of a test as we really want.
> 
>     Nodepool now supports allocating multiple nodes. We have a multinode job
>     in Nova regularly testing live migration using this.
> 
>     If we slice this problem differently, I think we get a better
>     architecture, a much easier way to add new configs, and a much more
>     realistic end test.
> 
>     Conceptually, use devstack-gate multinode support to set up 2 nodes, an
>     all in one, and a worker. Let grenade upgrade the all in one, leave the
>     worker alone.
> 
>     I think the only complexity here is the fact that grenade.sh implicitly
>     drives stack.sh. Which means one of:
> 
>     1) devstack-gate could build the worker first, then run grenade.sh
> 
>     2) we make it so grenade.sh can execute in parts more easily, so it can
>     hand something else running stack.sh for it.'
> 
>     3) we make grenade understand the subnode for partial upgrade, so it
>     will run the stack phase on the subnode itself (given credentials).
> 
>     This kind of approach means deciding which services you don't want to
>     upgrade doesn't require devstack changes, it's just a change of the
>     services on the worker.
> 
>     We need a volunteer for taking this on, but I think all the follow on
>     partial upgrade support will be much much easier to do after we have
>     this kind of mechanism in place.
> 
> 
> I think this is a great approach for the future of partial upgrade
> support in grenade. I would like to point out step 0 here, is to get
> tempest passing consistently in multinode.
> 
> Currently the neutron job is failing consistently, and nova-network
> fails roughly 10% of the time due
> to https://bugs.launchpad.net/nova/+bug/1462305
> and https://bugs.launchpad.net/nova/+bug/1445569

If multi-node isn't reliable more generally yet, do you think the
simpler implementation of partial-upgrade testing could proceed?  I've
already done all of the patches to do it for Neutron.  That way we could
quickly get something in place to help block regressions and work on the
longer-term multinode refactoring without as much time pressure.

For reference, the Neutron related partial-upgrade test related patches are:

devstack patches:
  https://review.openstack.org/189408
  https://review.openstack.org/189707
  https://review.openstack.org/189710

grenade patches:
  https://review.openstack.org/189417
  https://review.openstack.org/189712

devstack-gate patches:
  https://review.openstack.org/189424
  https://review.openstack.org/189715

project-config patches:
  https://review.openstack.org/189426
  https://review.openstack.org/189727

-- 
Russell Bryant