Open Stack

On Tue, Jun 16, 2015 at 9:58 AM, Sean Dague <sean at dague.net> wrote:

> Back when Nova first wanted to test partial upgrade, we did a bunch of
> slightly odd conditionals inside of grenade and devstack to make it so
> that if you were very careful, you could just not stop some of the old
> services on a single node, upgrade everything else, and as long as the
> old services didn't stop, they'd be running cached code in memory, and
> it would look a bit like a 2 node worker not upgraded model. It worked,
> but it was weird.
>
> There has been some interest by the Nova team to expand what's not being
> touched, as well as the Neutron team to add partial upgrade testing
> support. Both are great initiatives, but I think going about it the old
> way is going to add a lot of complexity in weird places, and not be as
> good of a test as we really want.
>
> Nodepool now supports allocating multiple nodes. We have a multinode job
> in Nova regularly testing live migration using this.
>
> If we slice this problem differently, I think we get a better
> architecture, a much easier way to add new configs, and a much more
> realistic end test.
>
> Conceptually, use devstack-gate multinode support to set up 2 nodes, an
> all in one, and a worker. Let grenade upgrade the all in one, leave the
> worker alone.
>
> I think the only complexity here is the fact that grenade.sh implicitly
> drives stack.sh. Which means one of:
>
> 1) devstack-gate could build the worker first, then run grenade.sh
>
> 2) we make it so grenade.sh can execute in parts more easily, so it can
> hand something else running stack.sh for it.'
>
> 3) we make grenade understand the subnode for partial upgrade, so it
> will run the stack phase on the subnode itself (given credentials).
>
> This kind of approach means deciding which services you don't want to
> upgrade doesn't require devstack changes, it's just a change of the
> services on the worker.
>
> We need a volunteer for taking this on, but I think all the follow on
> partial upgrade support will be much much easier to do after we have
> this kind of mechanism in place.
>

I think this is a great approach for the future of partial upgrade support
in grenade. I would like to point out step 0 here, is to get tempest
passing consistently in multinode.

Currently the neutron job is failing consistently, and nova-network fails
roughly 10% of the time due to https://bugs.launchpad.net/nova/+bug/1462305
and https://bugs.launchpad.net/nova/+bug/1445569


>
>         -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150624/1f914261/attachment.html>