Open Stack

On Wed, Jun 24, 2015 at 11:02 AM, Joe Gordon <joe.gordon0 at gmail.com> wrote:

>
>
> On Wed, Jun 24, 2015 at 11:01 AM, Joe Gordon <joe.gordon0 at gmail.com>
> wrote:
>
>>
>>
>> On Wed, Jun 24, 2015 at 10:45 AM, Sean Dague <sean at dague.net> wrote:
>>
>>> On 06/24/2015 01:31 PM, Joe Gordon wrote:
>>> >
>>> >
>>> > On Tue, Jun 16, 2015 at 9:58 AM, Sean Dague <sean at dague.net
>>> > <mailto:sean at dague.net>> wrote:
>>> >
>>> >     Back when Nova first wanted to test partial upgrade, we did a
>>> bunch of
>>> >     slightly odd conditionals inside of grenade and devstack to make
>>> it so
>>> >     that if you were very careful, you could just not stop some of the
>>> old
>>> >     services on a single node, upgrade everything else, and as long as
>>> the
>>> >     old services didn't stop, they'd be running cached code in memory,
>>> and
>>> >     it would look a bit like a 2 node worker not upgraded model. It
>>> worked,
>>> >     but it was weird.
>>> >
>>> >     There has been some interest by the Nova team to expand what's not
>>> being
>>> >     touched, as well as the Neutron team to add partial upgrade testing
>>> >     support. Both are great initiatives, but I think going about it
>>> the old
>>> >     way is going to add a lot of complexity in weird places, and not
>>> be as
>>> >     good of a test as we really want.
>>> >
>>> >     Nodepool now supports allocating multiple nodes. We have a
>>> multinode job
>>> >     in Nova regularly testing live migration using this.
>>> >
>>> >     If we slice this problem differently, I think we get a better
>>> >     architecture, a much easier way to add new configs, and a much more
>>> >     realistic end test.
>>> >
>>> >     Conceptually, use devstack-gate multinode support to set up 2
>>> nodes, an
>>> >     all in one, and a worker. Let grenade upgrade the all in one,
>>> leave the
>>> >     worker alone.
>>> >
>>> >     I think the only complexity here is the fact that grenade.sh
>>> implicitly
>>> >     drives stack.sh. Which means one of:
>>> >
>>> >     1) devstack-gate could build the worker first, then run grenade.sh
>>> >
>>> >     2) we make it so grenade.sh can execute in parts more easily, so
>>> it can
>>> >     hand something else running stack.sh for it.'
>>> >
>>> >     3) we make grenade understand the subnode for partial upgrade, so
>>> it
>>> >     will run the stack phase on the subnode itself (given credentials).
>>> >
>>> >     This kind of approach means deciding which services you don't want
>>> to
>>> >     upgrade doesn't require devstack changes, it's just a change of the
>>> >     services on the worker.
>>> >
>>> >     We need a volunteer for taking this on, but I think all the follow
>>> on
>>> >     partial upgrade support will be much much easier to do after we
>>> have
>>> >     this kind of mechanism in place.
>>> >
>>> >
>>> > I think this is a great approach for the future of partial upgrade
>>> > support in grenade. I would like to point out step 0 here, is to get
>>> > tempest passing consistently in multinode.
>>> >
>>> > Currently the neutron job is failing consistently, and nova-network
>>> > fails roughly 10% of the time due
>>> > to https://bugs.launchpad.net/nova/+bug/1462305
>>> > and https://bugs.launchpad.net/nova/+bug/1445569
>>>
>>> Grenade is only running tempest smoke, which is a quite small number of
>>> tests (and not the shelve/unshelve one for instance). I would expect
>>> it's pass rate to be much higher.
>>>
>>>
>> One way to find out. Want to get a multinode tempest smoke job running
>> and see how it looks after running for a few days.
>>
>
> smoke jobs*, one for nova-net and one for neutron.
>
>

Proposal for multinode smoke jobs: https://review.openstack.org/#/c/195259/




>
>>
>>>         -Sean
>>>
>>> --
>>> Sean Dague
>>> http://dague.net
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150624/483e1bf5/attachment.html>