Open Stack

@ DuncanT and @dule: 

I noticed from IRC log that you are discussing about c-bak upgrade, and I am working on this, please see following message. Hope I don't miss anything.

As you know, currently c-bak and c-vol are in same nodes. c-bak depends on c-vol service. 

But it is not necessary that all c-vols needs to be upgraded before c-backs.

The sequences can be random. As described in the patch https://review.openstack.org/#/c/269412/,
when c-api decides which c-bak service to create/restore, it checks the version of c-vol. If c-vol is new version, find a c-bak in new version.
If c-vol is in old version, find a c-bak in old version.

Let's us think a real case. Customers upgrade c-api, c-sch, and start to upgrade c-vol and c-bak.
There are two c-vol services c-vol1 and c-vol2, and two c-bak services c-bak1 and c-bak2.
There are four typical upgrade sequences as following. 
Meanwhile, please notice that c-vol and c-bak are in same nodes in Liberty. So during upgrades, if c-vol went down to upgrade, c-bak is also down. 

1. c-vol1->c-bak1->c-vol2->c-bak2
The only insufficiency is when c-vol1 upgrades, and other c-bak services haven't upgraded, the request to create/restore backups from/to volumes in c-vol1 will fail with reason "no valid c-bak service
found". It is acceptable, as it is similar to scenario in Liberty: c-vol active and c-bak fails.

2. c-vol1->c-vol2->c-bak1->c-bak2
Before c-bak1 upgrades, no back request can be completed as no active c-bak services. This is reasonable.

3. c-bak1->c-vol1->c-bak2->c-vol2:
The issue is when c-bak2 upgrades, the request to create/restore backups from/to volumes in c-vol2 will fail with reason c-vol not active. This is consistent with behaviors in Liberty.

4. c-bak1->c-bak2->c-vol1->c-vol2: 
Before c-vol1 upgrades, no back request can be completed as c-vol services not active This is reasonable.


Best wishes
Lisa

On Jan 20, 2016 21:59, Michał Dulko wrote:
> Hi,
> 
> In Mitaka Cinder team is implementing rolling upgrades capabilities. I
> want to get some feedback on possibilities of implementing some partial or
> multinode grenade check for our purposes.
> 
> Basically Cinder consists of 4 services calling each other over RPC the in
> following fashion:
> 
>   +---------+ c-api +---------+
>   |             +             |
>   |             |             |
>   v             v             v
> c-sch <-----> c-vol <-----+ c-bak
> 
> The order of upgrades we're forced to use (at least in this release) is
> c-api->c-sch->c-vol->c-bak. I wonder how we can test interoperability of
> services in different versions in that model. I have three ideas:
> 
> 1. One idea would be to have two nodes - one with full Cinder deployment
> in latest master and second one running c-sch, c-vol and c-bak in latest
> stable version. That way we would test interoperability of the services
> versions. A problem I see is that tests would be strongly nondeterministic,
> as a particular test result would depend on which RPC service got the
> request. That would make debugging CI failures harder and may result in
> breaking patches slipping in.
> 
> 2. We may simply run a controller<->compute multinode, similar to how
> Nova is running multinode grenade job. Controller would run c-api, c-sch
> and compute c-vol, c-bak. Disadvantage of this model is the fact that we
> wouldn't test c-api->c-sch compatibility.
> 
> 3. Do a single upgrade for each of the services. That would mean testing
> master c-api with stable rest of services, then upgrading c-sch, retesting,
> upgrading c-vol, retesting, upgrading c-bak, retesting. That way we would
> test all the possibilities but such run would take a lot of time. Moreover if I
> recall correctly such idea isn't possible in current state of Grenade.
> 
> Comments and feedback is very welcome.
> 
> Thanks,
> Michal (IRC: dulek)
> 
 _____________________________________________________________
 _____________ OpenStack Development Mailing List (not for usage
 questions) Unsubscribe: OpenStack-dev-
 request at lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Best wishes
Lisa