Open Stack

Hi,

In Mitaka Cinder team is implementing rolling upgrades capabilities. I
want to get some feedback on possibilities of implementing some partial
or multinode grenade check for our purposes.

Basically Cinder consists of 4 services calling each other over RPC the
in following fashion:

  +---------+ c-api +---------+
  |             +             |
  |             |             |
  v             v             v
c-sch <-----> c-vol <-----+ c-bak

The order of upgrades we're forced to use (at least in this release) is
c-api->c-sch->c-vol->c-bak. I wonder how we can test interoperability of
services in different versions in that model. I have three ideas:

1. One idea would be to have two nodes - one with full Cinder deployment
in latest master and second one running c-sch, c-vol and c-bak in latest
stable version. That way we would test interoperability of the services
versions. A problem I see is that tests would be strongly
nondeterministic, as a particular test result would depend on which RPC
service got the request. That would make debugging CI failures harder
and may result in breaking patches slipping in.

2. We may simply run a controller<->compute multinode, similar to how
Nova is running multinode grenade job. Controller would run c-api, c-sch
and compute c-vol, c-bak. Disadvantage of this model is the fact that we
wouldn't test c-api->c-sch compatibility.

3. Do a single upgrade for each of the services. That would mean testing
master c-api with stable rest of services, then upgrading c-sch,
retesting, upgrading c-vol, retesting, upgrading c-bak, retesting. That
way we would test all the possibilities but such run would take a lot of
time. Moreover if I recall correctly such idea isn't possible in current
state of Grenade.

Comments and feedback is very welcome.

Thanks,
Michal (IRC: dulek)