Open Stack

An interesting conversation has cropped up over the last few days in -qa
and -infra which I want to bring to the wider OpenStack community. When
discussing the use of Tempest as part of the Defcore validation we came
to an interesting question:

Why does Tempest have stable/* branches? Does it need them?

Historically the Tempest project has created a stable/foo tag the week
of release to lock the version of Tempest that will be tested against
stable branches. The reason we did that is until this cycle we had
really limited nobs in tempest to control which features were tested.
stable/havana means - test everything we know how to test in havana. So
when, for instance, a new API extension landed upstream in icehouse,
we'd just add the tests to Tempest. It wouldn't impact stable/havana,
because we wouldn't backport changes.

But is this really required?

For instance, we don't branch openstack clients. They are supposed to
work against multiple server versions. Tempest, at some level, is
another client. So there is some sense there.

Tempest now also have flags on features, and tests are skippable if
services, or even extensions aren't enabled (all explicitly setable in
the tempest.conf). This is a much better control mechanism than the
course grained selection of stable/foo.


If we decided not to set a stable/icehouse branch in 2 weeks, the gate
would change as follows:

Project masters: no change
Project stable/icehouse: would be gated against Tempest master
Tempest master: would double the gate jobs, gate on project master and
project stable/icehouse on every commit.

(That last one needs infra changes to work right, those are all in
flight right now to assess doability.)

Some interesting effects this would have:

 * Tempest test enhancements would immediately apply on stable/icehouse *

... giving us more confidence. A large amount of tests added to master
in every release are enhanced checking for existing function.

 * Tempest test changes would need server changes in master and
stable/icehouse *

In trying tempest master against stable/havana we found a number of
behavior changes in projects that there had been a 2 step change in the
Tempest tests to support. But this actually means that stable/havana and
stable/icehouse for the same API version are different. Going forward
this would require master + stable changes on the projects + Tempest
changes. Which would provide much more friction in changing these sorts
of things by accident.

 * Much more stable testing *

If every Tempest change is gating on stable/icehouse, the week long
stable/havana can't pass tests won't happen. There will be much more
urgency to keep stable branches functioning.


If we got rid of branches in Tempest the path would be:
 * infrastructure to support this in infra - in process, probably
landing today
 * don't set stable/icehouse - decision needed by Apr 17th
 * changes to d-g/devstack to be extra explicit about what features
stable/icehouse should support in tempest.conf
 * see if we can make master work with stable/havana to remove the
stable/havana Tempest branch (if this is doable in a month, great, if
not just wait for havana to age out).


I think we would still want to declare Tempest versions from time to
time. I'd honestly suggest a quarterly timebox. The events that are
actually important to Tempest are less the release itself, but the eol
of branches, as that would mean features which removed completely from
any supported tree could be removed.


My current leaning is that this approach would be a good thing, and
provide a better experience for both the community and the defcore
process. However it's a big enough change that we're still collecting
data, and it would be interesting to hear other thoughts from the
community at large on this approach.

	-Sean

-- 
Sean Dague
Samsung Research America
sean at dague.net / sean.dague at samsung.com
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140404/334ad582/attachment.pgp>