Open Stack

On Wed, Aug 10, 2016 at 09:52:55AM -0700, Clay Gerrard wrote:
> On Wed, Aug 10, 2016 at 7:42 AM, Ben Swartzlander <ben at swartzlander.org>
> wrote:
> 
> >
> > A big source of problems IMO is that tempest doesn't have stable branches.
> > We use the master branch of tempest to test stable branches of other
> > projects, and tempest regularly adds new features.
> >
> 
> How come not this +1000 just fix this?

Well, mostly because it's actually not a problem and ignores the history on why
tempest is branchless. We actually used to do this pre-icehouse and it actually
made things much worse. What was happening back then was we didn't have enough
activity to keep the stable branches working at all. So we'd go very long
periods where nothing actually could land. We also often wedged ourselves where
master tempest changed to a point where we couldn't sanely backport a fix to
the stable branch. This would often mean that up until right before a stable
release things just couldn't land until someone was actually motivated to try
and dig us out. But, what more often happened was we had to just disable tempest
on the branch, because we didn't have another option. It also turns out that
having different tests across a release boundary meant we weren't actually
validating that the OpenStack APIs were consistent and worked the same. We had
many instances where a projects API just changed between release boundaries,
which violates our API consistency and backwards compatibility guidelines.
Tempest is about verifying the API and just like an other API client it should
work against any OpenStack release.

Doing this has been a huge benefit for making things actually work on the stable
branches. (in fact just thinking back about how broken everything was all the
time back then makes me appreciate it even more) We also test every incoming
tempest change on all the stable branches, and nothing can land unless it works
on all supported branches. It means we have a consistent and stable api across
releases. We do have occasional bugs where a new test or change in tempest
triggers a new race in a project's stable branch. But, that's a real bug and
normally a fix can be backported.(which is the whole point of doing stable
branches) If it can't and the race is bad enough to actively interfere with
things, we have a mechanism to skip the test. (but that's normally a last
resort) Although, these issues tend to come up pretty infrequently in practice,
especially as we slowly ramp up the stability of things over time.

FWIW, a lot of these details are covered in the original spec for implementing
this: (although it's definitely assumes a bit of prior knowledge about the
state of things going on when it was written)

http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html


-Matt Treinish
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160810/52be2dc7/attachment.pgp>