Open Stack

On 08/11/2016 04:13 PM, Ben Swartzlander wrote:
> On 08/10/2016 01:57 PM, Matthew Treinish wrote:
>> On Wed, Aug 10, 2016 at 09:52:55AM -0700, Clay Gerrard wrote:
>>> On Wed, Aug 10, 2016 at 7:42 AM, Ben Swartzlander <ben at swartzlander.org>
>>> wrote:
>>>
>>>>
>>>> A big source of problems IMO is that tempest doesn't have stable branches.
>>>> We use the master branch of tempest to test stable branches of other
>>>> projects, and tempest regularly adds new features.
>>>>
>>>
>>> How come not this +1000 just fix this?
>>
>> Well, mostly because it's actually not a problem and ignores the history on why
>> tempest is branchless. We actually used to do this pre-icehouse and it actually
>> made things much worse. What was happening back then was we didn't have enough
>> activity to keep the stable branches working at all. So we'd go very long
>> periods where nothing actually could land. We also often wedged ourselves where
>> master tempest changed to a point where we couldn't sanely backport a fix to
>> the stable branch. This would often mean that up until right before a stable
>> release things just couldn't land until someone was actually motivated to try
>> and dig us out. But, what more often happened was we had to just disable tempest
>> on the branch, because we didn't have another option. It also turns out that
>> having different tests across a release boundary meant we weren't actually
>> validating that the OpenStack APIs were consistent and worked the same. We had
>> many instances where a projects API just changed between release boundaries,
>> which violates our API consistency and backwards compatibility guidelines.
>> Tempest is about verifying the API and just like an other API client it should
>> work against any OpenStack release.
>>
>> Doing this has been a huge benefit for making things actually work on the stable
>> branches. (in fact just thinking back about how broken everything was all the
>> time back then makes me appreciate it even more) We also test every incoming
>> tempest change on all the stable branches, and nothing can land unless it works
>> on all supported branches. It means we have a consistent and stable api across
>> releases. We do have occasional bugs where a new test or change in tempest
>> triggers a new race in a project's stable branch. But, that's a real bug and
>> normally a fix can be backported.(which is the whole point of doing stable
>> branches) If it can't and the race is bad enough to actively interfere with
>> things, we have a mechanism to skip the test. (but that's normally a last
>> resort) Although, these issues tend to come up pretty infrequently in practice,
>> especially as we slowly ramp up the stability of things over time.
>>
>> FWIW, a lot of these details are covered in the original spec for implementing
>> this: (although it's definitely assumes a bit of prior knowledge about the
>> state of things going on when it was written)
>>
>> http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html
>>
>
> I still don't agree with this stance. Code doesn't just magically stop working.
> Code breaks when things change which aren't version controlled properly or when
> you have undeclared dependencies.

Or testcases breaks when you change the timing of your tests, which exposes 
latent bugs that were always there but by fluke weren't being hit.

In this case the code was previously broken, we just weren't hitting it in the 
testcases.

Chris