Open Stack

On 08/10/2016 01:57 PM, Matthew Treinish wrote:
> On Wed, Aug 10, 2016 at 09:52:55AM -0700, Clay Gerrard wrote:
>> On Wed, Aug 10, 2016 at 7:42 AM, Ben Swartzlander <ben at swartzlander.org>
>> wrote:
>>
>>>
>>> A big source of problems IMO is that tempest doesn't have stable branches.
>>> We use the master branch of tempest to test stable branches of other
>>> projects, and tempest regularly adds new features.
>>>
>>
>> How come not this +1000 just fix this?
>
> Well, mostly because it's actually not a problem and ignores the history on why
> tempest is branchless. We actually used to do this pre-icehouse and it actually
> made things much worse. What was happening back then was we didn't have enough
> activity to keep the stable branches working at all. So we'd go very long
> periods where nothing actually could land. We also often wedged ourselves where
> master tempest changed to a point where we couldn't sanely backport a fix to
> the stable branch. This would often mean that up until right before a stable
> release things just couldn't land until someone was actually motivated to try
> and dig us out. But, what more often happened was we had to just disable tempest
> on the branch, because we didn't have another option. It also turns out that
> having different tests across a release boundary meant we weren't actually
> validating that the OpenStack APIs were consistent and worked the same. We had
> many instances where a projects API just changed between release boundaries,
> which violates our API consistency and backwards compatibility guidelines.
> Tempest is about verifying the API and just like an other API client it should
> work against any OpenStack release.
>
> Doing this has been a huge benefit for making things actually work on the stable
> branches. (in fact just thinking back about how broken everything was all the
> time back then makes me appreciate it even more) We also test every incoming
> tempest change on all the stable branches, and nothing can land unless it works
> on all supported branches. It means we have a consistent and stable api across
> releases. We do have occasional bugs where a new test or change in tempest
> triggers a new race in a project's stable branch. But, that's a real bug and
> normally a fix can be backported.(which is the whole point of doing stable
> branches) If it can't and the race is bad enough to actively interfere with
> things, we have a mechanism to skip the test. (but that's normally a last
> resort) Although, these issues tend to come up pretty infrequently in practice,
> especially as we slowly ramp up the stability of things over time.
>
> FWIW, a lot of these details are covered in the original spec for implementing
> this: (although it's definitely assumes a bit of prior knowledge about the
> state of things going on when it was written)
>
> http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html

I still don't agree with this stance. Code doesn't just magically stop 
working. Code breaks when things change which aren't version controlled 
properly or when you have undeclared dependencies.

Yes, managing dependencies and doing version control is a lot of work. I 
understand that the tempest team is strapped for resources. However 
simply declaring that the solution is to stop doing version control is 
an epic failure. I read the spec above and it sounds like cry for help 
rather than a well thought out idea.

Personally I would be interested in helping improve this situation, but 
I think the proper way to improve it is to actually do version control 
of everything that matters and use dependency management. If you do this 
correctly, then maintaining the old stuff stops being a chore because it 
never breaks. By definition, if a stable branch breaks without a change 
going into it, you failed at doing dependency management.

I'm not sure how I even could help with the current situation. It feels 
like we've dug ourselves into a hole and the plan of record is to get 
used to living underground. Does anyone have a will to make thing better 
and get to a place where stable branches could reasonably expected to 
remain stable for months or years without constant fixing?

-Ben Swartzlander




> -Matt Treinish
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>