Open Stack

On 08/10/2016 11:33 AM, Luigi Toscano wrote:
> On Wednesday, 10 August 2016 17:00:36 CEST Ihar Hrachyshka wrote:
>> Luigi Toscano <ltoscano at redhat.com> wrote:
>>> On Wednesday, 10 August 2016 10:42:41 CEST Ben Swartzlander wrote:
>>>> On 08/10/2016 04:33 AM, Duncan Thomas wrote:
>>>>> So I tried to get into helping with the cinder stable tree for a while,
>>>>> and while I wasn't very successful (lack of time and an inability to
>>>>> convince my employer it should be a priority), one thing I did notice it
>>>>> that much of the breakage seemed to come from outside cinder - many of
>>>>> the libraries we depend on make backwards incompatible changes by
>>>>> accident, for example. Would it be possible to have a long-term-support
>>>>> branch where we pinned the max version of everything for the gate, pips
>>>>> and devtstack? I'd have thought (and I'm very willing to be corrected)
>>>>> that would make the stable gate, well, stable, such that it required far
>>>>> less work to keep it able to run a basic devstack test plus unit tests.
>>>>>
>>>>> Does that sound at all sane?
>>>>
>>>> A big source of problems IMO is that tempest doesn't have stable
>>>> branches. We use the master branch of tempest to test stable branches of
>>>> other projects, and tempest regularly adds new features. This guarantees
>>>> instability if you rely on tempest anywhere in your gate (and cinder
>>>> does).
>>>
>>> Orthogonal to the discussion, but: this is not due to the lack of stable
>>> branch, but that part of the Tempest API are not stable yet. This is being
>>> addressed right now (in scope for Newton).
>>> Once the Tempest stable API are used, no breakages should happen.
>>
>> Well, it’s only partially true. But what happens when you add a new test to
>> tempest/master? It gets executed on all branches, and maybe some of them
>> are failing it. We can argue that it’s probably a bug revealed, but it
>> nevertheless requires attention from stable maintainers to solve.
>
> The new test should work on all support branches. As tester I find a lot of
> advantages of maintaining a unified set of tests than fighting with backports
> of tests.

I'm sure it makes YOUR life easier to not have to deal with backports. 
The rest of us who do maintain stable branches though don't appreciate 
it when tempest adds a new feature or a new dependency which breaks the 
stable gate jobs. This happens a few times each release, and tends to 
get fixed within a day or two, which is barely tolerable.

The fact that it happens at all though says that we're "doing it wrong" 
w.r.t. testing of stable branches, and completely explains why we find 
it so challenging to support stuff more than 12 months old. If 
everything we used had stable branches and proper dependency management, 
then it would easy to keep gate job running for years.

> There are mechanisms to skip tests based on the cloud capabilities.
> So this should not be an issue, and if a bug is found that should definitely
> be viewed as a good thing.

It's not new tests that cause the problem, because those would be easy 
to skip. It's changes to tempest core which force changes elsewhere.

-Ben