Open Stack

On 31/05/17 09:43, Doug Hellmann wrote:
> Excerpts from Chris Dent's message of 2017-05-31 11:22:50 +0100:
>> On Wed, 31 May 2017, Graham Hayes wrote:
>>> On 30/05/17 19:09, Doug Hellmann wrote:
>>>> Excerpts from Chris Dent's message of 2017-05-30 18:16:25 +0100:
>>>>> Note that this goal only applies to tempest _plugins_. Projects
>>>>> which have their tests in the core of tempest have nothing to do. I
>>>>> wonder if it wouldn't be more fair for all projects to use plugins
>>>>> for their tempest tests?
>>>>
>>>> All projects may have plugins, but all projects with tests used by
>>>> the Interop WG (formerly DefCore) for trademark certification must
>>>> place at least those tests in the tempest repo, to be managed by
>>>> the QA team [1]. As new projects are added to those trademark
>>>> programs, the tests are supposed to move to the central repo to
>>>> ensure the additional review criteria are applied properly.
>>
>> Thanks for the clarification, Doug. I don't think it changes the
>> main thrust of what I was trying to say (more below).
>>
>>>> [1] https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html
>>>
>>> In the InterOp discussions in Boston, it was indicated that some people
>>> on the QA team were not comfortable with "non core" project (even in
>>> the InterOp program) having tests in core tempest.
>>>
>>> I do think that may be a bigger discussion though.
>>
>> I'm not suggesting we change everything (because that would take a
>> lot of time and energy we probably don't have), but I had some
>> thoughts in reaction to this and sharing is caring:
>>
>> The way in which the tempest _repo_ is a combination of smoke,
>> integration, validation and trademark enforcement testing is very
>> confusing to me. If we then lay on top of that the concept of "core"
>> and "not core" with regard to who is supposed to put their tests in
>> a plugin and who isn't (except when it is trademark related!) it all
>> gets quite bewildering.
>>
>> The resolution above says: "the OpenStack community will benefit
>> from having the interoperability tests used by DefCore in a central
>> location". Findability is a good goal so this a reasonable
>> assertion, but then the directive to lump those tests in with a
>> bunch of other stuff seems off if the goal is to "easier to read and
>> understand a set of tests".
>>
>> If, instead, Tempest is a framework and all tests are in plugins
>> that each have their own repo then it is much easier to look for a
>> repo (if there is a common pattern) and know "these are the interop
>> tests for openstack" and "these are the integration tests for nova"
>> and even "these are the integration tests for the thing we are
>> currently describing as 'core'[1]".
>>
>> An area where this probably falls down is with validation. How do
>> you know which plugins to assemble in order to validate this cloud
>> you've just built? Except that we already have this problem now that
>> we are requiring most projects to manage their tempest tests as
>> plugins. Does it become worse by everything being a plugin?
>>
>> [1] We really need a better name for this.
>
> Yeah, it sounds like the current organization of the repo is not
> ideal in terms of equal playing field for all of our project teams.
> I would be fine with all of the interop tests being in a plugin
> together, or of saying that the tempest repo should only contain
> those tests and that others should move to their own plugins. If we're
> going to reorganize all of that, we should decide what new structure we
> want and work it into the goal.

+1

- ZB

> The point of centralizing review of that specific set of tests was
> to make it easier for interop folks to help ensure the tests continue
> to follow the additionally stringent review criteria that comes
> with being used as part of the trademark program. The QA team agreed
> to do that, so it's news to me that they're considering reversing
> course.  If the QA team isn't going to continue, we'll need to
> figure out what that means and potentially find another group to
> do it.
>
> Doug
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>