Open Stack

On Wed, 31 May 2017, Graham Hayes wrote:
> On 30/05/17 19:09, Doug Hellmann wrote:
>> Excerpts from Chris Dent's message of 2017-05-30 18:16:25 +0100:
>>> Note that this goal only applies to tempest _plugins_. Projects
>>> which have their tests in the core of tempest have nothing to do. I
>>> wonder if it wouldn't be more fair for all projects to use plugins
>>> for their tempest tests?
>>
>> All projects may have plugins, but all projects with tests used by
>> the Interop WG (formerly DefCore) for trademark certification must
>> place at least those tests in the tempest repo, to be managed by
>> the QA team [1]. As new projects are added to those trademark
>> programs, the tests are supposed to move to the central repo to
>> ensure the additional review criteria are applied properly.

Thanks for the clarification, Doug. I don't think it changes the
main thrust of what I was trying to say (more below).

>> [1] https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html
>
> In the InterOp discussions in Boston, it was indicated that some people
> on the QA team were not comfortable with "non core" project (even in
> the InterOp program) having tests in core tempest.
>
> I do think that may be a bigger discussion though.

I'm not suggesting we change everything (because that would take a
lot of time and energy we probably don't have), but I had some
thoughts in reaction to this and sharing is caring:

The way in which the tempest _repo_ is a combination of smoke,
integration, validation and trademark enforcement testing is very
confusing to me. If we then lay on top of that the concept of "core"
and "not core" with regard to who is supposed to put their tests in
a plugin and who isn't (except when it is trademark related!) it all
gets quite bewildering.

The resolution above says: "the OpenStack community will benefit
from having the interoperability tests used by DefCore in a central
location". Findability is a good goal so this a reasonable
assertion, but then the directive to lump those tests in with a
bunch of other stuff seems off if the goal is to "easier to read and
understand a set of tests".

If, instead, Tempest is a framework and all tests are in plugins
that each have their own repo then it is much easier to look for a
repo (if there is a common pattern) and know "these are the interop
tests for openstack" and "these are the integration tests for nova"
and even "these are the integration tests for the thing we are
currently describing as 'core'[1]".

An area where this probably falls down is with validation. How do
you know which plugins to assemble in order to validate this cloud
you've just built? Except that we already have this problem now that
we are requiring most projects to manage their tempest tests as
plugins. Does it become worse by everything being a plugin?

[1] We really need a better name for this.
-- 
Chris Dent                  ┬──┬◡ﾉ(° -°ﾉ)       https://anticdent.org/
freenode: cdent                                         tw: @anticdent