Open Stack

Excerpts from Chris Dent's message of 2017-05-31 11:22:50 +0100:
> On Wed, 31 May 2017, Graham Hayes wrote:
> > On 30/05/17 19:09, Doug Hellmann wrote:
> >> Excerpts from Chris Dent's message of 2017-05-30 18:16:25 +0100:
> >>> Note that this goal only applies to tempest _plugins_. Projects
> >>> which have their tests in the core of tempest have nothing to do. I
> >>> wonder if it wouldn't be more fair for all projects to use plugins
> >>> for their tempest tests?
> >>
> >> All projects may have plugins, but all projects with tests used by
> >> the Interop WG (formerly DefCore) for trademark certification must
> >> place at least those tests in the tempest repo, to be managed by
> >> the QA team [1]. As new projects are added to those trademark
> >> programs, the tests are supposed to move to the central repo to
> >> ensure the additional review criteria are applied properly.
> 
> Thanks for the clarification, Doug. I don't think it changes the
> main thrust of what I was trying to say (more below).
> 
> >> [1] https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html
> >
> > In the InterOp discussions in Boston, it was indicated that some people
> > on the QA team were not comfortable with "non core" project (even in
> > the InterOp program) having tests in core tempest.
> >
> > I do think that may be a bigger discussion though.
> 
> I'm not suggesting we change everything (because that would take a
> lot of time and energy we probably don't have), but I had some
> thoughts in reaction to this and sharing is caring:
> 
> The way in which the tempest _repo_ is a combination of smoke,
> integration, validation and trademark enforcement testing is very
> confusing to me. If we then lay on top of that the concept of "core"
> and "not core" with regard to who is supposed to put their tests in
> a plugin and who isn't (except when it is trademark related!) it all
> gets quite bewildering.
> 
> The resolution above says: "the OpenStack community will benefit
> from having the interoperability tests used by DefCore in a central
> location". Findability is a good goal so this a reasonable
> assertion, but then the directive to lump those tests in with a
> bunch of other stuff seems off if the goal is to "easier to read and
> understand a set of tests".
> 
> If, instead, Tempest is a framework and all tests are in plugins
> that each have their own repo then it is much easier to look for a
> repo (if there is a common pattern) and know "these are the interop
> tests for openstack" and "these are the integration tests for nova"
> and even "these are the integration tests for the thing we are
> currently describing as 'core'[1]".
> 
> An area where this probably falls down is with validation. How do
> you know which plugins to assemble in order to validate this cloud
> you've just built? Except that we already have this problem now that
> we are requiring most projects to manage their tempest tests as
> plugins. Does it become worse by everything being a plugin?
> 
> [1] We really need a better name for this.

Yeah, it sounds like the current organization of the repo is not
ideal in terms of equal playing field for all of our project teams.
I would be fine with all of the interop tests being in a plugin
together, or of saying that the tempest repo should only contain
those tests and that others should move to their own plugins. If we're
going to reorganize all of that, we should decide what new structure we
want and work it into the goal.

The point of centralizing review of that specific set of tests was
to make it easier for interop folks to help ensure the tests continue
to follow the additionally stringent review criteria that comes
with being used as part of the trademark program. The QA team agreed
to do that, so it's news to me that they're considering reversing
course.  If the QA team isn't going to continue, we'll need to
figure out what that means and potentially find another group to
do it.

Doug