Open Stack

On 31/05/17 11:22, Chris Dent wrote:
> On Wed, 31 May 2017, Graham Hayes wrote:
>> On 30/05/17 19:09, Doug Hellmann wrote:
>>> Excerpts from Chris Dent's message of 2017-05-30 18:16:25 +0100:
>>>> Note that this goal only applies to tempest _plugins_. Projects
>>>> which have their tests in the core of tempest have nothing to do. I
>>>> wonder if it wouldn't be more fair for all projects to use plugins
>>>> for their tempest tests?
>>>
>>> All projects may have plugins, but all projects with tests used by
>>> the Interop WG (formerly DefCore) for trademark certification must
>>> place at least those tests in the tempest repo, to be managed by
>>> the QA team [1]. As new projects are added to those trademark
>>> programs, the tests are supposed to move to the central repo to
>>> ensure the additional review criteria are applied properly.
> 
> Thanks for the clarification, Doug. I don't think it changes the
> main thrust of what I was trying to say (more below).
> 
>>> [1]
>>> https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html
>>>
>>
>> In the InterOp discussions in Boston, it was indicated that some people
>> on the QA team were not comfortable with "non core" project (even in
>> the InterOp program) having tests in core tempest.
>>
>> I do think that may be a bigger discussion though.
> 
> I'm not suggesting we change everything (because that would take a
> lot of time and energy we probably don't have), but I had some
> thoughts in reaction to this and sharing is caring:
> 
> The way in which the tempest _repo_ is a combination of smoke,
> integration, validation and trademark enforcement testing is very
> confusing to me. If we then lay on top of that the concept of "core"
> and "not core" with regard to who is supposed to put their tests in
> a plugin and who isn't (except when it is trademark related!) it all
> gets quite bewildering.
> 
> The resolution above says: "the OpenStack community will benefit
> from having the interoperability tests used by DefCore in a central
> location". Findability is a good goal so this a reasonable
> assertion, but then the directive to lump those tests in with a
> bunch of other stuff seems off if the goal is to "easier to read and
> understand a set of tests".
> 
> If, instead, Tempest is a framework and all tests are in plugins
> that each have their own repo then it is much easier to look for a
> repo (if there is a common pattern) and know "these are the interop
> tests for openstack" and "these are the integration tests for nova"
> and even "these are the integration tests for the thing we are
> currently describing as 'core'[1]".
> 
> An area where this probably falls down is with validation. How do
> you know which plugins to assemble in order to validate this cloud
> you've just built? Except that we already have this problem now that
> we are requiring most projects to manage their tempest tests as
> plugins. Does it become worse by everything being a plugin?

No - this was the gist of my point last year when I proposed the
plugins first (or plugins for all as I called it at the time).

It actually gets better - for a few reasons.

1. We can have a interop-tempest-plugin (under QA control) where all
   interop tests can go, and be tagged for each standard.
   This is good for many reasons, mainly that the tests are curated,
   and projects cannot change tests without someone from QA-core team
   checking it to make sure it does not break backwards compatibility.

2. With the some interop tests being out of the tempest tree, there is
   a very real possibility of a change in tempest blocking someone
   getting certification - tempest does not gate against plugins
   and has broken interfaces in the past.

3. With the new interop programs, the old definition of "core" is more
   murky - and it is looking more and more like the criteria for
   inclusion in openstack/tempest is "be there from the beginning".

> [1] We really need a better name for this.
100% - but I have tried and failed to find a better descriptor, that
everyone understands.

- Graham
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>