Open Stack

On 11/01/18 16:36, Colleen Murphy wrote:
> Hi everyone,
> 
> We have governance review under debate[1] that we need the community's help on.
> The debate is over what recommendation the TC should make to the Interop team
> on where the tests it uses for the OpenStack trademark program should be
> located, specifically those for the new add-on program being introduced. Let me
> badly summarize:
> 
> A couple of years ago we issued a resolution[2] officially recommending that
> the Interop team use solely tempest as its source of tests for capability
> verification. The Interop team has always had the view that the developers,
> being the people closest to the project they're creating, are the best people
> to write tests verifying correct functionality, and so the Interop team doesn't
> maintain its own test suite, instead selecting tests from those written in
> coordination between the QA team and the other project teams. These tests are
> used to validate clouds applying for the OpenStack Powered tag, and since all
> of the projects included in the OpenStack Powered program already had tests in
> tempest, this was a natural fit. When we consider adding new trademark programs
> comprising of other projects, the test source is less obvious. Two examples are
> designate, which has never had tests in the tempest repo, and heat, which
> recently had its tests removed from the tempest repo.
> 

<snip/>

> 
> As I'm not deeply steeped in the history of either the Interop or QA teams I am
> sure I've misrepresented some details here, I'm sorry about that. But we'd like
> to get this resolution moving forward and we're currently stuck, so this thread
> is intended to gather enough community input to get unstuck and avoid letting
> this proposal become stale. Please respond to this thread or comment on the
> resolution proposal[1] if you have any thoughts.
> 
> Colleen
> 
> [1] https://review.openstack.org/#/c/521602
> [2] https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html
> [3] https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html
> 

I had hoped for more of a discussion on this before I jumped back into
this debate  - but it seams to be stalled still, so here it goes.

I proposed this initially as we were unclear on where the tests should
go - we had a resolution that said all tests go into openstack/tempest
(with a list of reasons why), and the guidance and discussion that been
had in various summits was that "add-ons" should stay in plugins.

So right now, we (by the governance rules) should be pushing tests to
tempest for the new programs.

In the resolution that placed the tests in tempest there was a few
reasons proposed:

  For example, API and behavioral changes must be carefully managed, as
  must mundane aspects such as test and module naming and location
  within the test suite. Even changes that leave tests functionally
  equivalent may cause unexpected consequences for their use in DefCore
  processes and validation. Placing the tests in a central repository
  will make it easier to maintain consistency and avoid breaking the
  trademark enforcement tool.

This still applies, and even more so for teams that traditionally do not
have a strong QE contributor / reviewer base (aka projects not in
"core").

  Centralizing the tests also makes it easier for anyone running the
  validation tool against their cloud or cloud distribution to use the
  tests. It is easier to install the test suite and its dependencies,
  and it is easier to read and understand a set of tests following a
  consistent implementation pattern.

Apparently users do not need central tests anymore, feedback from
RefStack is that people who run these tests are comfortable dealing
with extra python packages.

The point about a single set of tests, in a single location and style
still stands.

  Finally, having the tests in a central location makes it easier to
  ensure that all members of the community have equal input into what
  the tests do and how they are implemented and maintained.

Seems like a good value to strive for.

One of the items that has been used to push back against adding
"add-ons" to tempest has been that tempest has a defined scope, and
neither of the current add-ons fit in that scope.

Can someone clarify what the set of criteria is? I think it will help
this discussion.

Another push back is the "scaling" issue - adding more tests will
overload the QA team.

Right now, DNS adds 10 tests, Orchestration adds 22, to a current suite
of 353.

I do not think there is many other add-ons proposed yet, and the new
Vertical programs will probably mainly be re-using tests in the
openstack/tempest repos as is.

This is not a big tent-esque influx of programs - the only projects
that can be added to the trademarks are programs in tc-approved-release
[4], so I do not see scaling as a big issue, especially as these tests
are such base concepts that if they need to be changed there is a
completely new API, so the only overhead will be ensuring that nothing
in tempest breaks the new tests (which is a good thing for trademark
tests).

Personally, for me, I like option 3. I did not initially add it, as I
knew it would cause endless bikesheding, but I do think it fits both
a technical and social model.

I see 2 immediate routes forward:

 - Option 1, and we start adding these tests asap
 - Pseudo Option 2, were we delete the resolution at [2] as it clearly
   does not apply anymore, and abandon the review at [1].

Finally - do not conflate my actions with those of the Designate team.
I have seen people talking about how this resolution was leverage the
team needed to move our tests in tree. This is definitely *not* true.
Having our tests in a plugin is useful to us, and if the above
resolution passed, I cannot see a situation where we would try to
move any tests that were not listed in the interop standards.

This is something I have done as an individual in the community, not
something the designate team have pushed for.


[4] -
https://governance.openstack.org/tc/reference/tags/tc_approved-release.html

> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180118/66f884e9/attachment.sig>