Open Stack

On Thu, Jan 18, 2018 at 3:33 PM Graham Hayes <gr at ham.ie> wrote:

>
>
> On 11/01/18 16:36, Colleen Murphy wrote:
> > Hi everyone,
> >
> > We have governance review under debate[1] that we need the community's
> help on.
> > The debate is over what recommendation the TC should make to the Interop
> team
> > on where the tests it uses for the OpenStack trademark program should be
> > located, specifically those for the new add-on program being introduced.
> Let me
> > badly summarize:
> >
> > A couple of years ago we issued a resolution[2] officially recommending
> that
> > the Interop team use solely tempest as its source of tests for capability
> > verification. The Interop team has always had the view that the
> developers,
> > being the people closest to the project they're creating, are the best
> people
> > to write tests verifying correct functionality, and so the Interop team
> doesn't
> > maintain its own test suite, instead selecting tests from those written
> in
> > coordination between the QA team and the other project teams. These
> tests are
> > used to validate clouds applying for the OpenStack Powered tag, and
> since all
> > of the projects included in the OpenStack Powered program already had
> tests in
> > tempest, this was a natural fit. When we consider adding new trademark
> programs
> > comprising of other projects, the test source is less obvious. Two
> examples are
> > designate, which has never had tests in the tempest repo, and heat, which
> > recently had its tests removed from the tempest repo.
> >
>
> <snip/>
>
> >
> > As I'm not deeply steeped in the history of either the Interop or QA
> teams I am
> > sure I've misrepresented some details here, I'm sorry about that. But
> we'd like
> > to get this resolution moving forward and we're currently stuck, so this
> thread
> > is intended to gather enough community input to get unstuck and avoid
> letting
> > this proposal become stale. Please respond to this thread or comment on
> the
> > resolution proposal[1] if you have any thoughts.
> >
> > Colleen
> >
> > [1] https://review.openstack.org/#/c/521602
> > [2]
> https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html
> > [3]
> https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html
> >
>
> I had hoped for more of a discussion on this before I jumped back into
> this debate  - but it seams to be stalled still, so here it goes.
>
> I proposed this initially as we were unclear on where the tests should
> go - we had a resolution that said all tests go into openstack/tempest
> (with a list of reasons why), and the guidance and discussion that been
> had in various summits was that "add-ons" should stay in plugins.
>
> So right now, we (by the governance rules) should be pushing tests to
> tempest for the new programs.
>
> In the resolution that placed the tests in tempest there was a few
> reasons proposed:
>
>   For example, API and behavioral changes must be carefully managed, as
>   must mundane aspects such as test and module naming and location
>   within the test suite. Even changes that leave tests functionally
>   equivalent may cause unexpected consequences for their use in DefCore
>   processes and validation. Placing the tests in a central repository
>   will make it easier to maintain consistency and avoid breaking the
>   trademark enforcement tool.
>
> This still applies, and even more so for teams that traditionally do not
> have a strong QE contributor / reviewer base (aka projects not in
> "core").
>
>   Centralizing the tests also makes it easier for anyone running the
>   validation tool against their cloud or cloud distribution to use the
>   tests. It is easier to install the test suite and its dependencies,
>   and it is easier to read and understand a set of tests following a
>   consistent implementation pattern.
>
> Apparently users do not need central tests anymore, feedback from
> RefStack is that people who run these tests are comfortable dealing
> with extra python packages.
>
> The point about a single set of tests, in a single location and style
> still stands.
>
>   Finally, having the tests in a central location makes it easier to
>   ensure that all members of the community have equal input into what
>   the tests do and how they are implemented and maintained.
>
> Seems like a good value to strive for.
>
> One of the items that has been used to push back against adding
> "add-ons" to tempest has been that tempest has a defined scope, and
> neither of the current add-ons fit in that scope.
>
> Can someone clarify what the set of criteria is? I think it will help
> this discussion.
>
> Another push back is the "scaling" issue - adding more tests will
> overload the QA team.
>
> Right now, DNS adds 10 tests, Orchestration adds 22, to a current suite
> of 353.
>
> I do not think there is many other add-ons proposed yet, and the new
> Vertical programs will probably mainly be re-using tests in the
> openstack/tempest repos as is.
>
> This is not a big tent-esque influx of programs - the only projects
> that can be added to the trademarks are programs in tc-approved-release
> [4], so I do not see scaling as a big issue, especially as these tests
> are such base concepts that if they need to be changed there is a
> completely new API, so the only overhead will be ensuring that nothing
> in tempest breaks the new tests (which is a good thing for trademark
> tests).
>
> Personally, for me, I like option 3. I did not initially add it, as I
> knew it would cause endless bikesheding, but I do think it fits both
> a technical and social model.
>
> I see 2 immediate routes forward:
>
>  - Option 1, and we start adding these tests asap
>  - Pseudo Option 2, were we delete the resolution at [2] as it clearly
>    does not apply anymore, and abandon the review at [1].
>

I think conditions changed a bit since that resolution was written.
The decision should be based on finding the best compromise
between upfront investment, ongoing maintenance cost and usability.
If the solution does not match with the resolution we adapt it to match
the current situation.

andreaf


> Finally - do not conflate my actions with those of the Designate team.
> I have seen people talking about how this resolution was leverage the
> team needed to move our tests in tree. This is definitely *not* true.
> Having our tests in a plugin is useful to us, and if the above
> resolution passed, I cannot see a situation where we would try to
> move any tests that were not listed in the interop standards.
>
> This is something I have done as an individual in the community, not
> something the designate team have pushed for.
>
>
> [4] -
> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html
>
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180119/775251b1/attachment.html>