Open Stack

On Thu, Jan 11, 2018 at 4:36 PM Colleen Murphy <colleen at gazlene.net> wrote:

> Hi everyone,
>
> We have governance review under debate[1] that we need the community's
> help on.
> The debate is over what recommendation the TC should make to the Interop
> team
> on where the tests it uses for the OpenStack trademark program should be
> located, specifically those for the new add-on program being introduced.
> Let me
> badly summarize:
>
> A couple of years ago we issued a resolution[2] officially recommending
> that
> the Interop team use solely tempest as its source of tests for capability
> verification. The Interop team has always had the view that the developers,
> being the people closest to the project they're creating, are the best
> people
> to write tests verifying correct functionality, and so the Interop team
> doesn't
> maintain its own test suite, instead selecting tests from those written in
> coordination between the QA team and the other project teams. These tests
> are
> used to validate clouds applying for the OpenStack Powered tag, and since
> all
> of the projects included in the OpenStack Powered program already had
> tests in
> tempest, this was a natural fit. When we consider adding new trademark
> programs
> comprising of other projects, the test source is less obvious. Two
> examples are
> designate, which has never had tests in the tempest repo, and heat, which
> recently had its tests removed from the tempest repo.
>

Thanks for the summary!

To be honest I don't see why this decision has to be difficult to take.

Nothing we decide today is written in stone and the main risk ahead of us
is to take
a decision that requires a lot of upfront work and that it ends up
providing no
significant benefit, or even making things worst in some aspect. So we may
try one way
today and if we hit some significant issue we can still change.

TL;DR my preferred option would be number (2) - it's the least initial
effort, so the
least risk, and deciding for (2) now won't make it any difficult in the
future to switch
to option (1) or option (3). I'm not pushing back on (2), I just think (1)
is more convenient.
Details below each option.


>
> So far the patch proposes three options:
>
> 1) All trademark-related tests should go in the tempest repo, in accordance
>    with the original resolution. This would mean that even projects that
> have
>    never had tests in tempest would now have to add at least some of their
>    black-box tests to tempest.
>
>
This option is a valid one, but I think it introduces too much extra work
and
testing complications for too little benefit.


> The value of this option is that centralizes tests used for the Interop
> program
> in a location where interop-minded folks from the QA team can control
> them.


There are other ways this can be achieved - it is possible to mark tests so
that
team may require a +1 from interop/qa when specific tests are modified.


> The
> downside is that projects that so far have avoided having a dependency on
> tempest will now lose some control over the black-box tests that they use
> for
> functional and integration that would now also be used for trademark
> certification.
> There's also concern for the review bandwidth of the QA team - we can't
> expect
> the QA team to be continually responsible for an ever-growing list of
> projects
> and their trademark tests.
>

If we restrict to interop tests, the review bandwidth issue is probably not
so bad.
The QA team would have to request the domain knowledge required for proper
review from the respective teams anyways.

There are other complications introduced though:

- service clients and other common bits (config and so) would have to move
to
  Tempest since we cannot have tempest depend on plugins. But then modifying
  those common bits on Tempest side would risk to break non-interop tests.
  Solution for that is to make all those bits stable interfaces for plugins

- tempest would have to add new CI jobs to run the interop tests from add-on
  program on every tempest change so that the new code is tested on a
regular
  basis.

- heat tests are wrapped in a Tempest plugin but actually written in Gabbi
so we
  would need to add Gabbi as a dependency to Tempest

Nothing too terrible really, but I think it might not be worth the extra
effort, especially
now that teams available resources are getting thinner and thinner.


> 2) All trademark-related tests for *add-on projects* should be sourced from
>    plugins external to tempest.
>
> I wouldn't go as far as saying they "should" be sourced. I think saying
that they
*may* be sourced from a plugin is enough. Apart from that this is my
favourite
option. The only thing required really is updating the resolution and we are
ready to go.

With all the plugins no in own branchless repositories, the usability
concern is
not so strong anymore really.


> The value of this option is it allows project teams to retain control over
> these tests.


Other value is given by simplicity, least changes to implement and low risk.


> The potential problem with it is that individual project teams are
> not necessarily reviewing test changes with an eye for interop concerns
> and so
> could inadvertently change the behavior of the trademark-verification
> tools.
>
> 3) All trademark-related tests should go in a single separate tempest
> plugin.
>

I definitely oppose this change. It requires a lot of up-front effort for
no value.
A variation may be to have an interop plugin where new interop tests go,
which
would reduce the initial effort to zero, but I think the result would be a
bit confusing
with interop tests being partly in Tempest, partly in project owned plugins
and
partly in the new plugin.


>
> This has the value of giving the QA and Interop teams control over
> interop-related tests while also making clear the distinction between tests
> used for trademark verification and tests used for CI. Matt's argument
> against
> this is that there actually is very little distinction between those two
> cases,
> and that a given test could have many different applications.
>

+1 on Matt's comment!

Also the CI and ACL for an interop plugin might be rather complicated.
The only way this might work would be if the interop team wrote their own
independent set of tests used only for interop purposes.
But a great advantage of using CI tests for interop purposes is that the
interop
tests are executed all the time and they just work.

Andrea Frittoli (andreaf)


>
> Other ideas that have been thrown around are:
>
> * Maintaining a branch in the tempest repo that Interop tests are pulled
> from.
>
> * Tagging Interop-related tests with decorators to make it clear that they
> need
>   to be handled carefully.
>
> At the heart of the issue is the perception that projects that keep their
> integration tests within the tempest tree are somehow blessed, maybe by
> the QA
> team or by the TC. It would be nice to try to clarify what technical
> and political
> reasons we have for why different projects have tests in different places -
> review bandwidth of the QA team, ownership/control by the project teams,
> technical interdependency between certain projects, or otherwise.
>
> Ultimately, as Jeremy said in the comments on the resolution patch, the
> recommendation should be one that works best for the QA and Interop teams.
> So
> far we've heard from Matt and Mark expressing moderate support for option
> 2.
> We'd like to hear more from those teams about how they see this working,
> especially with regard to concerns about the quality and stability
> standards
> that out-of-tree tests may be held to. We additionally need input from the
> whole community on how maintaining trademark-related tests in tempest will
> affect you if you don't already have your tests there. We'd especially
> like to
> address any perceptions of favoritism or exclusionism that stem from these
> issues.
>
> And to quickly clear up one detail before it makes it onto this thread, the
> Queens Community Goal about splitting tempest plugins out of the main
> project's
> tree[3] is entirely about addressing technical problems related to
> packaging for
> existing tempest plugins, it's not a decree about what should live
> within the tempest
> repository nor does it have anything to do with the Interop program.
>
> As I'm not deeply steeped in the history of either the Interop or QA teams
> I am
> sure I've misrepresented some details here, I'm sorry about that. But we'd
> like
> to get this resolution moving forward and we're currently stuck, so this
> thread
> is intended to gather enough community input to get unstuck and avoid
> letting
> this proposal become stale. Please respond to this thread or comment on the
> resolution proposal[1] if you have any thoughts.
>

> Colleen
>
> [1] https://review.openstack.org/#/c/521602
> [2]
> https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html
> [3]
> https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180119/9cb55f41/attachment.html>