<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Thu, Jan 11, 2018 at 4:36 PM Colleen Murphy <<a href="mailto:colleen@gazlene.net">colleen@gazlene.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi everyone,<br>
<br>
We have governance review under debate[1] that we need the community's help on.<br>
The debate is over what recommendation the TC should make to the Interop team<br>
on where the tests it uses for the OpenStack trademark program should be<br>
located, specifically those for the new add-on program being introduced. Let me<br>
badly summarize:<br>
<br>
A couple of years ago we issued a resolution[2] officially recommending that<br>
the Interop team use solely tempest as its source of tests for capability<br>
verification. The Interop team has always had the view that the developers,<br>
being the people closest to the project they're creating, are the best people<br>
to write tests verifying correct functionality, and so the Interop team doesn't<br>
maintain its own test suite, instead selecting tests from those written in<br>
coordination between the QA team and the other project teams. These tests are<br>
used to validate clouds applying for the OpenStack Powered tag, and since all<br>
of the projects included in the OpenStack Powered program already had tests in<br>
tempest, this was a natural fit. When we consider adding new trademark programs<br>
comprising of other projects, the test source is less obvious. Two examples are<br>
designate, which has never had tests in the tempest repo, and heat, which<br>
recently had its tests removed from the tempest repo.<br></blockquote><div><br></div><div><div>Thanks for the summary!</div><div><br></div><div>To be honest I don't see why this decision has to be difficult to take.</div><br><div>Nothing we decide today is written in stone and the main risk ahead of us is to take</div><div>a decision that requires a lot of upfront work and that it ends up providing no</div><div>significant benefit, or even making things worst in some aspect. So we may try one way</div><div>today and if we hit some significant issue we can still change.<br></div><div><br></div><div>TL;DR my preferred option would be number (2) - it's the least initial effort, so the</div><div>least risk, and deciding for (2) now won't make it any difficult in the future to switch</div><div>to option (1) or option (3). I'm not pushing back on (2), I just think (1) is more convenient.<br></div><div>Details below each option.<br></div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
So far the patch proposes three options:<br>
<br>
1) All trademark-related tests should go in the tempest repo, in accordance<br>
with the original resolution. This would mean that even projects that have<br>
never had tests in tempest would now have to add at least some of their<br>
black-box tests to tempest.<br>
<br></blockquote><div><br></div><div>This option is a valid one, but I think it introduces too much extra work and</div><div>testing complications for too little benefit.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The value of this option is that centralizes tests used for the Interop program<br>
in a location where interop-minded folks from the QA team can control them. </blockquote><div><br></div><div>There are other ways this can be achieved - it is possible to mark tests so that</div><div>team may require a +1 from interop/qa when specific tests are modified.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The<br>
downside is that projects that so far have avoided having a dependency on<br>
tempest will now lose some control over the black-box tests that they use for<br>
functional and integration that would now also be used for trademark<br>
certification.<br>
There's also concern for the review bandwidth of the QA team - we can't expect<br>
the QA team to be continually responsible for an ever-growing list of projects<br>
and their trademark tests.<br></blockquote><div><br></div><div>If we restrict to interop tests, the review bandwidth issue is probably not so bad.</div><div>The QA team would have to request the domain knowledge required for proper</div><div>review from the respective teams anyways.<br></div><div><br></div><div>There are other complications introduced though:</div><div><br></div><div>- service clients and other common bits (config and so) would have to move to</div><div> Tempest since we cannot have tempest depend on plugins. But then modifying</div><div> those common bits on Tempest side would risk to break non-interop tests. <br></div><div> Solution for that is to make all those bits stable interfaces for plugins<br></div><div> </div><div>- tempest would have to add new CI jobs to run the interop tests from add-on</div><div> program on every tempest change so that the new code is tested on a regular</div><div> basis. <br></div><div><br></div><div>- heat tests are wrapped in a Tempest plugin but actually written in Gabbi so we</div><div> would need to add Gabbi as a dependency to Tempest<br></div><div><br></div><div>Nothing too terrible really, but I think it might not be worth the extra effort, especially</div><div>now that teams available resources are getting thinner and thinner.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
2) All trademark-related tests for *add-on projects* should be sourced from<br>
plugins external to tempest.<br>
<br></blockquote><div>I wouldn't go as far as saying they "should" be sourced. I think saying that they</div><div>*may* be sourced from a plugin is enough. Apart from that this is my favourite</div><div>option. The only thing required really is updating the resolution and we are</div><div>ready to go.<br></div><div><br></div><div>With all the plugins no in own branchless repositories, the usability concern is</div><div>not so strong anymore really.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The value of this option is it allows project teams to retain control over<br>
these tests. </blockquote><div><br></div><div>Other value is given by simplicity, least changes to implement and low risk.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The potential problem with it is that individual project teams are<br>
not necessarily reviewing test changes with an eye for interop concerns and so<br>
could inadvertently change the behavior of the trademark-verification tools.<br>
<br>
3) All trademark-related tests should go in a single separate tempest plugin.<br></blockquote><div><br></div><div>I definitely oppose this change. It requires a lot of up-front effort for no value.</div><div>A variation may be to have an interop plugin where new interop tests go, which</div><div>would reduce the initial effort to zero, but I think the result would be a bit confusing</div><div>with interop tests being partly in Tempest, partly in project owned plugins and</div><div>partly in the new plugin.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
This has the value of giving the QA and Interop teams control over<br>
interop-related tests while also making clear the distinction between tests<br>
used for trademark verification and tests used for CI. Matt's argument against<br>
this is that there actually is very little distinction between those two cases,<br>
and that a given test could have many different applications.<br></blockquote><div><br></div><div>+1 on Matt's comment!</div><div><br></div>Also the CI and ACL for an interop plugin might be rather complicated.</div><div class="gmail_quote">The only way this might work would be if the interop team wrote their own</div><div class="gmail_quote">independent set of tests used only for interop purposes.</div><div class="gmail_quote">But a great advantage of using CI tests for interop purposes is that the interop</div><div class="gmail_quote">tests are executed all the time and they just work.<br></div><div class="gmail_quote"><br></div><div class="gmail_quote">Andrea Frittoli (andreaf)<br></div><div class="gmail_quote"><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Other ideas that have been thrown around are:<br>
<br>
* Maintaining a branch in the tempest repo that Interop tests are pulled from.<br>
<br>
* Tagging Interop-related tests with decorators to make it clear that they need<br>
to be handled carefully.<br>
<br>
At the heart of the issue is the perception that projects that keep their<br>
integration tests within the tempest tree are somehow blessed, maybe by the QA<br>
team or by the TC. It would be nice to try to clarify what technical<br>
and political<br>
reasons we have for why different projects have tests in different places -<br>
review bandwidth of the QA team, ownership/control by the project teams,<br>
technical interdependency between certain projects, or otherwise.<br>
<br>
Ultimately, as Jeremy said in the comments on the resolution patch, the<br>
recommendation should be one that works best for the QA and Interop teams. So<br>
far we've heard from Matt and Mark expressing moderate support for option 2.<br>
We'd like to hear more from those teams about how they see this working,<br>
especially with regard to concerns about the quality and stability standards<br>
that out-of-tree tests may be held to. We additionally need input from the<br>
whole community on how maintaining trademark-related tests in tempest will<br>
affect you if you don't already have your tests there. We'd especially like to<br>
address any perceptions of favoritism or exclusionism that stem from these<br>
issues.<br>
<br>
And to quickly clear up one detail before it makes it onto this thread, the<br>
Queens Community Goal about splitting tempest plugins out of the main project's<br>
tree[3] is entirely about addressing technical problems related to packaging for<br>
existing tempest plugins, it's not a decree about what should live<br>
within the tempest<br>
repository nor does it have anything to do with the Interop program.<br>
<br>
As I'm not deeply steeped in the history of either the Interop or QA teams I am<br>
sure I've misrepresented some details here, I'm sorry about that. But we'd like<br>
to get this resolution moving forward and we're currently stuck, so this thread<br>
is intended to gather enough community input to get unstuck and avoid letting<br>
this proposal become stale. Please respond to this thread or comment on the<br>
resolution proposal[1] if you have any thoughts. <br></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Colleen<br>
<br>
[1] <a href="https://review.openstack.org/#/c/521602" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/521602</a><br>
[2] <a href="https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html" rel="noreferrer" target="_blank">https://governance.openstack.org/tc/resolutions/20160504-defcore-test-location.html</a><br>
[3] <a href="https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html" rel="noreferrer" target="_blank">https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html</a><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div>