<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Mon, Jan 22, 2018 at 3:05 PM Graham Hayes <<a href="mailto:gr@ham.ie">gr@ham.ie</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
On 19/01/18 16:27, Andrea Frittoli wrote:<br>
><br>
> Thanks for the summary!<br>
><br>
> To be honest I don't see why this decision has to be difficult to take.<br>
<br>
I think the confusion comes from one thing being decided already, and<br>
now conflicting direction is being given to teams, without anyone<br>
updating the Governance repo.<br>
<br>
(It is not as if there was not plenty of warning, I have been raising it<br>
as an issue for over a year)<br>
<br>
> Nothing we decide today is written in stone and the main risk ahead of<br>
> us is to take<br>
> a decision that requires a lot of upfront work and that it ends up<br>
> providing no<br>
> significant benefit, or even making things worst in some aspect. So we<br>
> may try one way<br>
> today and if we hit some significant issue we can still change.<br>
><br>
> TL;DR my preferred option would be number (2) - it's the least initial<br>
> effort, so the<br>
> least risk, and deciding for (2) now won't make it any difficult in the<br>
> future to switch<br>
> to option (1) or option (3). I'm not pushing back on (2), I just think<br>
> (1) is more convenient.<br>
> Details below each option.<br>
> <br>
><br>
><br>
> So far the patch proposes three options:<br>
><br>
> 1) All trademark-related tests should go in the tempest repo, in<br>
> accordance<br>
> with the original resolution. This would mean that even projects<br>
> that have<br>
> never had tests in tempest would now have to add at least some of<br>
> their<br>
> black-box tests to tempest.<br>
><br>
><br>
> This option is a valid one, but I think it introduces too much extra<br>
> work and<br>
> testing complications for too little benefit.<br>
<br>
What it does do is *guarantee* that the InterOp suite will work, as it<br>
will be CI'd. I see these programs as important enough that we should CI<br>
the tooling used for them, but I seem to be in a minority.<br></blockquote><div><br></div><div>Add-ons intreoperability tests will be CI'd for every change in Tempest as long</div><div>as they are executed in a job that runs on every change in Tempest.</div><div><br></div><div>This can be achieve regardless of the location of the tests and having the tests</div><div>in the Tempest tree is not by itself a guarantee that they will be executed against</div><div>every change.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> <br>
><br>
> The value of this option is that centralizes tests used for the<br>
> Interop program<br>
> in a location where interop-minded folks from the QA team can<br>
> control them.<br>
><br>
><br>
> There are other ways this can be achieved - it is possible to mark tests<br>
> so that<br>
> team may require a +1 from interop/qa when specific tests are modified.<br>
<br>
Is there? AFAIK gerrit does not do per file path permissions, so unless<br>
we have a job that just checks the votes on a patch, and passes or fails<br>
if a test changes (which would be awful for the teams) we cannot do<br>
that.<br></blockquote><div><br></div><div>If we really want to enforce having a vote from someone it may be tricky,</div><div>yes, but I don't think enforcement is what we need, rather awareness,</div><div>For governance and project-config patches reviewed always ask for a +1</div><div>from the project PTL where relevant, and add-on project reviewers could</div><div>do the same. </div><div><br></div><div>To help building awareness we could have automation in place to post a</div><div>comment to Gerrit, like we do for elastic recheck. We could do that</div><div>on every change to the plugin in the beginning and include a link to the</div><div>interoperability recommendation to help reviewers in their job.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> <br>
><br>
> The<br>
> downside is that projects that so far have avoided having a<br>
> dependency on<br>
> tempest will now lose some control over the black-box tests that<br>
> they use for<br>
> functional and integration that would now also be used for trademark<br>
> certification.<br>
> There's also concern for the review bandwidth of the QA team - we<br>
> can't expect<br>
> the QA team to be continually responsible for an ever-growing list<br>
> of projects<br>
> and their trademark tests.<br>
><br>
><br>
> If we restrict to interop tests, the review bandwidth issue is probably<br>
> not so bad.<br>
> The QA team would have to request the domain knowledge required for proper<br>
> review from the respective teams anyways.<br>
><br>
> There are other complications introduced though:<br>
><br>
> - service clients and other common bits (config and so) would have to<br>
> move to<br>
> Tempest since we cannot have tempest depend on plugins. But then modifying<br>
> those common bits on Tempest side would risk to break non-interop tests.<br>
> Solution for that is to make all those bits stable interfaces for plugins<br>
<br>
Is this not already the case? e.g. the neutron plugin uses the nova<br>
service client already.<br></blockquote><div> </div><div>What I was saying is that Tempest cannot depend on a Tempest plugin,</div><div>so service clients should be in Tempest. Which is absolutely fine, I'm was</div><div>just listing out the work that needs to be done if we move add-on</div><div>interoperability tests into Tempest.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
This would also help for the neutron plugin which is currently importing<br>
DNS service clients from the designate-tempest-plugin repo - having them<br>
in the tempest repo would allow them to to be more stable, and remove<br>
the extra dependency.<br>
<br>
> <br>
> - tempest would have to add new CI jobs to run the interop tests from add-on<br>
> program on every tempest change so that the new code is tested on a<br>
> regular<br>
> basis.<br>
<br>
That is a good thing, and we should probably do that for the other 2<br>
options as well...<br>
<br>
><br>
> - heat tests are wrapped in a Tempest plugin but actually written in<br>
> Gabbi so we<br>
> would need to add Gabbi as a dependency to Tempest<br>
><br>
> Nothing too terrible really, but I think it might not be worth the extra<br>
> effort, especially<br>
> now that teams available resources are getting thinner and thinner.<br>
><br>
><br>
> 2) All trademark-related tests for *add-on projects* should be<br>
> sourced from<br>
> plugins external to tempest.<br>
><br>
> I wouldn't go as far as saying they "should" be sourced. I think saying<br>
> that they<br>
> *may* be sourced from a plugin is enough. Apart from that this is my<br>
> favourite<br>
> option. The only thing required really is updating the resolution and we are<br>
> ready to go.<br>
><br>
> With all the plugins no in own branchless repositories, the usability<br>
> concern is<br>
> not so strong anymore really.<br>
> <br>
><br>
> The value of this option is it allows project teams to retain<br>
> control over<br>
> these tests.<br>
><br>
><br>
> Other value is given by simplicity, least changes to implement and low risk.<br>
<br>
I do think if we do this for add-ons, we should be doing it for other<br>
programs as well, so that resolution will just be deleted,and it will<br>
allow other teams to have the same control, and further reduce the<br>
review load for QA.<br></blockquote><div><br></div><div>If you mean to move interoperability tests only, this would create an</div><div>artificial split in tests and it would not have a real impact on the review</div><div>load of the QA team.</div><div><br></div><div>If you mean moving all tests, that would take a while since we have</div><div>a number of stable interfaces (including config) that we cannot just</div><div>remove. Besides tests are an integral part of Tempest I would not</div><div>move them out.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> <br>
><br>
> The potential problem with it is that individual project teams are<br>
> not necessarily reviewing test changes with an eye for interop<br>
> concerns and so<br>
> could inadvertently change the behavior of the<br>
> trademark-verification tools.<br>
<br>
This is a big issue, and I think it is overlooked.<br>
<br>
<snip/><br>
<br>
So, thus far, we have had 3 responses from people working on the QA<br>
tooling, with one for Option 1, one for Option 2, and one for Option 1<br>
if Heat + Designate are now part of "core".<br>
<br>
If I start migrating tooling to tempest from designate, will it be -2'd?<br></blockquote><div> </div><div>I still don't understand why you'd want to invest much of your and of the</div><div>QA team time in moving things around now - I think it would be too much</div><div>of trying to solve a problem before it exists.</div><div><br></div><div>But if the majority agrees this is the best course of action I won't oppose it.</div><div>I would start with an etherpad or spec or something with a checklist of things<br></div><div>to be done and expected target state in terms of what's where and what's</div><div>tested where.</div><div><br></div><div>The plan of what needs to be tested where has to be done regardless of</div><div>the location of the tests.</div><div><br></div><div>Andrea Frittoli (andreaf)</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div>