<html><body>
<p><tt><font size="2">Sean Dague <sean@dague.net> wrote on 06/30/2014 06:03:50 AM:<br>
<br>
> From:</font></tt><br>
<tt><font size="2">> <br>
> Sean Dague <sean@dague.net></font></tt><br>
<tt><font size="2">> <br>
> To:</font></tt><br>
<tt><font size="2">> <br>
> "OpenStack Development Mailing List (not for usage questions)" <br>
> <openstack-dev@lists.openstack.org>, </font></tt><br>
<tt><font size="2">> <br>
> Date:</font></tt><br>
<tt><font size="2">> <br>
> 06/30/2014 06:09 AM</font></tt><br>
<tt><font size="2">> <br>
> Subject:</font></tt><br>
<tt><font size="2">> <br>
> Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit</font></tt><br>
<tt><font size="2">> <br>
> On 06/29/2014 09:39 AM, Joshua Hesketh wrote:<br>
> > On 6/28/14 10:40 AM, James E. Blair wrote:<br>
> >> An alternate approach would be to have third-party CI systems register<br>
> >> jobs with OpenStack's Zuul rather than using their own account. This<br>
> >> would mean only a single report of all jobs (upstream and 3rd-party)<br>
> >> per-patchset. It significantly reduces clutter and makes results more<br>
> >> accessible -- but even with one system we've never actually wanted to<br>
> >> have Jenkins results in comments, so I think one of the other options<br>
> >> would be preferred. Nonetheless, this is possible with a little bit of<br>
> >> work.<br>
> > <br>
> > I agree this isn't the preferred solution, but I disagree with the<br>
> > little bit of work. This would require CI systems registering with<br>
> > gearman which would mean security issues. The biggest problem with this<br>
> > though is that zuul would be stuck waiting from results from 3rd parties<br>
> > which often have very slow return times.<br>
> <br>
> Right, one of the other issues is the quality of the CI results varies<br>
> as well.</font></tt><br>
<br>
<tt><font size="2">Agreed. After last summit, Anita, Jay and I decided to start gathering a team</font></tt><br>
<tt><font size="2">of 3rd party testers that have the goal of improving the quality of the third</font></tt><br>
<tt><font size="2">party systems. We are starting with gathering global unwritten requirements,</font></tt><br>
<tt><font size="2">improving documentation and reaching out to new projects that will have third </font></tt><br>
<tt><font size="2">party testing needs. We are still in the early stages but now have weekly </font></tt><br>
<tt><font size="2">meetings to discuss what needs to be done and track progress.</font></tt><br>
<br>
<tt><font size="2"><a href="https://wiki.openstack.org/wiki/Meetings/ThirdParty">https://wiki.openstack.org/wiki/Meetings/ThirdParty</a></font></tt><br>
<tt><font size="2"> <br>
> I think one of the test result burn out issues right now is based on the<br>
> fact that they are too rolled up as it is. For instance, a docs only<br>
> change gets Tempest results, which humans know are irrelevant, but<br>
> Jenkins insists they aren't. I think that if we rolled up more<br>
> information, and waited longer, we'd be in a worse state.<br>
</font></tt><br>
<tt><font size="2">Maybe it could promptly timeout and then report the system that did not </font></tt><br>
<tt><font size="2">complete? That would also have the benefit of enforcing a time limit on</font></tt><br>
<tt><font size="2">reporting results.</font></tt><br>
<br>
<font size="2" face="sans-serif"><br>
Kurt Taylor (krtaylor)<br>
OpenStack Development Lead - PowerKVM CI<br>
IBM Linux Technology Center<br>
</font><br>
</body></html>