Open Stack

Sean Dague <sean at dague.net> wrote on 06/30/2014 06:03:50 AM:

> From:
>
> Sean Dague <sean at dague.net>
>
> To:
>
> "OpenStack Development Mailing List (not for usage questions)"
> <openstack-dev at lists.openstack.org>,
>
> Date:
>
> 06/30/2014 06:09 AM
>
> Subject:
>
> Re: [openstack-dev] [all] 3rd Party CI vs. Gerrit
>
> On 06/29/2014 09:39 AM, Joshua Hesketh wrote:
> > On 6/28/14 10:40 AM, James E. Blair wrote:
> >> An alternate approach would be to have third-party CI systems register
> >> jobs with OpenStack's Zuul rather than using their own account.  This
> >> would mean only a single report of all jobs (upstream and 3rd-party)
> >> per-patchset.  It significantly reduces clutter and makes results more
> >> accessible -- but even with one system we've never actually wanted to
> >> have Jenkins results in comments, so I think one of the other options
> >> would be preferred.  Nonetheless, this is possible with a little bit
of
> >> work.
> >
> > I agree this isn't the preferred solution, but I disagree with the
> > little bit of work. This would require CI systems registering with
> > gearman which would mean security issues. The biggest problem with this
> > though is that zuul would be stuck waiting from results from 3rd
parties
> > which often have very slow return times.
>
> Right, one of the other issues is the quality of the CI results varies
> as well.

Agreed. After last summit, Anita, Jay and I decided to start gathering a
team
of 3rd party testers that have the goal of improving the quality of the
third
party systems. We are starting with gathering global unwritten
requirements,
improving documentation and reaching out to new projects that will have
third
party testing needs. We are still in the early stages but now have weekly
meetings to discuss what needs to be done and track progress.

https://wiki.openstack.org/wiki/Meetings/ThirdParty

> I think one of the test result burn out issues right now is based on the
> fact that they are too rolled up as it is. For instance, a docs only
> change gets Tempest results, which humans know are irrelevant, but
> Jenkins insists they aren't. I think that if we rolled up more
> information, and waited longer, we'd be in a worse state.

Maybe it could promptly timeout and then report the system that did not
complete? That would also have the benefit of enforcing a time limit on
reporting results.


Kurt Taylor (krtaylor)
OpenStack Development Lead - PowerKVM CI
IBM Linux Technology Center
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140630/501369ea/attachment.html>