Open Stack

On Fri, Jul 01, 2016 at 02:35:34PM +0000, Jeremy Stanley wrote:
> On 2016-07-01 15:39:10 +0200 (+0200), Kashyap Chamarthy wrote:
> > [Snip description of some nice debugging.]
> > 
> > > I'd really love it if there was
> > > 
> > >  1. the ability to request checking of just specific jobs eg
> > > 
> > >       "recheck gate-tempest-dsvm-multinode-full"
> > 
> > Yes, this would really be desirable.  I recall once asking this exact
> > question on #openstack-infra, but can't find Infra team's response to
> > that.
> 
> The challenge here is that you want to make sure it can't be used to
> recheck individual jobs until you have them all passing (like
> picking a pin and tumbler lock). The temptation to recheck-spam
> nondeterministically failing changes is already present, but this
> would make it considerably easier still for people to introduce new
> nondeterministic failures in projects. Maybe if it were tied to a
> special pipeline type, and then we set it only for the experimental
> pipeline or something?

If we don't want it to interfere with "normal" testing, then
perhaps just don't hook it under 'recheck'. Have a competely
separate command ('run-job blah') to trigger that has no influence
on the normal check status applied to a changeset, and reports it
separately too.

> > >  2. the ability to request this recheck to run multiple
> > >     times in parallel. eg if i just repeat the 'recheck'
> > >     command many times on the same patchset # without
> > >     waiting for results
> > 
> > Yes, this too, would be _very_ useful for all the reasons you described.
> [...]
> 
> In the past we've discussed the option of having an "idle pipeline"
> which repeatedly runs specified jobs only when there are unused
> resources available, so that it doesn't significantly cut into our
> resource pool when we're under high demand but still allows to
> automatically collect a large amount of statistical data.

Yep, that could work as long as the 'idle pipeline' did have some
kind of minimal throughput. Debugging some of these things can
be time critical, so we don't neccessarily want to wait for a
fully idle time period. IOW a 'mostly-idle pipeline' which would
run jobs at any time, but rate limit them to prevent it swamping
out the normal jobs.

> Anyway, hopefully James Blair can weigh in on this, since Zuul is
> basically in a feature freeze for a while to limit the number of
> significant changes we'll need to forward-port into the v3 branch.
> We'd want to discuss these new features in the context of Zuul v3
> instead.

Sure, that's no problem - I got lucky and reproduced the problem
this time around after a few rechecks. I just wanted to raise this
as a general request, since we've hit this scenario several times
in the past, so it'd be useful to have a more general solution in
the future, whenever that's practical.


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|