Open Stack

On 2015-04-27 11:38:09 -0600 (-0600), Chris Friesen wrote:
[...]
> In circumstances like this I wonder if it might make sense to have Jenkins
> "abstain" rather than mark it -1.  We could then have a job that went around
> and re-ran Jenkins tests for cases where it "abstained" previously.
[...]

In a Jenkinsless future, this may be possible. At the moment, we're
using Jenkins and its status reporting is effectively limited to
"SUCCESS" and "FAILURE." We're going to need something which can
provide a finer-grained result (the current plans around non-Jenkins
job workers leave us open to this possibility).

Also, some of our projects actually ARE test frameworks, so for
changes proposed to those we definitely want failures of that
framework to be reported as failures of the jobs testing them. It
may be a little tough to differentiate between these cases.

Further, reviewers want to rely on job results to know whether the
change works. If the CI merely abstained from reporting when it
didn't get a chance to run the job to completion, there's no
indication of whether that job would have worked. We could try to
solve this by automatically requeuing the jobs which hit this
condition, but it leaves us open to a rather nasty problem for which
we'd need a separate release valve: if the test framework itself is
broken and not simply experiencing an intermittent problem, we'd
requeue and run all jobs using it in an infinite loop consuming all
our resources and effectively starving out worker availability for
other jobs which aren't broken.
-- 
Jeremy Stanley