[openstack-dev] Gate / Check jobs fail fast mode - don't start tempest until docs/pep8/unittests pass?

Clark Boylan clark.boylan at gmail.com
Fri Dec 6 00:38:17 UTC 2013


On Thu, Dec 5, 2013 at 3:33 PM, Christopher Yeoh <cbkyeoh at gmail.com> wrote:
> On Fri, Dec 6, 2013 at 9:32 AM, Clark Boylan <clark.boylan at gmail.com> wrote:
>>
>> On Thu, Dec 5, 2013 at 2:37 PM, Peter Portante
>> <peter.a.portante at gmail.com> wrote:
>> > Has anybody considered changing how check and gate jobs work such that
>> > the tempest and grenade checks only run once the docs/pep8/unittests
>> > jobs all succeed?
>> >
>> > It seems like they complete so much quicker that folks can fix those
>> > bugs before having to wait hours for the other jobs.
>> >
>> > Thoughts, concerns, problems with that?
>> >
>> We actually did this for a while where we ran pep8 first and after
>> doing it found that the additional 5-10 minutes that were added to
>> passing runs was more painful than either waiting for results or
>> checking the Zuul status page. There is the additional problem that we
>> occasionally run out of slaves to run specific jobs while being able
>> to run others. If we run out of slaves capable of running pep8 jobs we
>> sit and wait even longer (though there is work being done to make
>> those slaves scale out much better).
>>
>
> Is there any way to start them all off in parallel and then abort the long
> running tempest
> ones early if the short running pep8 or unittests fail?
>
I don't think Zuul is capable of doing that today. Even if we could do
that it wouldn't really save any resources. The tempest tests are run
on single use slaves that are thrown away after being used. We would
just throw them away and need to build new slaves in the current and
proposed scenarios. The other drawback to this approach is that you
cancel the tempest jobs after throwing 6 slaves away because pep8
failed. After pep8 is fixed you find that a tempest test is broken by
the change. You have now burned through 12 slaves instead of 6 to get
this info. I think it is best to provide reviewers (and change
proposers) with as much information upfront as possible.

Clark



More information about the OpenStack-dev mailing list