Open Stack

On Tue, Jun 18, 2013 at 9:49 AM, Sean Dague <sean at dague.net> wrote:
> On 06/18/2013 12:43 PM, Martina Kollarova wrote:
>>
>> Jenkins keeps running all the tests, even if the basic pep8 test fails,
>> and runs all of the (very slow) Tempest Quantum tests, even though
>> almost all of them are failing.
>>
>> I propose that it should fail and stop all of the other tests once there
>> is a failure in a voting test. For non-voting tests, it should stop only
>> itself, not the others.
>>
>> This would decrease the feedback loop and we wouldn't have to wait for
>> the non-voting Quantum tests to see that they failed as always.
>>
>> The nosetest tool has a parameter '--stop' which will cause it to exit
>> on first failure.
>>
We tried something similar where we required all projects to pass pep8
before we ran the other tests (not as flexible as what you describe
above but it does shorten the feedback loop when there are
syntax/format issues). The problem we ran into there is the one
Jaypipes describes where you fix one thing and now there is another
thing to fix then another and so on. It ends up being advantageous to
get as much information as possible rather than as little as possible.

I do think Jim Blair would like to add more generic support for what
you describe in Zuul, but that hasn't happened yet. I will let him
elaborate.
>
> You can always got out to zuul directly and see patch status in progress -
> http://status.openstack.org/zuul/
This should be helpful. If you want to be proactive you can see
failures here before they are reported back to Gerrit.

Finally, please run `tox` locally before you push commits to Gerrit.
This will catch pep8/pyflakes/flake8 problems (as configured by the
project) and run the unit tests which gives you control of that
feedback loop before Jenkins tests the change. I think these
suggestions come out of tempest development where running `tox` isn't
going to be as lightweight and quick, but in general it should help
quite a bit.

Clark