Open Stack

We're at a 18hr backup in the gate, which is really unusual given the
amount of decoupling. Even under our current load that means we're
seeing huge failure rates causing resets.

It appears one of the major culprits is the python34 tests in neutron,
which were over a 40% failure rate recently - http://goo.gl/9wCerK

That tends to lead to things like -
http://dl.dropbox.com/u/6514884/screenshot_249.png - which means a huge
amount of work has been reset. Right now 3 of 7 neutron patches in the
gate that are within the sliding window are in a failure state (they are
also the only current visible fails in the window).

Looking at one of the patches in question -
https://review.openstack.org/#/c/202207/ - shows it's been rechecked 3
times, and these failures were seen in earlier runs.

I do understand that people want to get their code merged, but
rechecking patches that are failing this much without going after the
root causes means everyone pays for it. This is blocking a lot of other
projects from landing code in a timely manner.

The functional tests seem to have a quite high failure rate as well from
spot checking. If the results of these tests are mostly going to be
ignored and rechecked, can we remove them from the gate definition on
neutron so they aren't damaging the overall flow of the gate?

Thanks,

	-Sean

-- 
Sean Dague
http://dague.net