Open Stack

On Mon, Mar 24, 2014 at 09:56:09PM +0100, Salvatore Orlando wrote:
> Thanks a lot!
> 
> We now need to get on these bugs, and define with QA an acceptable failure
> rate criterion for switching the full job to voting.
> It would be good to have a chance to only run the tests against code which
> is already in master.
> To this aim we might push a dummy patch, and keep it spinning in the check
> queue.

Honestly, there isn't really a number. I had a thread trying to get consensus on
that back when I first made tempest run in parallel. What I ended up doing back
then and what we've done since for this kind of change is to just pick a slower
week for the gate and just green light it, of course after checking to make sure
if it blows up we're not blocking anything critical. If it looks like it's
passing at roughly the same rate as everything else and you guys think it's
ready. 25% is definitely too high, for comparison when I looked at a couple of
min. ago at the numbers for the past 4 days on the equivalent job with
nova-network it only failed 4% of the time. (12 out of 300) But that number does
fluctuate quite a bit for example looking at the past week the number grows to
11.6%. (171 out of 1480) 

Doing it this way doesn't seem like the best, but until it's gating things
really don't get the attention they deserve and more bugs will just slip in
while you wait. There will most likely be initial pain after it merges, but it's
the only real way to lock it down and make forward progress.

-Matt Treinish

> 
> 
> On 24 March 2014 21:45, Rossella Sblendido <rsblendido at suse.com> wrote:
> 
> > Hello all,
> >
> > here is an update regarding the Neutron full parallel job.
> > I used the following Logstash query [1]  that checks the failures of the
> > last
> > 4 days (the last bug fix related with the full job was merged 4 days ago).
> > These are the results:
> >
> > 123 failure (25% of the total)
> >
> > I took a sample of 50 failures and I obtained the following:
> >
> > 22% legitimate failures (they are due to the code change introduced by the
> > patch)
> > 22% infra issues
> > 12% https://bugs.launchpad.net/openstack-ci/+bug/1291611
> > 12% https://bugs.launchpad.net/tempest/+bug/1281969
> > 8% https://bugs.launchpad.net/tempest/+bug/1294603
> > 3% https://bugs.launchpad.net/neutron/+bug/1283522
> > 3% https://bugs.launchpad.net/neutron/+bug/1291920
> > 3% https://bugs.launchpad.net/nova/+bug/1290642
> > 3% https://bugs.launchpad.net/tempest/+bug/1252971
> > 3% https://bugs.launchpad.net/horizon/+bug/1257885
> > 3% https://bugs.launchpad.net/tempest/+bug/1292242
> > 3% https://bugs.launchpad.net/neutron/+bug/1277439
> > 3% https://bugs.launchpad.net/neutron/+bug/1283599
> >
> > cheers,
> >
> > Rossella
> >
> > [1] http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9uYW1lOi
> > BcImNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWZ1bGxcIiBBTkQgbWVzc2
> > FnZTpcIkZpbmlzaGVkOiBGQUlMVVJFXCIgQU5EIHRhZ3M6Y29uc29sZSIsIm
> > ZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiY3VzdG9tIiwiZ3
> > JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7ImZyb20iOiIyMDE0LTAzLTIwVD
> > EzOjU0OjI1KzAwOjAwIiwidG8iOiIyMDE0LTAzLTI0VDEzOjU0OjI1KzAwOj
> > AwIiwidXNlcl9pbnRlcnZhbCI6IjAifSwibW9kZSI6IiIsImFuYWx5emVfZm
> > llbGQiOiIiLCJzdGFtcCI6MTM5NTY3MDY2ODc0OX0=
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev