[openstack-dev] [qa] openstack health accounting problem

Andrea Frittoli andrea.frittoli at gmail.com
Wed Mar 2 22:17:21 UTC 2016

Thanks Sean for bringing this up.

It's a known pain point that we discussed back in Tokyo [0].

Failures in class level fixtures are difficult to handle consistently,
because there is no concept of success / failure at class level in the data

If a failure happens during setup, no test is actually executed. Marking
all the tests in the class as skipped would provide false data. Failures in
setup are related to the creation / deletion of resources that are required
by most test in the class, but not necessarily all of them. Because of that
marking all the tests in the class as failed may also be non accurate -
besides certain tests might have been skipped anyways, and they would be
reported as failed for no good reason.

If a failure happens during cleanup, all tests already had a result, so
what to do with the teardown failure? An option would be to avoid failing
during teardown, and only log issues in cleanup as warnings.

Before we can fix this, we need to get an agreement on a way to treat these
failures, and be are ready to accept the noise that might be added to the

Personally I would prefer to:
- fix tests so that we never fail in a tearDownClass (that should be easy
to do)
- fix the test framework so that all tests are marked as failed upon
failure in setUpClass

The downside of this approach is that it may require changes in each test
framework that reports data in OpenStack Health.


[0] https://etherpad.openstack.org/p/mitaka-qa-testr-datastore-layering

On Wed, Mar 2, 2016 at 1:19 PM Sean Dague <sean at dague.net> wrote:

> I noticed in looking at the last 10 failures in the gate view -
> http://status.openstack.org//openstack-health/#/ that 2 of them were
> tearDown in test_volume_boot_pattern.
> However, it looks like these don't count against the
> test_volume_boot_pattern success rate. Which means the real fail rate of
> the boot from volumes tests is not accounted for.
> Can we get that folded in correctly? I do realize the complexities of
> tearDownClass accounting. But it seems like that should be done here
> otherwise we don't really realize how bad our hot spots are.
>         -Sean
> --
> Sean Dague
> http://dague.net
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160302/768702c7/attachment.html>

More information about the OpenStack-dev mailing list