Open Stack

With the advent of gabbi tests in both ceilometer and gnocchi, we've
started using xfail (expected failure) as a way of highlighting HTTP
behavior that is wrong or poor[1] and linking to bugs on launchpad in
the description of the tests.

This means that we need to start monitoring local test runs for
unexpected success to see which of these have been fixed, and
update the tests accordingly. testr itself will not alert you.

Gnocchi has already started using pretty_tox.sh, so as long as there
is a recent version of subunit-trace (from tempest-lib) installed,
gnocchi test runs will tell when there has been an unexpected
success.

Ceilometer hasn't made that change, but it is possible to do it by
hand. After any test run:

    testr last --subunit |subunit-trace

will report on the most recent test run and give a summary. Again,
this assume a recent tempest-lib has been installed (from PyPI).

[1] Often but not always related to the framework being used.
Usually, but not always, WSME failing to trap exceptions in the face
of unexpected data.

-- 
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent