In gabbi, there's a way [1] to mark a test as an expected failure, which
makes it show up in your stestr run thusly:
{0}
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
[0.710821s] ... ok
======
Totals
======
Ran: 1 tests in 9.0000 sec.
- Passed: 0
- Skipped: 0
- Expected Fail: 1
- Unexpected Success: 0
- Failed: 0
If I go fix the thing causing the heretofore-expected failure, but
forget to take out the `xfail: True`, it does this:
{0}
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
[0.710517s] ... FAILED
{0}
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
[0.000000s] ... ok
==============================
Failed 1 tests - output below:
==============================
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
----------------------------------------------------------------------------------------------------------------------------------
======
Totals
======
Ran: 2 tests in 9.0000 sec.
- Passed: 1
- Skipped: 0
- Expected Fail: 0
- Unexpected Success: 1
- Failed: 0
BUT it does not cause the run to fail. For example, see the
nova-tox-functional results for [2] (specifically PS4): the test appears
twice in the middle of the run [3] and prints failure output [4] but the
job passes [5].
So I'm writing this email because I have no idea if this is expected
behavior or a bug (I'm hoping the latter, cause it's whack, yo); and if
a bug, I have no idea whose bug it should be. Help?
Thanks,
efried
[1] https://gabbi.readthedocs.io/en/latest/format.html?highlight=xfail
[2] https://review.openstack.org/#/c/579921/4
[3]
http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/job-output.txt.gz#_2018-07-09_17_22_11_846366
[4]
http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/job-output.txt.gz#_2018-07-09_17_31_07_229271
[5]
http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/testr_results.html.gz