[openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure

Eric Fried openstack at fried.cc
Mon Jul 9 23:59:42 UTC 2018


In gabbi, there's a way [1] to mark a test as an expected failure, which
makes it show up in your stestr run thusly:

{0}
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
[0.710821s] ... ok

======
Totals
======
Ran: 1 tests in 9.0000 sec.
 - Passed: 0
 - Skipped: 0
 - Expected Fail: 1
 - Unexpected Success: 0
 - Failed: 0

If I go fix the thing causing the heretofore-expected failure, but
forget to take out the `xfail: True`, it does this:

{0}
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
[0.710517s] ... FAILED
{0}
nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
[0.000000s] ... ok

==============================
Failed 1 tests - output below:
==============================

nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
----------------------------------------------------------------------------------------------------------------------------------


======
Totals
======
Ran: 2 tests in 9.0000 sec.
 - Passed: 1
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 1
 - Failed: 0

BUT it does not cause the run to fail. For example, see the
nova-tox-functional results for [2] (specifically PS4): the test appears
twice in the middle of the run [3] and prints failure output [4] but the
job passes [5].

So I'm writing this email because I have no idea if this is expected
behavior or a bug (I'm hoping the latter, cause it's whack, yo); and if
a bug, I have no idea whose bug it should be. Help?

Thanks,
efried

[1] https://gabbi.readthedocs.io/en/latest/format.html?highlight=xfail
[2] https://review.openstack.org/#/c/579921/4
[3]
http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/job-output.txt.gz#_2018-07-09_17_22_11_846366
[4]
http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/job-output.txt.gz#_2018-07-09_17_31_07_229271
[5]
http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/testr_results.html.gz



More information about the OpenStack-dev mailing list