Open Stack

On Mon, Jul 09, 2018 at 06:59:42PM -0500, Eric Fried wrote:
> In gabbi, there's a way [1] to mark a test as an expected failure, which
> makes it show up in your stestr run thusly:
> 
> {0}
> nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
> [0.710821s] ... ok
> 
> ======
> Totals
> ======
> Ran: 1 tests in 9.0000 sec.
>  - Passed: 0
>  - Skipped: 0
>  - Expected Fail: 1
>  - Unexpected Success: 0
>  - Failed: 0
> 
> If I go fix the thing causing the heretofore-expected failure, but
> forget to take out the `xfail: True`, it does this:
> 
> {0}
> nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
> [0.710517s] ... FAILED
> {0}
> nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
> [0.000000s] ... ok
> 
> ==============================
> Failed 1 tests - output below:
> ==============================
> 
> nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request
> ----------------------------------------------------------------------------------------------------------------------------------
> 
> 
> ======
> Totals
> ======
> Ran: 2 tests in 9.0000 sec.
>  - Passed: 1
>  - Skipped: 0
>  - Expected Fail: 0
>  - Unexpected Success: 1
>  - Failed: 0
> 
> BUT it does not cause the run to fail. For example, see the
> nova-tox-functional results for [2] (specifically PS4): the test appears
> twice in the middle of the run [3] and prints failure output [4] but the
> job passes [5].
> 
> So I'm writing this email because I have no idea if this is expected
> behavior or a bug (I'm hoping the latter, cause it's whack, yo); and if
> a bug, I have no idea whose bug it should be. Help?
It's definitely  a bug, and likely a bug in stestr (or one of the lower level
packages like testtools or python-subunit), because that's what's generating
the return code. Tox just looks at the return code from the commands to figure
out if things were successful or not. I'm a bit surprised by this though I
thought we covered the unxsuccess and xfail cases because I would have expected
cdent to file a bug if it didn't. Looking at the stestr tests we don't have
coverage for the unxsuccess case so I can see how this slipped through.

Looking at the where the return code for the output from the run command is
generated (it's a bit weird because run calls the load command internally which
handles the output generation, result storage, and final return code):

https://github.com/mtreinish/stestr/blob/master/stestr/commands/load.py#L222-L225

I'm thinking it might be an issue in testtools or python-subunit, I don't remember
which generates the results object used there (if it is subunit it'll be a
subclass from testtools). But I'll have to trace through it to be sure. In the
mean time we can easily workaround the issue in stestr itself by just manually
checking the result status instead of relying on the existing function from the
results class.

-Matt Treinish

> 
> [1] https://gabbi.readthedocs.io/en/latest/format.html?highlight=xfail
> [2] https://review.openstack.org/#/c/579921/4
> [3]
> http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/job-output.txt.gz#_2018-07-09_17_22_11_846366
> [4]
> http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/job-output.txt.gz#_2018-07-09_17_31_07_229271
> [5]
> http://logs.openstack.org/21/579921/4/check/nova-tox-functional/5fb6ee9/testr_results.html.gz
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180709/8275f5bb/attachment.sig>