[openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure

Matthew Treinish mtreinish at kortar.org
Tue Jul 10 19:16:14 UTC 2018


On Tue, Jul 10, 2018 at 10:16:37AM +0100, Chris Dent wrote:
> On Mon, 9 Jul 2018, Matthew Treinish wrote:
> 
> > It's definitely  a bug, and likely a bug in stestr (or one of the lower level
> > packages like testtools or python-subunit), because that's what's generating
> > the return code. Tox just looks at the return code from the commands to figure
> > out if things were successful or not. I'm a bit surprised by this though I
> > thought we covered the unxsuccess and xfail cases because I would have expected
> > cdent to file a bug if it didn't. Looking at the stestr tests we don't have
> > coverage for the unxsuccess case so I can see how this slipped through.
> 
> This was reported on testrepository some years ago and a bit of
> analysis was done: https://bugs.launchpad.net/testrepository/+bug/1429196
> 

This actually helps a lot, because I was seeing the same issue when I tried
writing a quick patch to address this. When I manually poked the TestResult
object it didn't have anything in the unxsuccess list. So instead of relying
on that I wrote this patch:

https://github.com/mtreinish/stestr/pull/188

which uses the output filter's internal function for counting results to
find unxsuccess tests. It's still not perfect though because if someone
runs with the --no-subunit-trace flag it still doesn't work (because that
call path never gets run) but it's at least a starting point. I've
marked it as WIP for now, but I'm thinking we could merge it as is and
leave the --no-subunit-trace and unxsuccess as a known issues for now,
since xfail and unxsuccess are pretty uncommon in practice. (gabbi is the
only thing I've seen really use it)

-Matt Treinish


> So yeah, I did file a bug but it fell off the radar during those
> dark times.
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180710/0dc81fdd/attachment.sig>


More information about the OpenStack-dev mailing list