Open Stack

On Tue, Jul 10, 2018 at 03:16:14PM -0400, Matthew Treinish wrote:
> On Tue, Jul 10, 2018 at 10:16:37AM +0100, Chris Dent wrote:
> > On Mon, 9 Jul 2018, Matthew Treinish wrote:
> > 
> > > It's definitely  a bug, and likely a bug in stestr (or one of the lower level
> > > packages like testtools or python-subunit), because that's what's generating
> > > the return code. Tox just looks at the return code from the commands to figure
> > > out if things were successful or not. I'm a bit surprised by this though I
> > > thought we covered the unxsuccess and xfail cases because I would have expected
> > > cdent to file a bug if it didn't. Looking at the stestr tests we don't have
> > > coverage for the unxsuccess case so I can see how this slipped through.
> > 
> > This was reported on testrepository some years ago and a bit of
> > analysis was done: https://bugs.launchpad.net/testrepository/+bug/1429196
> > 
> 
> This actually helps a lot, because I was seeing the same issue when I tried
> writing a quick patch to address this. When I manually poked the TestResult
> object it didn't have anything in the unxsuccess list. So instead of relying
> on that I wrote this patch:
> 
> https://github.com/mtreinish/stestr/pull/188
> 
> which uses the output filter's internal function for counting results to
> find unxsuccess tests. It's still not perfect though because if someone
> runs with the --no-subunit-trace flag it still doesn't work (because that
> call path never gets run) but it's at least a starting point. I've
> marked it as WIP for now, but I'm thinking we could merge it as is and
> leave the --no-subunit-trace and unxsuccess as a known issues for now,
> since xfail and unxsuccess are pretty uncommon in practice. (gabbi is the
> only thing I've seen really use it)
> 
> 
> 
> > So yeah, I did file a bug but it fell off the radar during those
> > dark times.
> > 
> 

Just following up here, after digging some more and getting a detailed
bug filed by electrofelix [1] I was able to throw together a different patch
that should solve this in a better way:

https://github.com/mtreinish/stestr/pull/190

Once that lands I can push a bugfix release to get it out there so people
can actually use the fix.

-Matt Treinish
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180803/7aac9618/attachment.sig>