Open Stack

On Fri, Jan 22, 2016 at 06:46:48PM -0800, melanie witt wrote:
> Hi everyone,
> 
> I noticed because of the way we run the py34 tests in tox.ini, I'm not able to specify a filter regex the way I normally do as a positional arg, for example: 'tox -epy34 nova.tests.unit.network' doesn't filter and it runs everything. ('tox -epy27 nova.tests.unit.network' will only run tests that match nova.tests.unit.network).
> 
> I couldn't figure out how we could add something to tox.ini to make it work -- we're calling ostestr with the --blacklist_file option. I'm not completely clear on what 'ostestr --blacklist_file <file> --regex <regex>' does but I couldn't get it to do what I want. From the documentation [1], it adds --regex to the regex created from the --blacklist_file. The regex from the blacklist file looks something like this '^((?!blacklistedstuff).)*$' and if I can only append to it, the best I could do was a positive lookbehind but that can't match at the beginning of a line. (For example, I tried "ostestr --blacklist_file tests-py3.txt --regex '(?<=network)'" and it matched all the non-blacklisted tests that ended with the word "network"). It seems like what I would need is for --regex to do another re.search() and match the line only if the previous regex from the blacklist also matched.

So os-testr literally just generates a regex [2] and passes that to testr
directly when it's called with subprocess. [3] os-testr still relies on testr
to do the actual test selection.

This does sound like a real bug in os-testr's regex generation, can you file
a bug here:

https://bugs.launchpad.net/os-testr

Although, now that I look at the list it definitely looks like this one,
which I had forgotten about:

https://bugs.launchpad.net/os-testr/+bug/1506215

It sounds like that bug proposes a direction so if you could patch os-testr
to give that a try and see if that fixes your issue. If it does then we can
just land that in os-testr.

However, I'm wondering if it makes more sense to change how the selection
actually works with os-testr. This regex generation approach does get pretty
unwieldy and seems very error prone. I'm thinking it might make more sense to
generate a test-list and then have os-testr filter that list itself and then
pass the pruned list to --load-list in testr run.

Thanks,

Matt Treinish


> [1] http://docs.openstack.org/developer/os-testr/ostestr.html#test-selection
[2] https://github.com/openstack/os-testr/blob/master/os_testr/os_testr.py#L145-L174
[3] https://github.com/openstack/os-testr/blob/master/os_testr/os_testr.py#L191
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160122/a4d3a62d/attachment.pgp>