Open Stack

On 04/08/2014 03:13 AM, victor stinner wrote:
> Hi,
>
> I have some issues when running unit tests in OpenStack. I would like to help, but I don't know where I should start and how I can fix these bugs. My use case is to run unit tests and rerun a single test if one or more tests failed. Well, it should be the most basic use case, no?
>
>
> (1) First problem: if a Python module cannot be loaded, my terminal is flooded with a binary stream which looks like:
>
> ... tCase.test_deprecated_without_replacement\xd7\xe1\x06\xa1\xb3)\x01 at l@atests.unit.test_versionutils.DeprecatedTestCa ...
>
> IMO it's a huge bug in the testr tool: "testr run" command should not write binary data into stdout. It makes development very hard.
>
>
> (2) When a test fails, it's hard to find the command to rerun a single failing test.
>
> Using the tool trial, I can just copy/paste the "FQDN" name of the failing test and run "trial FQDN". Example:
>
>     trial tests.unit.test_timeutils.TestIso8601Time.test_west_normalize
>
> Using the tool nosetests, you have to add a column between the module and the method. Example:
>
>     nosetests tests.unit.test_timeutils:TestIso8601Time.test_west_normalize
>
> Using tox, in most OpenStack projects, adding the name of the failing test to the tox command is usually ignored. I guess that it's an issue with tox.ini of the project? tox rerun the whole test suite which is usually very slow (it takes some minutes even on fast computer). Example:
>
>     tox -e py27 tests.unit.test_timeutils.TestIso8601Time.test_west_normalize

The way to do this that works in every project where I've tried it is to 
add a -- before the name of the test.  That way it will get passed 
straight to testr.  So your command above would become:

tox -e py27 -- tests.unit.test_timeutils.TestIso8601Time.test_west_normalize

(hopefully that isn't going to wrap)

Note that in a some projects (tempest and oslo-incubator) there are tox 
targets that already use testr regexes, so in those targets this syntax 
won't work because you can't specify two regexes, at least as far as I 
can tell.  Both of those projects have an "all" tox target that doesn't 
filter using a regex so you can pass the test name as above and it 
should work.  For oslo-incubator that should be going away once the rpc 
code is removed because rpc tests are the only reason it's filtering 
tests in the py27 target today.

>
> I try sometimes to activate the virtualenv and then type:
>
>     testr run tests.unit.test_timeutils.TestIso8601Time.test_west_normalize
>
> It usually fails for different reasons.
>
> Example with python-swiftclient. I run unit tests using "tox -e py33. Some tests are failing. I enter the virtual environment and type the following command to rerun a failing test:
>
>     testr run tests.test_swiftclient.TestPutObject.test_unicode_ok
>
> The test is not run again, no test is run. It's surprising because the same command works with Python 2. It's probably a bug in testr?
>
>
>
> (3) testscenarios doesn't work with nosetests. It's annoying because for the reasons listed above, I prefer to run tests using nosetests. Why do we use testscenarios and not something else? Do we plan to support nosetests (and other Python test runners) for testscenarios?

It should be possible to switch back to testr for the py33 tests if we 
use the regex syntax for filtering out the tests that don't work in py3. 
  The rpc example is 
https://github.com/openstack/oslo-incubator/blob/master/tox.ini#L21 and 
if you look at what tests2skip.py does in the tripleo tempest element it 
should be an example of how to filter out failing tests: 
https://github.com/openstack/tripleo-image-elements/blob/master/elements/tempest/bin/run-tempest#L110

>
>
> Victor
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>