Open Stack

Hi!

On 01/21/2013 08:42 AM, Daniel P. Berrange wrote:
> In Folsom if you ran Nova's unit tests you'd get an informative display of
> what was being run
> 
> $ ./run_tests.sh test_libvirt_vif
> 
> LibvirtVifTestCase
>     test_bridge_driver                                           OK    0.98
>     test_model_kvm                                               OK    0.02
>     test_model_novirtio                                          OK    0.02
>     test_model_qemu                                              OK    0.02
>     test_model_xen                                               OK    0.03
>     test_ovs_ethernet_driver                                     OK    0.07
>     test_ovs_virtualport_driver                                  OK    0.02
>     test_quantum_bridge_ethernet_driver                          OK    0.02
>     test_quantum_hybrid_driver                                   OK    0.02
> 
> Slowest 3 tests took 1.08 secs:
>     0.98    LibvirtVifTestCase.test_bridge_driver
>     0.07    LibvirtVifTestCase.test_ovs_ethernet_driver
>     0.03    LibvirtVifTestCase.test_model_xen
> ----------------------------------------------------------------------
> Ran 9 tests in 1.227s
> 
> OK
> 
> 
> In Grizzly with the re-written run_tests.sh script we now get this:

So, it actually has nothing to do with run_tests.sh, it's related to the
move from nosetests to testr. But I'm picking nits. :)

> $ ./run_tests.sh test_libvirt_vif
> Running `tools/with_venv.sh testr run --parallel   test_libvirt_vif`
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --list 
> /home/berrange/src/cloud/nova/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:8: UserWarning: Module argparse was already imported from /usr/lib64/python2.7/argparse.pyc, but /home/berrange/src/cloud/nova/.venv/lib/python2.7/site-packages is being added to sys.path
>   import pkg_resources
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests  --load-list /tmp/tmpOYyP07
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests  --load-list /tmp/tmpTywG0B
> /home/berrange/src/cloud/nova/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:8: UserWarning: Module argparse was already imported from /usr/lib64/python2.7/argparse.pyc, but /home/berrange/src/cloud/nova/.venv/lib/python2.7/site-packages is being added to sys.path
>   import pkg_resources
> /home/berrange/src/cloud/nova/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:8: UserWarning: Module argparse was already imported from /usr/lib64/python2.7/argparse.pyc, but /home/berrange/src/cloud/nova/.venv/lib/python2.7/site-packages is being added to sys.path
>   import pkg_resources
> 2013-01-21 13:38:28.100 18125 WARNING nova.virt.libvirt.firewall [-] Libvirt module could not be loaded. NWFilterFirewall will not work correctly.
> 2013-01-21 13:38:28.100 18126 WARNING nova.virt.libvirt.firewall [-] Libvirt module could not be loaded. NWFilterFirewall will not work correctly.

(Quck note -the 4 lines above are things I'd like to figure out how to
properly suppress)

> Ran 10 tests in 0.512s
> PASSED (id=0)

SO ...

This is largely because the tests run in parallel now, which itself
leads to a test-suite speedup. Let me call out two lines in your pastes
real quick:

> Ran 9 tests in 1.227s
> Ran 10 tests in 0.512s

When we were looking at getting this going, getting faster overall test
runs seemed like a good tradeoff to printing a bunch of colors to the
screen. HOWEVER - the output from the test runner is in subunit format,
so there is literally nothing preventing the writing of a stream
processor similar to the old nose plugin which could do a similar thing.
It's just been lower priority.

> IMHO, this is a horrible step backwards in terms of develper friendliness.

I can understand your point of view. However, let me point out a couple
of things real quick.

a) run_tests.sh is missing the "slowest X tests" output. Sorry, we
missed that, I think because we all use tox directly. Clark is working
on adding that back real quick
b) testr (the test runner that replaced nose) has a WHOLE SLEW of
developer friendly features that were gained as part of this. For instance:

testr run --failing

will re-run any tests that failed in the previous run

testr failing

will print out a list of failing tests

That's just two off the top of my head. It's a pretty cool system - I
totally recommend checking it out!

> Can we get back the useful display we had before ?

Totally. It's on the todo list. In fact, we just chatted with Sean Dague
in the infra channel about how it could be hacked on and I think he's
going to take a peek.

Thanks for helping us spot the missing slow tests, and also for
jumpstarting a discussion of how to provide that feature for folks!

Monty