Open Stack

On Mon, Jan 21, 2013 at 06:54:09PM -0500, Monty Taylor wrote:
> Hi!
> 
> On 01/21/2013 08:42 AM, Daniel P. Berrange wrote:
> > In Folsom if you ran Nova's unit tests you'd get an informative display of
> > what was being run
> > 
> > $ ./run_tests.sh test_libvirt_vif
> > 
> > LibvirtVifTestCase
> >     test_bridge_driver                                           OK    0.98
> >     test_model_kvm                                               OK    0.02
> >     test_model_novirtio                                          OK    0.02
> >     test_model_qemu                                              OK    0.02
> >     test_model_xen                                               OK    0.03
> >     test_ovs_ethernet_driver                                     OK    0.07
> >     test_ovs_virtualport_driver                                  OK    0.02
> >     test_quantum_bridge_ethernet_driver                          OK    0.02
> >     test_quantum_hybrid_driver                                   OK    0.02
> > 
> > Slowest 3 tests took 1.08 secs:
> >     0.98    LibvirtVifTestCase.test_bridge_driver
> >     0.07    LibvirtVifTestCase.test_ovs_ethernet_driver
> >     0.03    LibvirtVifTestCase.test_model_xen
> > ----------------------------------------------------------------------
> > Ran 9 tests in 1.227s
> > 
> > OK
> > 
> > 
> > In Grizzly with the re-written run_tests.sh script we now get this:
> 
> So, it actually has nothing to do with run_tests.sh, it's related to the
> move from nosetests to testr. But I'm picking nits. :)

Yep, understood :-)

> (Quck note -the 4 lines above are things I'd like to figure out how to
> properly suppress)
> 
> > Ran 10 tests in 0.512s
> > PASSED (id=0)
> 
> SO ...
> 
> This is largely because the tests run in parallel now, which itself
> leads to a test-suite speedup. Let me call out two lines in your pastes
> real quick:
> 
> > Ran 9 tests in 1.227s
> > Ran 10 tests in 0.512s
> 
> When we were looking at getting this going, getting faster overall test
> runs seemed like a good tradeoff to printing a bunch of colors to the
> screen. HOWEVER - the output from the test runner is in subunit format,
> so there is literally nothing preventing the writing of a stream
> processor similar to the old nose plugin which could do a similar thing.
> It's just been lower priority.

I appreciate that from the CI pov, getting the performance improvement
was the number 1 priority. As I'm developing patches, I will only ever
run individual tests so I don't care that much about the running time,
there - 1.2s vs 0.5s is pretty much just lost in the noise. So I'd be
fine if a mode switch to print out verbose progress alslo required the
serialization of their execution.

Mostly I just like to see that the test suite is actually doing something.
One of the things I find particularly troubling is that when you are
running a specific test, if you typo in the test name, it doesn't lead
to any particularly obvious change in display. eg if I ran 

  'run_tests test_libvurt'

the only bit of output that changes is the line 'Ran 0 tests' - at a
glance this is quite hard to spot. With it displaying the list of each
test case that is run, then it is very easy to spot that you typo'd
and it didn't run anything.

If running *every* test in Nova, then listing every single test name
is probably overkill. If it could just print out a "% complete" number
every second, or a nice progress bar '[============       ]' that
would give nice indication that it was actually doing something and
how long I might have left to wait.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|