Open Stack

On 26 January 2014 02:25, Sean Dague <sean at dague.net> wrote:
> Something that's currently just far to manual to figure out is how testr
> is functioning in the parallel case, which tests are running on which
> process, and when does one of the pipelines run out of tests. The testr
> assumption that you don't want output unless there is a failure is a
> really bad assumption, because it's important to know that things are
> progressing, especially in complicated systems.

IMO There are three distinct use cases - we need to be able to tell
what was concurrent (the saved stream should do that post-hoc).
Secondly we need to identify hangs and so on - human scale. Lastly we
need automation around identifying broken things...

> In tempest we actually have a wrapper that gives us a stream. But it's
> got a couple of fundamental flaws.
>
> First - it is very clear that the expected behavior of start / end of
> tests doesn't work:
>
> 2014-01-24 00:49:59.005 |
> tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_image_with_min_ram[gate]
> 2014-01-24 00:49:59.005 |
> tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_image_with_min_ram[gate]
> ... ok

subunit2pyunit just routes everything through a regular python
'unittest.TextTestResult' which doesn't know about timing data or
concurrency. So we need to replace that.


> Question #1: how do we either make start message be when things actually
> start? or turn it off entirely?
>
>
> Second - it's not entirely obvious how unbalanced the pipelines are, and
> that would be extremely useful to see in the output. A starting point
> for a UI would be to add...
>
> " " x (5 * pipelinenumber)
>
> To the test results. Then you'd have a visual waterfall to the front of
> the tests, and could clearly see pipeline differences.

would (worker-N) be sufficient? Or do you want *'s/x's specifically?

> Question #2: where would this be injectable in the stream of testtools /
> subunit / testr / unittest?

Long term I defintely want a richer UI in testr, but right now the
most appropriate thing to do is to prototype the UI you need locally.

-Rob


-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud