Open Stack

On 01/29/2014 04:19 PM, Robert Collins wrote:
> On 26 January 2014 02:25, Sean Dague <sean at dague.net> wrote:
>> Something that's currently just far to manual to figure out is how testr
>> is functioning in the parallel case, which tests are running on which
>> process, and when does one of the pipelines run out of tests. The testr
>> assumption that you don't want output unless there is a failure is a
>> really bad assumption, because it's important to know that things are
>> progressing, especially in complicated systems.
> 
> IMO There are three distinct use cases - we need to be able to tell
> what was concurrent (the saved stream should do that post-hoc).
> Secondly we need to identify hangs and so on - human scale. Lastly we
> need automation around identifying broken things...

Sure.

>> In tempest we actually have a wrapper that gives us a stream. But it's
>> got a couple of fundamental flaws.
>>
>> First - it is very clear that the expected behavior of start / end of
>> tests doesn't work:
>>
>> 2014-01-24 00:49:59.005 |
>> tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_image_with_min_ram[gate]
>> 2014-01-24 00:49:59.005 |
>> tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_image_with_min_ram[gate]
>> ... ok
> 
> subunit2pyunit just routes everything through a regular python
> 'unittest.TextTestResult' which doesn't know about timing data or
> concurrency. So we need to replace that.

Ok, sounds fair.

>> Question #1: how do we either make start message be when things actually
>> start? or turn it off entirely?
>>
>>
>> Second - it's not entirely obvious how unbalanced the pipelines are, and
>> that would be extremely useful to see in the output. A starting point
>> for a UI would be to add...
>>
>> " " x (5 * pipelinenumber)
>>
>> To the test results. Then you'd have a visual waterfall to the front of
>> the tests, and could clearly see pipeline differences.
> 
> would (worker-N) be sufficient? Or do you want *'s/x's specifically?

So if we had worker-N we could play with a filter above that layer to do
UI formatting.

Realistically I had a stream with enough metadata, and preferably a
solid parsing lib for that stream (so I don't screw that part up), then
we could handle the UI in a filter.

Honestly, UX here is going to require the ability to iterate and
experiment. In much the same way that we've got a completely
disconnected visualization system for zuul from zuul itself, and only
rarely do we need to touch zuul proper to improve status.openstack.org.

>> Question #2: where would this be injectable in the stream of testtools /
>> subunit / testr / unittest?
> 
> Long term I defintely want a richer UI in testr, but right now the
> most appropriate thing to do is to prototype the UI you need locally.

Sounds great. Again if we had some of the piece parts, I could do the
rest of the UI locally, just diving down through the layers made me
realize I really needed to figure out where to hook into, as it wasn't
entirely clear (at least to me).

	-Sean

-- 
Sean Dague
Samsung Research America
sean at dague.net / sean.dague at samsung.com
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 547 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140129/1fd8f836/attachment.pgp>