Open Stack

On Wed, Sep 21, 2016 at 10:44:51AM -0400, Bob Hansen wrote:
> 
> 
> I have been looking at some of the stackviz output as I'm trying to improve
> the run time of my thrid-party CI. As an example:
> 
> http://logs.openstack.org/36/371836/1/check/gate-tempest-dsvm-full-ubuntu-xenial/087db0f/logs/stackviz/#/stdin/timeline
> 
> What jumps out is the amount of time that each worker is not running any
> tests. I would have expected quite a bit more concurrecy between the two
> workers in the chart, e.g. more overlap. I've noticed a simliar thing with
> my test runs using 4 workers.

So the gaps between tests aren't actually wait time, the workers are saturated
doing stuff during a run. Those gaps are missing data in the subunit streams
that are used as the soure of the data for rendering those timelines. The gaps
are where things like setUp, setUpClass, tearDown, tearDownClass, and
addCleanups which are not added to the subunit stream. It's just an artifact of
the incomplete data, not bad scheduling. This also means that testr does not
take into account any of the missing timing when it makes decisions based on
previous runs.

> 
> Can anyone explain why this is and where can I find out more information
> about the scheduler and what information it is using to decide when to
> dispatch tests? I'm already feeding my system a prior subunit stream to
> help influence the scheduler as my test run times are different due to the
> way our openstack implementation is architected. A simple round-robin
> approach is not the most efficeint in my case.

If you're curious about how testr does scheduling most of that happens here:

https://github.com/testing-cabal/testrepository/blob/master/testrepository/testcommand.py

One thing to remember is that testr isn't actually a test runner, it's a test
runner runner. It partitions the tests based on time information and passes
those to (multiple) test runner workers. The actual order of execution inside
those partitions is handled by the test runner itself. (in our case subunit.run)

-Matt Treinish
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160921/90086167/attachment.pgp>