Open Stack

On 2016-03-07 23:54:49 +0800 (+0800), Duncan Thomas wrote:
> Complexity can be tricky to spot by hand, and expecting reviewers to get it
> right all of the time is not a reasonable expectation.
> 
> My ideal would be something that processes the commit and the jenkins logs,
> extracts the timing info of any new tests, and if they are outside some
> (fairly tight) window, then posts a comment to the review indicating that
> these tests should get closer scrutiny. This does not remove reviewer
> judgement from the equation, just provides a helpful prod that there's
> something to be considered.

Has any analysis been performed on the existing body of test timing
data so far (e.g., by querying the subunit2sql data used for
<URL: http://status.openstack.org/openstack-health/#/job/gate-cinder-python27?groupKey=project&resolutionKey=hour >)?
I suspect you'll find that evaluating durations of tests on the
basis of individual runs is fraught with false positive results due
to the significant variability in our CI workers across different
service providers. If it really were predictably consistent, then
it seems like just adding a timeout fixture at whatever you
determine is a sane duration would be sufficient.
-- 
Jeremy Stanley