Open Stack

Hi all,

while I was waiting again for the CI to be fixed and didn't want to
torture it with additional rechecks, I wanted to find out, how much of
our CI infrastructure we waste with rechecks. My assumption was that
every recheck is a waste of resources based on a false negative, because
it renders the previous build useless. So I wrote a small script[1] to
calculate how many rechecks are made on average per built patch-set. It
calculates the number of patch-sets of merged changes that CI was
testing (some patch-sets are not, because they were updated before CI
started testing), the number of rechecks issued on these patch-sets, and
a value "CI-factor", which is the factor by which the rechecks increased
the the CI runs, that is, without rechecks it would be 1, if every
tested patch-set would have exactly one recheck it would be 2.

The results were not as bad as my feeling, we are below 2 for most of
the projects I tested. :-) But still, on THT for instance we use 71%
more resources because of the false negatives. I made monthly
breakdowns, so you can see a positive trend at least.


Here the results:

Project: tripleo-heat-templates

 month  patches  rechecks  CI-factor
     1      221       102       1.46
     2      282       300       2.06
     3      588       567       1.96
     4      220       253       2.15
     5      333       242       1.73
     6      459       325       1.71
     7      612       390       1.64
     8      694       442       1.64
     9      717       440       1.61
    10      474       316       1.67
    11      358       189       1.53
    12      168        80       1.48
 total     5126      3646       1.71

Project: tripleo-common

 month  patches  rechecks  CI-factor
     1       73        29        1.4
     2       59        48       1.81
     3       92       101        2.1
     4       17        19       2.12
     5       47        27       1.57
     6       83        46       1.55
     7       66        26       1.39
     8      209       102       1.49
     9      261       129       1.49
    10      110        51       1.46
    11      121        47       1.39
    12       40        19       1.48
 total     1178       644       1.55

Project: tripleo-puppet-elements

 month  patches  rechecks  CI-factor
     1       24         9       1.38
     2        9        20       3.22
     3        7        16       3.29
     4        9        24       3.67
     5       14        17       2.21
     6       17        33       2.94
     7       12        16       2.33
     8       15        21        2.4
     9       10        14        2.4
    10       12         5       1.42
    11       34        25       1.74
    12       10        13        2.3
 total      173       213       2.23

Project: puppet-tripleo

 month  patches  rechecks  CI-factor
     1       29        23       1.79
     2       36        68       2.89
     3       40        44        2.1
     4       68        74       2.09
     5      129        43       1.33
     6      265       206       1.78
     7      235       118        1.5
     8      193       130       1.67
     9      147       123       1.84
    10      233       159       1.68
    11      137        86       1.63
    12       20         5       1.25
 total     1532      1079        1.7


[1] https://gist.github.com/ansiwen/e139cbf25bc243d30629e0157fc753ff