Open Stack

On Thu, Dec 15, 2016 at 12:22 PM, Sven Anderson <sven at redhat.com> wrote:
> Hi all,
>
> while I was waiting again for the CI to be fixed and didn't want to
> torture it with additional rechecks, I wanted to find out, how much of
> our CI infrastructure we waste with rechecks. My assumption was that
> every recheck is a waste of resources based on a false negative, because
> it renders the previous build useless. So I wrote a small script[1] to
> calculate how many rechecks are made on average per built patch-set. It
> calculates the number of patch-sets of merged changes that CI was
> testing (some patch-sets are not, because they were updated before CI
> started testing), the number of rechecks issued on these patch-sets, and
> a value "CI-factor", which is the factor by which the rechecks increased
> the the CI runs, that is, without rechecks it would be 1, if every
> tested patch-set would have exactly one recheck it would be 2.

I see 2 different topics here.

# One is not related to $topic but still worth mentioning:
"while I was waiting again for the CI to be fixed"

This week has been tough, and many of us burnt our time to resolve
different complex problems in TripleO CI, mostly related to external
dependencies (qemu upgrade, centos 7.3 upgrade, tripleo-ci infra,
etc).
Resolving these problems is very challenging and you'll notice that
only a few of us actually work on this task, while a lot of people
continue to push their features "hoping" that it will pass CI
sometimes and if not, well, we'll do 'recheck'.
That is a way of working I would say. I personally can't continue to
code if the project I'm working on has broken CI.

In a previous experience, I've been working in a team where everyone
stopped regular work when CI was broken and focus on fixing it.
I'm not saying everyone should stop their tasks and help, but this
"wait and see" comment doesn't actually help us to move forward.
People need to get more involved in CI and be more helpful. I know
it's difficult, but it's something anyone can learn, like you would
learn how to write Python code for example.


# The second one is about the actual $topic and your stats.
Yes we have been thinking about a way to optimize the way we restart
CI jobs and this is under discussion:
https://review.openstack.org/#/c/411450/

As you can see, there is some pushback from Clark who is infra-core,
so we might want to continue the discussion here and see how it goes.


On the long-term, our goal is to have more consistency in the way we
test TripleO and get more adoption in the tools we're developing for
CI, so they are more consumable from anyone in our community. Also we
hope to have more people involved when things are broken, and not
always the same folks spending days and evenings to "extinguish
fires". We are working hard on CI stabilization and consolidation with
multinode scenarios and OVB improvements, but it takes time and
iterations.

Any help is highly welcome here.


> The results were not as bad as my feeling, we are below 2 for most of
> the projects I tested. :-) But still, on THT for instance we use 71%
> more resources because of the false negatives. I made monthly
> breakdowns, so you can see a positive trend at least.
>
>
> Here the results:
>
> Project: tripleo-heat-templates
>
>  month  patches  rechecks  CI-factor
>      1      221       102       1.46
>      2      282       300       2.06
>      3      588       567       1.96
>      4      220       253       2.15
>      5      333       242       1.73
>      6      459       325       1.71
>      7      612       390       1.64
>      8      694       442       1.64
>      9      717       440       1.61
>     10      474       316       1.67
>     11      358       189       1.53
>     12      168        80       1.48
>  total     5126      3646       1.71
>
> Project: tripleo-common
>
>  month  patches  rechecks  CI-factor
>      1       73        29        1.4
>      2       59        48       1.81
>      3       92       101        2.1
>      4       17        19       2.12
>      5       47        27       1.57
>      6       83        46       1.55
>      7       66        26       1.39
>      8      209       102       1.49
>      9      261       129       1.49
>     10      110        51       1.46
>     11      121        47       1.39
>     12       40        19       1.48
>  total     1178       644       1.55
>
> Project: tripleo-puppet-elements
>
>  month  patches  rechecks  CI-factor
>      1       24         9       1.38
>      2        9        20       3.22
>      3        7        16       3.29
>      4        9        24       3.67
>      5       14        17       2.21
>      6       17        33       2.94
>      7       12        16       2.33
>      8       15        21        2.4
>      9       10        14        2.4
>     10       12         5       1.42
>     11       34        25       1.74
>     12       10        13        2.3
>  total      173       213       2.23
>
> Project: puppet-tripleo
>
>  month  patches  rechecks  CI-factor
>      1       29        23       1.79
>      2       36        68       2.89
>      3       40        44        2.1
>      4       68        74       2.09
>      5      129        43       1.33
>      6      265       206       1.78
>      7      235       118        1.5
>      8      193       130       1.67
>      9      147       123       1.84
>     10      233       159       1.68
>     11      137        86       1.63
>     12       20         5       1.25
>  total     1532      1079        1.7
>
>
> [1] https://gist.github.com/ansiwen/e139cbf25bc243d30629e0157fc753ff
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi