Open Stack

On 5/23/18 6:49 PM, Sagi Shnaidman wrote:
> Alex,
> 
> the problem is that you're working and focusing mostly on release 
> specific code like featuresets and some scripts. But 
> tripleo-quickstart(-extras) and tripleo-ci is much *much* more than set 
> of featuresets. Only 10% of the code may be related to releases and 
> branches, while other 90% is completely independent and not related to 
> releases.
> 
> So in 90% code we DO need to backport every change, take for example the 
> latest patch to extras: https://review.openstack.org/#/c/570167/, it's 
> fixing reproducer. If oooq-extra was branched, we would need to backport 
> this fix to every and every branch. And the same for all other 90% of 
> code, which is complete nonsense.
> Just because not using "{% if release %}" construct - to block the whole 
> work of CI team and make the CI code is absolutely unmaintainable?
> 
> Some of release related templates we moved recently from tripleo-ci to 
> THT repo like scenarios, OC templates, etc. If we discover another 
> things in oooq that could be moved to branched THT I'd be only happy for 
> that.
> 
> Sometimes it could be hard to maintain one file in extras templates with 
> different logic for releases, like we have in tempest configuration for 
> example. The solution is to create a few release-related templates and 
> use one that match the current branch. It doesn't affect 90% of code and 
> still "branch-like" approach. But I didn't see other scripts that are so 
> release dependent. If we'll have ones, we could do the same. For now I 
> see "{% if release %}" construct working very well.
> 
> I didn't see still any advantage of branching CI code, except of a 
> little bit nicer jinja templates without "{% if release ", but amount of 
> disadvantages is so huge, that it'll literally block all current work in CI.

[tl;dr] branching allows to not run cloned branched jobs against master 
patches. Or patches will wait longer in queues, and fail more often cuz 
of intermittent infra issues. See explanation and some calculations below.

So my main concern against additional stable release cloned jobs 
executed for master branches is that there is an "infra failure fee", 
which is a failure unrelated to the patch under check or gate, like an 
intermittent connectivity/timeout inducted failure. This normally is 
followed by a 'recheck' comment posted by an engineer, and sometimes is 
noticed by the elastic recheck bot as well. Say, that sort of a failure 
has a probability of N. And the real "product failure", which is related 
to the subject patch and not infra, takes P. So chances to fail for a job is

F = (1 - ((1 - N)*(1 - P)).

Now that we have added a two more "branched clones" for RDO CI OVB jobs 
and a two more zuul jobs, we have this equation as

F = (1 - ((1 - N)^4*(1 - P)).

(I assumed the chances to face a product defect for the cloned branched 
jobs remain unchanged).

This might bring significantly increased chances to fail (see some 
examples [0] for the N/P distribution cases). So folks will start 
posting 'recheck' comments now even more often, like x2 times more 
often. Which would make zuul and RDO CI queues larger, and patches 
sitting there longer - ending up with more time to wait for jobs to 
start its check/gate pipelines. That's what I call 'recheck storms'. And 
w/o branched quickstart/extras, we might have those storms amplified, 
tho that fully depends on real N/P distributions.

[0] https://pastebin.com/ckG5G7NG

> 
> Thanks
> 
> 
> 
> On Wed, May 23, 2018 at 7:04 PM, Alex Schultz <aschultz at redhat.com 
> <mailto:aschultz at redhat.com>> wrote:
> 
>     On Wed, May 23, 2018 at 8:30 AM, Sagi Shnaidman <sshnaidm at redhat.com
>     <mailto:sshnaidm at redhat.com>> wrote:
>     > Hi, Sergii
>     >
>     > thanks for the question. It's not first time that this topic is raised and
>     > from first view it could seem that branching would help to that sort of
>     > issues.
>     >
>     > Although it's not the case. Tripleo-quickstart(-extras) is part of CI code,
>     > as well as tripleo-ci repo which have never been branched. The reason for
>     > that is relative small impact on CI code from product branching. Think about
>     > backport almost *every* patch to oooq and extras to all supported branches,
>     > down to newton at least. This will be a really *huge* price and non
>     > reasonable work. Just think about active maintenance of 3-4 versions of CI
>     > code in each of 3 repositories. It will take all time of CI team with almost
>     > zero value of this work.
>     >
> 
>     So I'm not sure I completely agree with this assessment as there is a
>     price paid for every {%if release in [...]%} that we have to carry in
>     oooq{,-extras}.  These go away if we branch because we don't have to
>     worry about breaking previous releases or current release (which may
>     or may not actually have CI results).
> 
>     > What regards patch you listed, we would have backport this change to *every*
>     > branch, and it wouldn't really help to avoid the issue. The source of
>     > problem is not branchless repo here.
>     >
> 
>     No we shouldn't be backporting every change.  The logic in oooq-extras
>     should be version specific and if we're changing an interface in
>     tripleo in a breaking fashion we're doing it wrong in tripleo. If
>     we're backporting things to work around tripleo issues, we're doing it
>     wrong in quickstart.
> 
>     > Regarding catching such issues and Bogdans point, that's right we added a
>     > few jobs to catch such issues in the future and prevent breakages, and a few
>     > running jobs is reasonable price to keep configuration working in all
>     > branches. Comparing to maintenance nightmare with branches of CI code, it's
>     > really a *zero* price.
>     >
> 
>     Nothing is free. If there's a high maintenance cost, we haven't
>     properly identified the optimal way to separate functionality between
>     tripleo/quickstart.  I have repeatedly said that the provisioning
>     parts of quickstart should be separate because those aren't tied to a
>     tripleo version and this along with the scenario configs should be the
>     only unbranched repo we have. Any roles related to how to
>     configure/work with tripleo should be branched and tied to a stable
>     branch of tripleo. This would actually be beneficial for tripleo as
>     well because then we can see when we are introducing backwards
>     incompatible changes.
> 
>     Thanks,
>     -Alex
> 
>      > Thanks
>      >
>      >
>      > On Wed, May 23, 2018 at 3:43 PM, Sergii Golovatiuk
>     <sgolovat at redhat.com <mailto:sgolovat at redhat.com>>
>      > wrote:
>      >>
>      >> Hi,
>      >>
>      >> Looking at [1], I am thinking about the price we paid for not
>      >> branching tripleo-quickstart. Can we discuss the options to prevent
>      >> the issues such as [1]? Thank you in advance.
>      >>
>      >> [1] https://review.openstack.org/#/c/569830/4
>     <https://review.openstack.org/#/c/569830/4>
>      >>
>      >> --
>      >> Best Regards,
>      >> Sergii Golovatiuk
>      >>
>      >>
>     __________________________________________________________________________
>      >> OpenStack Development Mailing List (not for usage questions)
>      >> Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>      >>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>      >
>      >
>      >
>      >
>      > --
>      > Best regards
>      > Sagi Shnaidman
>      >
>      >
>     __________________________________________________________________________
>      > OpenStack Development Mailing List (not for usage questions)
>      > Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>      > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>      >
> 
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> 
> 
> 
> -- 
> Best regards
> Sagi Shnaidman
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando