Open Stack

Thank you Dmitry for very detailed plan and risks assessment.
Do we want to run swarm against custom iso with centos7 on Thu evening to
measure level of regression? I remember that we were considering this
approach.

Regards,
Andrey Maximov


On Wed, Dec 2, 2015 at 12:48 AM, Dmitry Borodaenko <dborodaenko at mirantis.com
> wrote:

> With bit more details, I hope this covers all the risks and decision
> points now.
>
> First of all, current list of outstanding commits:
> https://etherpad.openstack.org/p/fuel_on_centos7
>
> The above list has two sections: backwards compatible changes that can
> be merged one at a time even if the rest of CentOS7 support isn't
> merged, and backwards incompatible changes that break support for
> CentOS6 and must be merged (and, if needed, reverted) all at once.
>
> Decision point 1: FFE for CentOS7
>
> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can it
> be allowed a Feature Freeze Exception? So far, the disruption of the
> Fuel development process implied by the proposed merge plan is
> acceptable, if anything goes wrong and we become unable to have a stable
> ISO with merged CentOS7 support on Monday, December 7, the FFE will be
> revoked.
>
> Wed, Dec 2: Merge party
>
> Merge party before 8.0 FF, we should do our best to merge all remaining
> feature commits before end of day (including backwards compatible
> CentOS7 support commits), without breaking the build too much.
>
> At the end of the day we'll start a swarm test over the result of the
> merge party, and we expect QA to analyze and summarize the results by
> 17:00 MSK (6:00 PST) on Thu Dec 3.
>
> Risk 1: Merge party breaks the build
>
> If there is a large regression in swarm pass percentage, we won't be
> able to afford a merge freeze which is necessary to merge CentOS7
> support, we'll have to be merging bugfixes until swarm test pass rate is
> back around 70%.
>
> Risk 2: More features get FFE
>
> If some essential 8.0 features are not completely merged by end of day
> Wed Dec 2 and are granted FFE, merging the remaining commits can
> interfere with merging CentOS7 support, not just from merge conflicts
> perspective, but also invalidating swarm results and making it
> practically impossible to bisect and attribute potential regressions.
>
> Thu, Dec 3: Start merge freeze for CentOS7
>
> Decision point 2: Other FFEs
>
> In the morning MSK time, we will assess Risk 2 and decide what to do
> with the other FFEs. The options are: integrate remaining commits into
> CentOS7 merge plan, block remaining commits until Monday, revoke CentOS7
> FFE.
>
> If the decision is to go ahead with CentOS7 merge, we announce merge
> freeze for all git repositories that go into Fuel ISO, and spend the
> rest of the day rebasing and cleaning up the rest of the CentOS7 commits
> to make sure they're all in mergeable state by the end of the day. The
> outcome of this work must be a custom ISO image with all remaining
> commits, with additional requirement that it must not use Jenkins job
> parameters (only patches to fuel-main that change default repository
> paths) to specify all required package repositories. This will validate
> the proposed fuel-main patches and ensure that no unmerged package
> changes are used to produce the ISO.
>
> Decision point 3: Swarm pass rate
>
> After swarm results from Wed are available, we will assess the Risk 1.
> If the pass rate regression is significant, CentOS7 FFE is revoked and
> merge freeze is lifted. If regression is acceptable, we proceed with
> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
>
> Fri, Dec 4: Merge and test CentOS7
>
> The team will have until 17:00 MSK to produce a non-custom ISO that
> passes BVT and can be run through swarm.
>
> Sat, Dec 5: Assess CentOS7 swarm and bugfix
>
> First of all, someone from CI and QA teams should commit to monitoring
> the CentOS7 swarm run and report the results as soon as possible. Based
> on the results (which once again must be available by 17:00 MSK), we can
> decide on the final step of the plan.
>
> Decision point 4: Keep or revert
>
> If CentOS7 based swarm shows significant regression, we have to spend
> the rest of the weekend including Sunday reverting all CentOS7 commits
> that were merged during merge freeze. Once revert is completed, we will
> lift the merge freeze.
>
> If the regression is acceptable, we lift the merge freeze straight away
> and proceed with bugfixing as usual. At this point CI team will need to
> update the Fuel ISO used for deployment tests in our CI to this same
> ISO.
>
> One way or the other, we will be able to resume bugfixing on Monday
> morning MSK time, and will have lost 2 business days (Thu-Fri) during
> which we won't be able to merge bugfixes. In addition to that, someone
> from QA and everyone from CentOS7 support team has to work on Saturday,
> and someone from CI will have to work a few hours on Sunday.
>
> --
> Dmitry Borodaenko
>
>
> On Tue, Dec 01, 2015 at 05:58:42PM +0300, Dmitry Teselkin wrote:
> > Hello,
> >
> > We're almost got green BVT on custom CentOS7 ISO and it seems that it's
> > the time to discuss the plan how this feature could be merged.
> >
> > This is not the only one feature that is in a queue. Unfortunately,
> > almost any other feature will be broken if merged after CentOS7, so it
> > was decided to merge our changes last.
> >
> > This is not an official announcement, rather a notification letter to
> > start a discussion and find any objections.
> >
> > So the plan is:
> >
> > * merge all features that are going to be merged before Thusday, Dec 3
> > * call for merge freeze starting at Dec 3, due Dec 7
> > * rebase all CentOS7-related pathsets and resolve any conflicts with
> >   merged code (Dec 3)
> > * build custom ISO, pass BVT (and other tests) (Dec 3)
> > * merge all CentOS7-related patchsets at once (Dec 4)
> > * build an ISO and pass BVT again (Dec 4)
> > * run additional test during weekend (Dec 5, 6) to be sure that ISO
> >   good enough
> >
> > According to this plan on Monday, Dec 7 we should either get CentOS7
> > based ISO, or revert all incompatible changes.
> >
> > --
> > Thanks,
> > Dmitry Teselkin
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151202/81c47930/attachment.html>