Open Stack

Great progress, thank you for keeping everyone informed.

On Fri, Dec 4, 2015 at 4:43 PM Dmitry Teselkin <dteselkin at mirantis.com>
wrote:

> Hello,
>
> Status update for Dec, 4 - we're on track.
>
> As there were no significant issues found during BVT / swarm test runs
> on custom CentOS7 ISO it was decided to go with merge party and merge
> all CentOS7 related CRs. Our CI was partially updated then and we
> started building ISO using production jobs. We've got several green
> ISOs which passed BVT. At the moment full set of swarm tests is being
> run, we will analyze them tomorrow.
>
> There are sporadic failures of OSTF tests tied with cinder, we're going
> to analyze them tomorrow too.
>
> On Tue, 1 Dec 2015 13:48:00 -0800
> Dmitry Borodaenko <dborodaenko at mirantis.com> wrote:
>
> > With bit more details, I hope this covers all the risks and decision
> > points now.
> >
> > First of all, current list of outstanding commits:
> > https://etherpad.openstack.org/p/fuel_on_centos7
> >
> > The above list has two sections: backwards compatible changes that can
> > be merged one at a time even if the rest of CentOS7 support isn't
> > merged, and backwards incompatible changes that break support for
> > CentOS6 and must be merged (and, if needed, reverted) all at once.
> >
> > Decision point 1: FFE for CentOS7
> >
> > CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can
> > it be allowed a Feature Freeze Exception? So far, the disruption of
> > the Fuel development process implied by the proposed merge plan is
> > acceptable, if anything goes wrong and we become unable to have a
> > stable ISO with merged CentOS7 support on Monday, December 7, the FFE
> > will be revoked.
> >
> > Wed, Dec 2: Merge party
> >
> > Merge party before 8.0 FF, we should do our best to merge all
> > remaining feature commits before end of day (including backwards
> > compatible CentOS7 support commits), without breaking the build too
> > much.
> >
> > At the end of the day we'll start a swarm test over the result of the
> > merge party, and we expect QA to analyze and summarize the results by
> > 17:00 MSK (6:00 PST) on Thu Dec 3.
> >
> > Risk 1: Merge party breaks the build
> >
> > If there is a large regression in swarm pass percentage, we won't be
> > able to afford a merge freeze which is necessary to merge CentOS7
> > support, we'll have to be merging bugfixes until swarm test pass rate
> > is back around 70%.
> >
> > Risk 2: More features get FFE
> >
> > If some essential 8.0 features are not completely merged by end of day
> > Wed Dec 2 and are granted FFE, merging the remaining commits can
> > interfere with merging CentOS7 support, not just from merge conflicts
> > perspective, but also invalidating swarm results and making it
> > practically impossible to bisect and attribute potential regressions.
> >
> > Thu, Dec 3: Start merge freeze for CentOS7
> >
> > Decision point 2: Other FFEs
> >
> > In the morning MSK time, we will assess Risk 2 and decide what to do
> > with the other FFEs. The options are: integrate remaining commits into
> > CentOS7 merge plan, block remaining commits until Monday, revoke
> > CentOS7 FFE.
> >
> > If the decision is to go ahead with CentOS7 merge, we announce merge
> > freeze for all git repositories that go into Fuel ISO, and spend the
> > rest of the day rebasing and cleaning up the rest of the CentOS7
> > commits to make sure they're all in mergeable state by the end of the
> > day. The outcome of this work must be a custom ISO image with all
> > remaining commits, with additional requirement that it must not use
> > Jenkins job parameters (only patches to fuel-main that change default
> > repository paths) to specify all required package repositories. This
> > will validate the proposed fuel-main patches and ensure that no
> > unmerged package changes are used to produce the ISO.
> >
> > Decision point 3: Swarm pass rate
> >
> > After swarm results from Wed are available, we will assess the Risk 1.
> > If the pass rate regression is significant, CentOS7 FFE is revoked and
> > merge freeze is lifted. If regression is acceptable, we proceed with
> > merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
> >
> > Fri, Dec 4: Merge and test CentOS7
> >
> > The team will have until 17:00 MSK to produce a non-custom ISO that
> > passes BVT and can be run through swarm.
> >
> > Sat, Dec 5: Assess CentOS7 swarm and bugfix
> >
> > First of all, someone from CI and QA teams should commit to monitoring
> > the CentOS7 swarm run and report the results as soon as possible.
> > Based on the results (which once again must be available by 17:00
> > MSK), we can decide on the final step of the plan.
> >
> > Decision point 4: Keep or revert
> >
> > If CentOS7 based swarm shows significant regression, we have to spend
> > the rest of the weekend including Sunday reverting all CentOS7 commits
> > that were merged during merge freeze. Once revert is completed, we
> > will lift the merge freeze.
> >
> > If the regression is acceptable, we lift the merge freeze straight
> > away and proceed with bugfixing as usual. At this point CI team will
> > need to update the Fuel ISO used for deployment tests in our CI to
> > this same ISO.
> >
> > One way or the other, we will be able to resume bugfixing on Monday
> > morning MSK time, and will have lost 2 business days (Thu-Fri) during
> > which we won't be able to merge bugfixes. In addition to that, someone
> > from QA and everyone from CentOS7 support team has to work on
> > Saturday, and someone from CI will have to work a few hours on Sunday.
> >
>
>
>
> --
> Thanks,
> Dmitry Teselkin
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 
Mike Scherbakov
#mihgen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151205/e9d42709/attachment.html>