Open Stack

Hello,

According to the CentOS7 merging plan we've merged all CentOS7 CRs and
are going to get our first builds in production CI.

Please note that failures are possible for short period of time, please
don't panic, we will keep an eye on every build to investigate and fix
ASAP. We will send another notification letter when it'll be stable
again. We are going to work on fixing issues (if any) until Sunday
(Dec, 6) since it's the next decision point when it'll be clear should
we go with CentOS7 further or revert.

I'd like to remind that merge freeze still in place, no merges allowed
until it will be explicitly notified.


On Tue, 1 Dec 2015 13:48:00 -0800
Dmitry Borodaenko <dborodaenko at mirantis.com> wrote:

> With bit more details, I hope this covers all the risks and decision
> points now.
> 
> First of all, current list of outstanding commits:
> https://etherpad.openstack.org/p/fuel_on_centos7
> 
> The above list has two sections: backwards compatible changes that can
> be merged one at a time even if the rest of CentOS7 support isn't
> merged, and backwards incompatible changes that break support for
> CentOS6 and must be merged (and, if needed, reverted) all at once.
> 
> Decision point 1: FFE for CentOS7
> 
> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can
> it be allowed a Feature Freeze Exception? So far, the disruption of
> the Fuel development process implied by the proposed merge plan is
> acceptable, if anything goes wrong and we become unable to have a
> stable ISO with merged CentOS7 support on Monday, December 7, the FFE
> will be revoked.
> 
> Wed, Dec 2: Merge party
> 
> Merge party before 8.0 FF, we should do our best to merge all
> remaining feature commits before end of day (including backwards
> compatible CentOS7 support commits), without breaking the build too
> much.
> 
> At the end of the day we'll start a swarm test over the result of the
> merge party, and we expect QA to analyze and summarize the results by
> 17:00 MSK (6:00 PST) on Thu Dec 3.
> 
> Risk 1: Merge party breaks the build
> 
> If there is a large regression in swarm pass percentage, we won't be
> able to afford a merge freeze which is necessary to merge CentOS7
> support, we'll have to be merging bugfixes until swarm test pass rate
> is back around 70%.
> 
> Risk 2: More features get FFE
> 
> If some essential 8.0 features are not completely merged by end of day
> Wed Dec 2 and are granted FFE, merging the remaining commits can
> interfere with merging CentOS7 support, not just from merge conflicts
> perspective, but also invalidating swarm results and making it
> practically impossible to bisect and attribute potential regressions.
> 
> Thu, Dec 3: Start merge freeze for CentOS7
> 
> Decision point 2: Other FFEs
> 
> In the morning MSK time, we will assess Risk 2 and decide what to do
> with the other FFEs. The options are: integrate remaining commits into
> CentOS7 merge plan, block remaining commits until Monday, revoke
> CentOS7 FFE.
> 
> If the decision is to go ahead with CentOS7 merge, we announce merge
> freeze for all git repositories that go into Fuel ISO, and spend the
> rest of the day rebasing and cleaning up the rest of the CentOS7
> commits to make sure they're all in mergeable state by the end of the
> day. The outcome of this work must be a custom ISO image with all
> remaining commits, with additional requirement that it must not use
> Jenkins job parameters (only patches to fuel-main that change default
> repository paths) to specify all required package repositories. This
> will validate the proposed fuel-main patches and ensure that no
> unmerged package changes are used to produce the ISO.
> 
> Decision point 3: Swarm pass rate
> 
> After swarm results from Wed are available, we will assess the Risk 1.
> If the pass rate regression is significant, CentOS7 FFE is revoked and
> merge freeze is lifted. If regression is acceptable, we proceed with
> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
> 
> Fri, Dec 4: Merge and test CentOS7
> 
> The team will have until 17:00 MSK to produce a non-custom ISO that
> passes BVT and can be run through swarm.
> 
> Sat, Dec 5: Assess CentOS7 swarm and bugfix
> 
> First of all, someone from CI and QA teams should commit to monitoring
> the CentOS7 swarm run and report the results as soon as possible.
> Based on the results (which once again must be available by 17:00
> MSK), we can decide on the final step of the plan.
> 
> Decision point 4: Keep or revert
> 
> If CentOS7 based swarm shows significant regression, we have to spend
> the rest of the weekend including Sunday reverting all CentOS7 commits
> that were merged during merge freeze. Once revert is completed, we
> will lift the merge freeze.
> 
> If the regression is acceptable, we lift the merge freeze straight
> away and proceed with bugfixing as usual. At this point CI team will
> need to update the Fuel ISO used for deployment tests in our CI to
> this same ISO.
> 
> One way or the other, we will be able to resume bugfixing on Monday
> morning MSK time, and will have lost 2 business days (Thu-Fri) during
> which we won't be able to merge bugfixes. In addition to that, someone
> from QA and everyone from CentOS7 support team has to work on
> Saturday, and someone from CI will have to work a few hours on Sunday.
> 



-- 
Thanks,
Dmitry Teselkin