Open Stack

With bit more details, I hope this covers all the risks and decision
points now.

First of all, current list of outstanding commits:
https://etherpad.openstack.org/p/fuel_on_centos7

The above list has two sections: backwards compatible changes that can
be merged one at a time even if the rest of CentOS7 support isn't
merged, and backwards incompatible changes that break support for
CentOS6 and must be merged (and, if needed, reverted) all at once.

Decision point 1: FFE for CentOS7

CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can it
be allowed a Feature Freeze Exception? So far, the disruption of the
Fuel development process implied by the proposed merge plan is
acceptable, if anything goes wrong and we become unable to have a stable
ISO with merged CentOS7 support on Monday, December 7, the FFE will be
revoked.

Wed, Dec 2: Merge party

Merge party before 8.0 FF, we should do our best to merge all remaining
feature commits before end of day (including backwards compatible
CentOS7 support commits), without breaking the build too much.

At the end of the day we'll start a swarm test over the result of the
merge party, and we expect QA to analyze and summarize the results by
17:00 MSK (6:00 PST) on Thu Dec 3.

Risk 1: Merge party breaks the build

If there is a large regression in swarm pass percentage, we won't be
able to afford a merge freeze which is necessary to merge CentOS7
support, we'll have to be merging bugfixes until swarm test pass rate is
back around 70%.

Risk 2: More features get FFE

If some essential 8.0 features are not completely merged by end of day
Wed Dec 2 and are granted FFE, merging the remaining commits can
interfere with merging CentOS7 support, not just from merge conflicts
perspective, but also invalidating swarm results and making it
practically impossible to bisect and attribute potential regressions.

Thu, Dec 3: Start merge freeze for CentOS7

Decision point 2: Other FFEs

In the morning MSK time, we will assess Risk 2 and decide what to do
with the other FFEs. The options are: integrate remaining commits into
CentOS7 merge plan, block remaining commits until Monday, revoke CentOS7
FFE.

If the decision is to go ahead with CentOS7 merge, we announce merge
freeze for all git repositories that go into Fuel ISO, and spend the
rest of the day rebasing and cleaning up the rest of the CentOS7 commits
to make sure they're all in mergeable state by the end of the day. The
outcome of this work must be a custom ISO image with all remaining
commits, with additional requirement that it must not use Jenkins job
parameters (only patches to fuel-main that change default repository
paths) to specify all required package repositories. This will validate
the proposed fuel-main patches and ensure that no unmerged package
changes are used to produce the ISO.

Decision point 3: Swarm pass rate

After swarm results from Wed are available, we will assess the Risk 1.
If the pass rate regression is significant, CentOS7 FFE is revoked and
merge freeze is lifted. If regression is acceptable, we proceed with
merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.

Fri, Dec 4: Merge and test CentOS7

The team will have until 17:00 MSK to produce a non-custom ISO that
passes BVT and can be run through swarm.

Sat, Dec 5: Assess CentOS7 swarm and bugfix

First of all, someone from CI and QA teams should commit to monitoring
the CentOS7 swarm run and report the results as soon as possible. Based
on the results (which once again must be available by 17:00 MSK), we can
decide on the final step of the plan.

Decision point 4: Keep or revert

If CentOS7 based swarm shows significant regression, we have to spend
the rest of the weekend including Sunday reverting all CentOS7 commits
that were merged during merge freeze. Once revert is completed, we will
lift the merge freeze.

If the regression is acceptable, we lift the merge freeze straight away
and proceed with bugfixing as usual. At this point CI team will need to
update the Fuel ISO used for deployment tests in our CI to this same
ISO.

One way or the other, we will be able to resume bugfixing on Monday
morning MSK time, and will have lost 2 business days (Thu-Fri) during
which we won't be able to merge bugfixes. In addition to that, someone
from QA and everyone from CentOS7 support team has to work on Saturday,
and someone from CI will have to work a few hours on Sunday.

-- 
Dmitry Borodaenko


On Tue, Dec 01, 2015 at 05:58:42PM +0300, Dmitry Teselkin wrote:
> Hello,
> 
> We're almost got green BVT on custom CentOS7 ISO and it seems that it's
> the time to discuss the plan how this feature could be merged.
> 
> This is not the only one feature that is in a queue. Unfortunately,
> almost any other feature will be broken if merged after CentOS7, so it
> was decided to merge our changes last.
> 
> This is not an official announcement, rather a notification letter to
> start a discussion and find any objections.
> 
> So the plan is:
> 
> * merge all features that are going to be merged before Thusday, Dec 3
> * call for merge freeze starting at Dec 3, due Dec 7
> * rebase all CentOS7-related pathsets and resolve any conflicts with
>   merged code (Dec 3)
> * build custom ISO, pass BVT (and other tests) (Dec 3)
> * merge all CentOS7-related patchsets at once (Dec 4)
> * build an ISO and pass BVT again (Dec 4)
> * run additional test during weekend (Dec 5, 6) to be sure that ISO
>   good enough
> 
> According to this plan on Monday, Dec 7 we should either get CentOS7
> based ISO, or revert all incompatible changes.
> 
> -- 
> Thanks,
> Dmitry Teselkin
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev