Open Stack

Hello,

Status update for Dec, 5-6 - we're on track.

CentOS7 based ISO looks good, we are building it on product CI using
the same jobs as for CentOS6.

ISO #258 passed nightly swarm with a slightly reduced coverage (69%).
We've got one known issue [0] that affects some jobs when run on a
loaded slave (there is a fix [1]), and issues on QA side.

Smoke and BVT tests also green [2], however, we've got interference
from Ubuntu upstream here, and had to disable trusty-proposed channel
to get tests passed. More details in a bug [3].

According to our plan [4] we've got over decision point #4 with a
decision to go with CentOS7.

Please note that merge freeze still taken place, it will be explicitly
notified when it's lifted.

[0] https://bugs.launchpad.net/mos/+bug/1523117
[1] https://review.openstack.org/#/c/253843/
[2] https://product-ci.infra.mirantis.net/job/8.0.test_all/245/
[3] https://bugs.launchpad.net/fuel/+bug/1523092
[4]
http://lists.openstack.org/pipermail/openstack-dev/2015-December/081026.html

On Tue, 1 Dec 2015 13:48:00 -0800
Dmitry Borodaenko <dborodaenko at mirantis.com> wrote:

> With bit more details, I hope this covers all the risks and decision
> points now.
> 
> First of all, current list of outstanding commits:
> https://etherpad.openstack.org/p/fuel_on_centos7
> 
> The above list has two sections: backwards compatible changes that can
> be merged one at a time even if the rest of CentOS7 support isn't
> merged, and backwards incompatible changes that break support for
> CentOS6 and must be merged (and, if needed, reverted) all at once.
> 
> Decision point 1: FFE for CentOS7
> 
> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can
> it be allowed a Feature Freeze Exception? So far, the disruption of
> the Fuel development process implied by the proposed merge plan is
> acceptable, if anything goes wrong and we become unable to have a
> stable ISO with merged CentOS7 support on Monday, December 7, the FFE
> will be revoked.
> 
> Wed, Dec 2: Merge party
> 
> Merge party before 8.0 FF, we should do our best to merge all
> remaining feature commits before end of day (including backwards
> compatible CentOS7 support commits), without breaking the build too
> much.
> 
> At the end of the day we'll start a swarm test over the result of the
> merge party, and we expect QA to analyze and summarize the results by
> 17:00 MSK (6:00 PST) on Thu Dec 3.
> 
> Risk 1: Merge party breaks the build
> 
> If there is a large regression in swarm pass percentage, we won't be
> able to afford a merge freeze which is necessary to merge CentOS7
> support, we'll have to be merging bugfixes until swarm test pass rate
> is back around 70%.
> 
> Risk 2: More features get FFE
> 
> If some essential 8.0 features are not completely merged by end of day
> Wed Dec 2 and are granted FFE, merging the remaining commits can
> interfere with merging CentOS7 support, not just from merge conflicts
> perspective, but also invalidating swarm results and making it
> practically impossible to bisect and attribute potential regressions.
> 
> Thu, Dec 3: Start merge freeze for CentOS7
> 
> Decision point 2: Other FFEs
> 
> In the morning MSK time, we will assess Risk 2 and decide what to do
> with the other FFEs. The options are: integrate remaining commits into
> CentOS7 merge plan, block remaining commits until Monday, revoke
> CentOS7 FFE.
> 
> If the decision is to go ahead with CentOS7 merge, we announce merge
> freeze for all git repositories that go into Fuel ISO, and spend the
> rest of the day rebasing and cleaning up the rest of the CentOS7
> commits to make sure they're all in mergeable state by the end of the
> day. The outcome of this work must be a custom ISO image with all
> remaining commits, with additional requirement that it must not use
> Jenkins job parameters (only patches to fuel-main that change default
> repository paths) to specify all required package repositories. This
> will validate the proposed fuel-main patches and ensure that no
> unmerged package changes are used to produce the ISO.
> 
> Decision point 3: Swarm pass rate
> 
> After swarm results from Wed are available, we will assess the Risk 1.
> If the pass rate regression is significant, CentOS7 FFE is revoked and
> merge freeze is lifted. If regression is acceptable, we proceed with
> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
> 
> Fri, Dec 4: Merge and test CentOS7
> 
> The team will have until 17:00 MSK to produce a non-custom ISO that
> passes BVT and can be run through swarm.
> 
> Sat, Dec 5: Assess CentOS7 swarm and bugfix
> 
> First of all, someone from CI and QA teams should commit to monitoring
> the CentOS7 swarm run and report the results as soon as possible.
> Based on the results (which once again must be available by 17:00
> MSK), we can decide on the final step of the plan.
> 
> Decision point 4: Keep or revert
> 
> If CentOS7 based swarm shows significant regression, we have to spend
> the rest of the weekend including Sunday reverting all CentOS7 commits
> that were merged during merge freeze. Once revert is completed, we
> will lift the merge freeze.
> 
> If the regression is acceptable, we lift the merge freeze straight
> away and proceed with bugfixing as usual. At this point CI team will
> need to update the Fuel ISO used for deployment tests in our CI to
> this same ISO.
> 
> One way or the other, we will be able to resume bugfixing on Monday
> morning MSK time, and will have lost 2 business days (Thu-Fri) during
> which we won't be able to merge bugfixes. In addition to that, someone
> from QA and everyone from CentOS7 support team has to work on
> Saturday, and someone from CI will have to work a few hours on Sunday.
> 



-- 
Thanks,
Dmitry Teselkin