Open Stack

Dmitry,
thank you!

I confirm that the plan looks good for our team, we'll follow it.

Regards,
Igor Marnat

On Wed, Dec 2, 2015 at 9:39 PM, Andrew Maksimov <amaksimov at mirantis.com>
wrote:

> Thank you Dmitry for very detailed plan and risks assessment.
> Do we want to run swarm against custom iso with centos7 on Thu evening to
> measure level of regression? I remember that we were considering this
> approach.
>
> Regards,
> Andrey Maximov
>
>
> On Wed, Dec 2, 2015 at 12:48 AM, Dmitry Borodaenko <
> dborodaenko at mirantis.com> wrote:
>
>> With bit more details, I hope this covers all the risks and decision
>> points now.
>>
>> First of all, current list of outstanding commits:
>> https://etherpad.openstack.org/p/fuel_on_centos7
>>
>> The above list has two sections: backwards compatible changes that can
>> be merged one at a time even if the rest of CentOS7 support isn't
>> merged, and backwards incompatible changes that break support for
>> CentOS6 and must be merged (and, if needed, reverted) all at once.
>>
>> Decision point 1: FFE for CentOS7
>>
>> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can it
>> be allowed a Feature Freeze Exception? So far, the disruption of the
>> Fuel development process implied by the proposed merge plan is
>> acceptable, if anything goes wrong and we become unable to have a stable
>> ISO with merged CentOS7 support on Monday, December 7, the FFE will be
>> revoked.
>>
>> Wed, Dec 2: Merge party
>>
>> Merge party before 8.0 FF, we should do our best to merge all remaining
>> feature commits before end of day (including backwards compatible
>> CentOS7 support commits), without breaking the build too much.
>>
>> At the end of the day we'll start a swarm test over the result of the
>> merge party, and we expect QA to analyze and summarize the results by
>> 17:00 MSK (6:00 PST) on Thu Dec 3.
>>
>> Risk 1: Merge party breaks the build
>>
>> If there is a large regression in swarm pass percentage, we won't be
>> able to afford a merge freeze which is necessary to merge CentOS7
>> support, we'll have to be merging bugfixes until swarm test pass rate is
>> back around 70%.
>>
>> Risk 2: More features get FFE
>>
>> If some essential 8.0 features are not completely merged by end of day
>> Wed Dec 2 and are granted FFE, merging the remaining commits can
>> interfere with merging CentOS7 support, not just from merge conflicts
>> perspective, but also invalidating swarm results and making it
>> practically impossible to bisect and attribute potential regressions.
>>
>> Thu, Dec 3: Start merge freeze for CentOS7
>>
>> Decision point 2: Other FFEs
>>
>> In the morning MSK time, we will assess Risk 2 and decide what to do
>> with the other FFEs. The options are: integrate remaining commits into
>> CentOS7 merge plan, block remaining commits until Monday, revoke CentOS7
>> FFE.
>>
>> If the decision is to go ahead with CentOS7 merge, we announce merge
>> freeze for all git repositories that go into Fuel ISO, and spend the
>> rest of the day rebasing and cleaning up the rest of the CentOS7 commits
>> to make sure they're all in mergeable state by the end of the day. The
>> outcome of this work must be a custom ISO image with all remaining
>> commits, with additional requirement that it must not use Jenkins job
>> parameters (only patches to fuel-main that change default repository
>> paths) to specify all required package repositories. This will validate
>> the proposed fuel-main patches and ensure that no unmerged package
>> changes are used to produce the ISO.
>>
>> Decision point 3: Swarm pass rate
>>
>> After swarm results from Wed are available, we will assess the Risk 1.
>> If the pass rate regression is significant, CentOS7 FFE is revoked and
>> merge freeze is lifted. If regression is acceptable, we proceed with
>> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
>>
>> Fri, Dec 4: Merge and test CentOS7
>>
>> The team will have until 17:00 MSK to produce a non-custom ISO that
>> passes BVT and can be run through swarm.
>>
>> Sat, Dec 5: Assess CentOS7 swarm and bugfix
>>
>> First of all, someone from CI and QA teams should commit to monitoring
>> the CentOS7 swarm run and report the results as soon as possible. Based
>> on the results (which once again must be available by 17:00 MSK), we can
>> decide on the final step of the plan.
>>
>> Decision point 4: Keep or revert
>>
>> If CentOS7 based swarm shows significant regression, we have to spend
>> the rest of the weekend including Sunday reverting all CentOS7 commits
>> that were merged during merge freeze. Once revert is completed, we will
>> lift the merge freeze.
>>
>> If the regression is acceptable, we lift the merge freeze straight away
>> and proceed with bugfixing as usual. At this point CI team will need to
>> update the Fuel ISO used for deployment tests in our CI to this same
>> ISO.
>>
>> One way or the other, we will be able to resume bugfixing on Monday
>> morning MSK time, and will have lost 2 business days (Thu-Fri) during
>> which we won't be able to merge bugfixes. In addition to that, someone
>> from QA and everyone from CentOS7 support team has to work on Saturday,
>> and someone from CI will have to work a few hours on Sunday.
>>
>> --
>> Dmitry Borodaenko
>>
>>
>> On Tue, Dec 01, 2015 at 05:58:42PM +0300, Dmitry Teselkin wrote:
>> > Hello,
>> >
>> > We're almost got green BVT on custom CentOS7 ISO and it seems that it's
>> > the time to discuss the plan how this feature could be merged.
>> >
>> > This is not the only one feature that is in a queue. Unfortunately,
>> > almost any other feature will be broken if merged after CentOS7, so it
>> > was decided to merge our changes last.
>> >
>> > This is not an official announcement, rather a notification letter to
>> > start a discussion and find any objections.
>> >
>> > So the plan is:
>> >
>> > * merge all features that are going to be merged before Thusday, Dec 3
>> > * call for merge freeze starting at Dec 3, due Dec 7
>> > * rebase all CentOS7-related pathsets and resolve any conflicts with
>> >   merged code (Dec 3)
>> > * build custom ISO, pass BVT (and other tests) (Dec 3)
>> > * merge all CentOS7-related patchsets at once (Dec 4)
>> > * build an ISO and pass BVT again (Dec 4)
>> > * run additional test during weekend (Dec 5, 6) to be sure that ISO
>> >   good enough
>> >
>> > According to this plan on Monday, Dec 7 we should either get CentOS7
>> > based ISO, or revert all incompatible changes.
>> >
>> > --
>> > Thanks,
>> > Dmitry Teselkin
>> >
>> >
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151202/b2eee0ee/attachment.html>