Open Stack

Sure,

TL;DR was actually hidden in the middle of the email, here's an even 
shorter version:

0) we're suffering from closing master for feature work for too long

1) continuously rebased future branch is most likely a no-go

2) short FF (SCF and stable branch after 2 weeks) is an option for 8.0

3) short FF with stable in a separate internal gerrit was also proposed

4) merits and cost of enabling CI setup for private forks should be 
   carefully considered independently from other options

HTH
-DmitryB

On Tue, Aug 25, 2015 at 12:01:26PM +0200, Roman Prykhodchenko wrote:
>Dmitry,
>
>thank you for a well described plan.
>May I please ask you for a little TL;DR excerpt of what’s going to be 
>changed, because I’m affraid some folks may get lost or may not have 
>enough time to analyze it deeply (I actually see that happening but I 
>won’t do fingerpointing :) ).
>
>
>- romcheg
>
>> 24 серп. 2015 о 20:29 Dmitry Borodaenko <dborodaenko at mirantis.com> написав(ла):
>>
>> We have several problems with Fuel branching strategy that have become
>> enough of a bottleneck to consider re-thinking our whole release
>> management process. In this email, I will describe the problems, propose
>> a couple of alternative strategies, and try to analyze their relative
>> merits and associated risks.
>>
>> I have my opinions and preferences, but I will try my best to
>> objectively compare all available options. My goal is to improve
>> efficiency of the existing team and make it significantly easier for new
>> people to contribute to Fuel, even when their schedule and their agenda
>> is not 100% aligned with those of Fuel core team and Mirantis OpenStack.
>>
>> It is essential for the new process to be acceptable for Fuel
>> contributors, and is is just as essential to reach a consensus quickly:
>> with Fuel 7.0 Hard Code Freeze less than two weeks away [0], we're
>> already late with planning 8.0, and we absolutely must have the whole
>> plan for 8.0 before Fuel 7.0 is released (September 24).
>>
>> [0] https://wiki.openstack.org/wiki/Fuel/7.0_Release_Schedule
>>
>> I propose a time-bound rough consensus to make this decision: raise all
>> concerns and risks before end of this week (Friday, August 28), propose
>> and discuss ways to address the raised concerns, and make final decision
>> before end of next week (Friday, September 4).
>>
>> Here's how Fuel versions, branches, and release milestones work now.
>>
>> Major Fuel version corresponds to an OpenStack release. For example,
>> Fuel 7.0 is the first version to support Kilo. Minor version denotes a
>> feature release. For example, Fuel 6.1 is still based on Juno but
>> contains new features relative to Fuel 6.0. Tiny versions (e.g. 5.1.1)
>> and maintenance updates (e.g. 6.1-mu-1) include only bugfixes.
>>
>> Most Fuel development happens on the master branch. On Hard Code Freeze,
>> a stable branch is created in all Fuel git repositories (e.g. stable/7.0
>> will be created on September 3). After that, all changes targeted to
>> that release series must be proposed, reviewed, and merged to master
>> before they are proposed for any stable branches [1]. One release series
>> and one stable branch is created per minor Fuel version (e.g. stable/6.1
>> for 6.1.x).
>>
>> [1] https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Backport_bugfixes_to_stable_release_series
>>
>> Fuel release cycle starts from Hard Code freeze of the previous release:
>>
>> - Hard Code Freeze: stable branch created, master branch is open for all
>> changes
>> - Design Complete: all release features have specs merged in fuel-specs
>> - Feature Freeze: feature changes no longer accepted in master branch
>> - Soft Code Freeze: medium and lower priority bugfixes no longer
>> accepted in master branch
>>
>> Looks similar to the way OpenStack branch model works [2], but there's
>> an important difference.
>>
>> [2] https://wiki.openstack.org/wiki/Branch_Model
>>
>> In OpenStack, master branch is closed for feature commits for 2 weeks
>> per release [3], or 1/13th of the whole 26-week release cycle.
>>
>> [3] https://wiki.openstack.org/wiki/Liberty_Release_Schedule
>>
>> In Fuel, master branch remains closed for almost half of the release
>> cycle:
>>
>> 5.0 -- 32 of 81 days
>> 5.1 -- 63 of 119 days
>> 6.0 -- 35 of 93 days
>> 6.1 -- 85 of 180 days
>> 7.0 -- 42 of 93 days (planned)
>>
>> This renders it unusable as an integration branch: if you are bound by a
>> schedule that is not aligned with Fuel release milestones and have more
>> changes than a single commit which you could keep rebasing until Fuel
>> master is open again, you're better off merging your changes into your
>> own integration branch (i.e. fork Fuel).
>>
>> The same problem is even worse if you're working on the next OpenStack
>> release. Even when Fuel master is open, it's developed and tested
>> against latest stable release of OpenStack. For example, even though
>> OpenStack developers started working on Liberty features in May,
>> reflecting that work in Fuel master is blocked until September.
>>
>> There are 4 partially overlapping solutions to this problem:
>>
>> 1) Future branch: create a "future" integration branch on FF, rebase it
>> weekly onto master (or merge weekly from master to future), merge future
>> to master after stable branch is created on HCF.
>>
>> 2) Short FF: create stable branch 2 weeks after FF (same as OpenStack)
>> instead of waiting for HCF.
>>
>> 3) Short FF and internal fork: create stable branch 2 weeks after FF,
>> create an internal fork of stable branch for Mirantis OpenStack.
>>
>> 4) CI for external forks: package and document Fuel development
>> infrastructure so that anyone who wants to fork Fuel can set up their
>> own CI.
>>
>> In theory, we could implement all four, none of them, or come up with a
>> fifth option. But before we vote, we should identify all possible risks
>> and concerns, so that voting is based on facts more than on personal
>> preferences.
>>
>> 1. Future branch
>> ----------------
>>
>> This option was originally proposed by Vladimir Kuklin as a way to
>> maintain continuous support of OpenStack master branch outside of Fuel
>> master branch.
>>
>> When I first came to Mike Scherbakov with a proposal to create
>> stable/7.0 early to unblock feature work for Fuel 8.0, his biggest
>> concern was the overhead of backporting bugfixing commits from master:
>> it's negligible for the dozen or so bugs we fix between HCF and GA, but
>> may add up to a lot for the hundreds of bugs we typically fix between FF
>> and HCF. As an alternative that mitigates this risk, I've recycled
>> Vladimir's proposal.
>>
>> Eventually, Mike has realized that the confusion of targeting different
>> branches during different stages of Fuel release cycle, on top of the
>> contentious question of who's supposed to do the rebase work, adds up to
>> just as much overhead and risk, so he has withdrawn his support for this
>> option in favor of short FF [4].
>>
>> [4] http://eavesdrop.openstack.org/meetings/fuel/2015/fuel.2015-08-20-16.01.log.html#l-221
>>
>> While it might have worked as a one-off solution during 7.0 FF if we had
>> done it in mid-August (and even by then, it was already too late for the
>> more intrusive options), by the time we conclude this discussion it will
>> be too late to implement it for 7.0. As a long-term solution for 8.0 and
>> onward, it certainly looks less attractive than the alternatives. Still,
>> I'm including it in the list so that we consider this option now and
>> don't have to come back to it again later.
>>
>> Pros:
>> - does not require changes to Fuel release cycle
>> - puts overhead on feature developers who are motivated to deal with it
>> - keeps overhead away from low-motivation and high-risk bugfixing work
>>
>> Contras:
>> - different target branch for feature commits during FF adds confusion
>> - backporting bugfix commits one at a time scales better than rebasing a
>> whole integration branch
>> - cores are distracted from bugfixing to review feature commits after FF
>>
>> 2. Short FF
>> -----------
>>
>> Long feature freeze is not an oversight in the Fuel release cycle, it
>> was a conscious decision intended to encourage a small team of Fuel
>> developers to put feature work on hold and focus on bugs. Since 2013,
>> the number of active Fuel contributors has grown from 30 to over 230
>> people [5][6]. At this scale, bugfixing is less of a bottleneck than
>> integrating external contributions, and it looks like the long feature
>> freeze has outlived its usefulness.
>>
>> [5] http://stackalytics.com/?project_type=stackforge&module=fuel-group&release=havana
>> [6] http://stackalytics.com/?project_type=stackforge&module=fuel-group&release=liberty
>>
>> Following the same FF process as OpenStack is also a benefit in itself:
>> since Fuel versions are coupled with OpenStack releases, it makes sense
>> for Fuel to follow the same release cycle. We won't be able to do this
>> in 8.0 (Liberty FF is only 1 week after Fuel 7.0 GA), but if we open
>> stable/8.0 early enough in the Mitaka cycle, having 9.0 FF at the same
>> time as Mitaka FF becomes possible.
>>
>> There was some ambiguity in our internal discussions about how short
>> exactly the FF is supposed to become (create stable branch on FF or 2
>> weeks after FF), for the sake of simplicity I propose to stick to the
>> OpenStack process (2 weeks after FF), and also to declare SCF at the
>> same time (so that we don't have to deal with backports of Medium
>> priority bugs from master to stable).
>>
>> Pros:
>> - no changes in branch naming and bugfix backporting rules
>> - consistent with OpenStack release cycle
>> - backporting bugfix commits one at a time scales better than rebasing a
>> whole integration branch
>>
>> Contras:
>> - adds overhead to the biggest release scheduling risk: bugfixing
>> - cores are distracted from bugfixing to review feature commits after FF
>> - merging commits becomes dependent on community code review and
>> stability of OpenStack master CI
>>
>> 3. Short FF and internal fork
>> -----------------------------
>>
>> Igor Marnat has proposed to go one step further and replicate the branch
>> model we use for OpenStack components in Mirantis OpenStack: for each
>> MOS release, create an internal branch from upstream stable branch.
>>
>> The motivation is to completely decouple Fuel release cycle from
>> Mirantis OpenStack, and to remove community code review and stability of
>> OpenStack master CI from factors limiting our ability to merge commits
>> needed by MOS.
>>
>> Igor did not explicitly state whether the internal Mirantis branch of
>> Fuel would take only bugfixes, or feature work, too. In option #2, I
>> removed ambiguity by eliminating the higher risk variants. Here, the
>> variants are fundamentally different and deserve an independent
>> consideration.
>>
>> If we use the internal branch only for bugfixes, it does not decouple
>> MOS release cycle from Fuel: either MOS release cycle aligns with that
>> of upstream OpenStack, and we develop new Fuel features for MOS against
>> OpenStack master, or Fuel release cycle remains aligned with MOS, and we
>> develop new Fuel features against OpenStack stable branch.
>>
>> If we use the internal branch to develop new Fuel features based on a
>> stable branch of OpenStack, we're creating a fork of community version
>> of Fuel, something we're not doing with other OpenStack projects, and
>> something that has already generated a lot of well-deserved flack [7].
>>
>> [7] https://lwn.net/Articles/648331/
>>
>> 3a. Short FF, internal fork with bugfixes
>>
>> Pros:
>> - Mirantis has a branch where bugfixes can be merged before they pass CI
>> and community review against OpenStack master
>> - consistent with MOS branch model
>>
>> Contras:
>> - have to duplicate community CI on internal infrastructure
>> - bugfixes have to be ported across 3 branches instead of 2
>> - cores are distracted from bugfixing to review feature commits after FF
>> - Fuel feature development remains coupled with MOS release cycle
>>
>> 3b. Short FF, internal fork with features
>>
>> Pros:
>> - Fuel release cycle is decoupled from MOS
>> - Mirantis has a branch where features and bugfixes can be merged before
>> they pass CI and community review against OpenStack master
>>
>> Contras
>> - internal fork of Fuel turns away potential contributors
>> - community Fuel may be abandoned by Mirantis before it has gathered
>> enough interest from external contributors to stand on its own
>> - have to duplicate community CI on internal infrastructure
>> - MOS release cycle stays one release behind upstream OpenStack
>>
>> 4. CI for external forks
>> ------------------------
>>
>> Finally, there's an argument that no matter how much we simplify
>> external contribution, some people will still want to stay out of tree.
>> Some want the features we're developing for Liberty based Fuel 8.0 to be
>> backported to Juno, some want their features to remain closed from the
>> community, some want to deploy a quick-and-dirty solution without going
>> through community code review.
>>
>> There is no agreement on how much we should encourage and support such
>> private forks, but one thing everyone agrees on is that working with
>> such forks without a complete copy of upstream CI is a nightmare.
>>
>> Packaging and documenting our whole CI infrastructure in a way that can
>> be brought up locally by any user of Fuel is a massive task by itself,
>> but before we even go there we should also consider all the negative
>> externalities associated with supporting this use case.
>>
>> On top of initial cost of creating a fork and backing it with CI, every
>> fork can put following burdens on the different parties in the greater
>> Fuel ecosystem:
>>
>> - one more version to confirm and triage bugs against
>> - one more set of git branches to port patches to
>> - one more set of git tags and branches to manage
>> - one more version to run manual test cases against
>> - one more version the support organization has to onboard (learn, set
>> up a test lab, re-confirm bugs against mainline)
>> - one more version to set up patching pipeline for
>>
>> More often than not, the decision to fork is made without considering
>> all these externalities, and for reasons that could have been addressed
>> without a fork. You can't stay on Juno forever and at some point will
>> have to upgrade, backporting Liberty features may temporarily relieve
>> the need to upgrade but will not reduce its cost. Proprietary extension
>> of Fuel functionality could be extracted into a plugin where it could
>> live on its own release cycle. And the quick-and-dirty production setup
>> always comes back to haunt you.
>>
>> Still, Fuel is open source and anyone can fork it. The only choice we
>> have is whether to discourage it, or spend additional effort to embrace
>> external forks and provide them with well documented CI tools.
>>
>> Pros:
>> - enables Fuel users to have their own release cycle
>> - does not require any changes to Fuel release cycle or branch model
>>
>> Cons:
>> - negative externalities for Fuel ecosystem
>> - effort to package and document Fuel CI
>>
>> Believe it or not, the above is a condensed summary of the discussions
>> we had on this topic across many different communication channels. I
>> probably have missed some of the concerns that were raised elsewhere,
>> please feel free to add them to the discussion here, but make sure you
>> do so before the end of this week.
>>
>> Thanks,
>>
>> --
>> Dmitry Borodaenko
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev