Open Stack

Dmitry,

thank you for a well described plan.
May I please ask you for a little TL;DR excerpt of what’s going to be changed, because I’m affraid some folks may get lost or may not have enough time to analyze it deeply (I actually see that happening but I won’t do fingerpointing :) ).


- romcheg

> 24 серп. 2015 о 20:29 Dmitry Borodaenko <dborodaenko at mirantis.com> написав(ла):
> 
> We have several problems with Fuel branching strategy that have become
> enough of a bottleneck to consider re-thinking our whole release
> management process. In this email, I will describe the problems, propose
> a couple of alternative strategies, and try to analyze their relative
> merits and associated risks.
> 
> I have my opinions and preferences, but I will try my best to
> objectively compare all available options. My goal is to improve
> efficiency of the existing team and make it significantly easier for new
> people to contribute to Fuel, even when their schedule and their agenda
> is not 100% aligned with those of Fuel core team and Mirantis OpenStack.
> 
> It is essential for the new process to be acceptable for Fuel
> contributors, and is is just as essential to reach a consensus quickly:
> with Fuel 7.0 Hard Code Freeze less than two weeks away [0], we're
> already late with planning 8.0, and we absolutely must have the whole
> plan for 8.0 before Fuel 7.0 is released (September 24).
> 
> [0] https://wiki.openstack.org/wiki/Fuel/7.0_Release_Schedule
> 
> I propose a time-bound rough consensus to make this decision: raise all
> concerns and risks before end of this week (Friday, August 28), propose
> and discuss ways to address the raised concerns, and make final decision
> before end of next week (Friday, September 4).
> 
> Here's how Fuel versions, branches, and release milestones work now.
> 
> Major Fuel version corresponds to an OpenStack release. For example,
> Fuel 7.0 is the first version to support Kilo. Minor version denotes a
> feature release. For example, Fuel 6.1 is still based on Juno but
> contains new features relative to Fuel 6.0. Tiny versions (e.g. 5.1.1)
> and maintenance updates (e.g. 6.1-mu-1) include only bugfixes.
> 
> Most Fuel development happens on the master branch. On Hard Code Freeze,
> a stable branch is created in all Fuel git repositories (e.g. stable/7.0
> will be created on September 3). After that, all changes targeted to
> that release series must be proposed, reviewed, and merged to master
> before they are proposed for any stable branches [1]. One release series
> and one stable branch is created per minor Fuel version (e.g. stable/6.1
> for 6.1.x).
> 
> [1] https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Backport_bugfixes_to_stable_release_series
> 
> Fuel release cycle starts from Hard Code freeze of the previous release:
> 
> - Hard Code Freeze: stable branch created, master branch is open for all
> changes
> - Design Complete: all release features have specs merged in fuel-specs
> - Feature Freeze: feature changes no longer accepted in master branch
> - Soft Code Freeze: medium and lower priority bugfixes no longer
> accepted in master branch
> 
> Looks similar to the way OpenStack branch model works [2], but there's
> an important difference.
> 
> [2] https://wiki.openstack.org/wiki/Branch_Model
> 
> In OpenStack, master branch is closed for feature commits for 2 weeks
> per release [3], or 1/13th of the whole 26-week release cycle.
> 
> [3] https://wiki.openstack.org/wiki/Liberty_Release_Schedule
> 
> In Fuel, master branch remains closed for almost half of the release
> cycle:
> 
> 5.0 -- 32 of 81 days
> 5.1 -- 63 of 119 days
> 6.0 -- 35 of 93 days
> 6.1 -- 85 of 180 days
> 7.0 -- 42 of 93 days (planned)
> 
> This renders it unusable as an integration branch: if you are bound by a
> schedule that is not aligned with Fuel release milestones and have more
> changes than a single commit which you could keep rebasing until Fuel
> master is open again, you're better off merging your changes into your
> own integration branch (i.e. fork Fuel).
> 
> The same problem is even worse if you're working on the next OpenStack
> release. Even when Fuel master is open, it's developed and tested
> against latest stable release of OpenStack. For example, even though
> OpenStack developers started working on Liberty features in May,
> reflecting that work in Fuel master is blocked until September.
> 
> There are 4 partially overlapping solutions to this problem:
> 
> 1) Future branch: create a "future" integration branch on FF, rebase it
> weekly onto master (or merge weekly from master to future), merge future
> to master after stable branch is created on HCF.
> 
> 2) Short FF: create stable branch 2 weeks after FF (same as OpenStack)
> instead of waiting for HCF.
> 
> 3) Short FF and internal fork: create stable branch 2 weeks after FF,
> create an internal fork of stable branch for Mirantis OpenStack.
> 
> 4) CI for external forks: package and document Fuel development
> infrastructure so that anyone who wants to fork Fuel can set up their
> own CI.
> 
> In theory, we could implement all four, none of them, or come up with a
> fifth option. But before we vote, we should identify all possible risks
> and concerns, so that voting is based on facts more than on personal
> preferences.
> 
> 1. Future branch
> ----------------
> 
> This option was originally proposed by Vladimir Kuklin as a way to
> maintain continuous support of OpenStack master branch outside of Fuel
> master branch.
> 
> When I first came to Mike Scherbakov with a proposal to create
> stable/7.0 early to unblock feature work for Fuel 8.0, his biggest
> concern was the overhead of backporting bugfixing commits from master:
> it's negligible for the dozen or so bugs we fix between HCF and GA, but
> may add up to a lot for the hundreds of bugs we typically fix between FF
> and HCF. As an alternative that mitigates this risk, I've recycled
> Vladimir's proposal.
> 
> Eventually, Mike has realized that the confusion of targeting different
> branches during different stages of Fuel release cycle, on top of the
> contentious question of who's supposed to do the rebase work, adds up to
> just as much overhead and risk, so he has withdrawn his support for this
> option in favor of short FF [4].
> 
> [4] http://eavesdrop.openstack.org/meetings/fuel/2015/fuel.2015-08-20-16.01.log.html#l-221
> 
> While it might have worked as a one-off solution during 7.0 FF if we had
> done it in mid-August (and even by then, it was already too late for the
> more intrusive options), by the time we conclude this discussion it will
> be too late to implement it for 7.0. As a long-term solution for 8.0 and
> onward, it certainly looks less attractive than the alternatives. Still,
> I'm including it in the list so that we consider this option now and
> don't have to come back to it again later.
> 
> Pros:
> - does not require changes to Fuel release cycle
> - puts overhead on feature developers who are motivated to deal with it
> - keeps overhead away from low-motivation and high-risk bugfixing work
> 
> Contras:
> - different target branch for feature commits during FF adds confusion
> - backporting bugfix commits one at a time scales better than rebasing a
> whole integration branch
> - cores are distracted from bugfixing to review feature commits after FF
> 
> 2. Short FF
> -----------
> 
> Long feature freeze is not an oversight in the Fuel release cycle, it
> was a conscious decision intended to encourage a small team of Fuel
> developers to put feature work on hold and focus on bugs. Since 2013,
> the number of active Fuel contributors has grown from 30 to over 230
> people [5][6]. At this scale, bugfixing is less of a bottleneck than
> integrating external contributions, and it looks like the long feature
> freeze has outlived its usefulness.
> 
> [5] http://stackalytics.com/?project_type=stackforge&module=fuel-group&release=havana
> [6] http://stackalytics.com/?project_type=stackforge&module=fuel-group&release=liberty
> 
> Following the same FF process as OpenStack is also a benefit in itself:
> since Fuel versions are coupled with OpenStack releases, it makes sense
> for Fuel to follow the same release cycle. We won't be able to do this
> in 8.0 (Liberty FF is only 1 week after Fuel 7.0 GA), but if we open
> stable/8.0 early enough in the Mitaka cycle, having 9.0 FF at the same
> time as Mitaka FF becomes possible.
> 
> There was some ambiguity in our internal discussions about how short
> exactly the FF is supposed to become (create stable branch on FF or 2
> weeks after FF), for the sake of simplicity I propose to stick to the
> OpenStack process (2 weeks after FF), and also to declare SCF at the
> same time (so that we don't have to deal with backports of Medium
> priority bugs from master to stable).
> 
> Pros:
> - no changes in branch naming and bugfix backporting rules
> - consistent with OpenStack release cycle
> - backporting bugfix commits one at a time scales better than rebasing a
> whole integration branch
> 
> Contras:
> - adds overhead to the biggest release scheduling risk: bugfixing
> - cores are distracted from bugfixing to review feature commits after FF
> - merging commits becomes dependent on community code review and
> stability of OpenStack master CI
> 
> 3. Short FF and internal fork
> -----------------------------
> 
> Igor Marnat has proposed to go one step further and replicate the branch
> model we use for OpenStack components in Mirantis OpenStack: for each
> MOS release, create an internal branch from upstream stable branch.
> 
> The motivation is to completely decouple Fuel release cycle from
> Mirantis OpenStack, and to remove community code review and stability of
> OpenStack master CI from factors limiting our ability to merge commits
> needed by MOS.
> 
> Igor did not explicitly state whether the internal Mirantis branch of
> Fuel would take only bugfixes, or feature work, too. In option #2, I
> removed ambiguity by eliminating the higher risk variants. Here, the
> variants are fundamentally different and deserve an independent
> consideration.
> 
> If we use the internal branch only for bugfixes, it does not decouple
> MOS release cycle from Fuel: either MOS release cycle aligns with that
> of upstream OpenStack, and we develop new Fuel features for MOS against
> OpenStack master, or Fuel release cycle remains aligned with MOS, and we
> develop new Fuel features against OpenStack stable branch.
> 
> If we use the internal branch to develop new Fuel features based on a
> stable branch of OpenStack, we're creating a fork of community version
> of Fuel, something we're not doing with other OpenStack projects, and
> something that has already generated a lot of well-deserved flack [7].
> 
> [7] https://lwn.net/Articles/648331/
> 
> 3a. Short FF, internal fork with bugfixes
> 
> Pros:
> - Mirantis has a branch where bugfixes can be merged before they pass CI
> and community review against OpenStack master
> - consistent with MOS branch model
> 
> Contras:
> - have to duplicate community CI on internal infrastructure
> - bugfixes have to be ported across 3 branches instead of 2
> - cores are distracted from bugfixing to review feature commits after FF
> - Fuel feature development remains coupled with MOS release cycle
> 
> 3b. Short FF, internal fork with features
> 
> Pros:
> - Fuel release cycle is decoupled from MOS
> - Mirantis has a branch where features and bugfixes can be merged before
> they pass CI and community review against OpenStack master
> 
> Contras
> - internal fork of Fuel turns away potential contributors
> - community Fuel may be abandoned by Mirantis before it has gathered
> enough interest from external contributors to stand on its own
> - have to duplicate community CI on internal infrastructure
> - MOS release cycle stays one release behind upstream OpenStack
> 
> 4. CI for external forks
> ------------------------
> 
> Finally, there's an argument that no matter how much we simplify
> external contribution, some people will still want to stay out of tree.
> Some want the features we're developing for Liberty based Fuel 8.0 to be
> backported to Juno, some want their features to remain closed from the
> community, some want to deploy a quick-and-dirty solution without going
> through community code review.
> 
> There is no agreement on how much we should encourage and support such
> private forks, but one thing everyone agrees on is that working with
> such forks without a complete copy of upstream CI is a nightmare.
> 
> Packaging and documenting our whole CI infrastructure in a way that can
> be brought up locally by any user of Fuel is a massive task by itself,
> but before we even go there we should also consider all the negative
> externalities associated with supporting this use case.
> 
> On top of initial cost of creating a fork and backing it with CI, every
> fork can put following burdens on the different parties in the greater
> Fuel ecosystem:
> 
> - one more version to confirm and triage bugs against
> - one more set of git branches to port patches to
> - one more set of git tags and branches to manage
> - one more version to run manual test cases against
> - one more version the support organization has to onboard (learn, set
> up a test lab, re-confirm bugs against mainline)
> - one more version to set up patching pipeline for
> 
> More often than not, the decision to fork is made without considering
> all these externalities, and for reasons that could have been addressed
> without a fork. You can't stay on Juno forever and at some point will
> have to upgrade, backporting Liberty features may temporarily relieve
> the need to upgrade but will not reduce its cost. Proprietary extension
> of Fuel functionality could be extracted into a plugin where it could
> live on its own release cycle. And the quick-and-dirty production setup
> always comes back to haunt you.
> 
> Still, Fuel is open source and anyone can fork it. The only choice we
> have is whether to discourage it, or spend additional effort to embrace
> external forks and provide them with well documented CI tools.
> 
> Pros:
> - enables Fuel users to have their own release cycle
> - does not require any changes to Fuel release cycle or branch model
> 
> Cons:
> - negative externalities for Fuel ecosystem
> - effort to package and document Fuel CI
> 
> Believe it or not, the above is a condensed summary of the discussions
> we had on this topic across many different communication channels. I
> probably have missed some of the concerns that were raised elsewhere,
> please feel free to add them to the discussion here, but make sure you
> do so before the end of this week.
> 
> Thanks,
> 
> --
> Dmitry Borodaenko
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150825/bda5c6b1/attachment.pgp>