Open Stack

On Mon, Mar 2, 2015 at 3:38 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:

>
>
> On Mon, Mar 2, 2015 at 9:59 AM, Kyle Mestery <mestery at mestery.com> wrote:
>
>> On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka <ihrachys at redhat.com>
>> wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Hi Daniel,
>>>
>>> thanks for a clear write-up of the matter and food for thought.
>>>
>>> I think the idea of having more smooth development mode that would not
>>> make people to wait for 6+ months to release a new feature is great.
>>>
>>> ++
>>
>>
>>> It's insane to expect that feature priorities won't ever slightly
>>> shift in the next 6 months. Having a limited list of features targeted
>>> for the next 6 months is prone to mistakes, since people behind some
>>>
>>
> * Sure, we have had a few things that popped up, nova EC2 split for
> example. But this is fairly rare.
>
>
>> of approved features may need to postpone the effort for any type of
>>> reasons (personal, job rush, bad resource allocation, ...), and we
>>
>> then end up with approved specs with no actual code drops, using
>>> review 'slots' that would better be spent for other features that were
>>> not that lucky to get a rubber stamp during spec phase. Prior resource
>>>
>>
> * As stated below specs approval is very much not rubber stamping
> * As stated below this doesn't even make sense, we are *not* using review
> slots.
>
>
>> allocation would probably work somehow if we were working for the same
>>> company that would define priorities to us, but it's not the case.
>>
>>
>>> It should be noted that even though Nova is using slots for reviews,
>> Neutron is not. I've found that it's hard to try and "slot" people in to
>> review specific things. During Juno I tried this for Neutron, and it failed
>> miserably. For Kilo in Neutron, we're not using slots but instead I've
>> tried to highlight the approved specs of "Essential" and "High" priority
>> for review by all reviews, core and non-core included. It's gone ok, but
>> the reality is you can't force people to review things. There are steps
>> submitters can take to try and get timely review (lots of small, easy to
>> digest patches, quick turnaround of comments, engagement in IRC and ML,
>> etc.).
>>
>
> So this is a big fat lie, one that others believe as well. Nova is *not*
> using slots for reviews. We discussed using slots for reviews but did not
> adopt them.
>
>
"But I read it on the internet, it must be true."

As I said in a prior email, I'm sorry for that. I recalled reading about
nova's use of slots.



>
>>
>>> Anecdotally, in neutron, we have two Essential blueprints for Kilo,
>>> and there are no code drops or patches in review for any of those, so
>>> I would expect them to fail to merge. At the same time, I will need to
>>> wait for the end of Kilo to consider adding support for guru reports
>>> to the project. Or in oslo world, I will need to wait for Liberty to
>>> introduce features in oslo.policy that are needed by neutron to switch
>>> to it, etc.
>>>
>>> To be fair, there are many reasons those to "Essential" BPs do not have
>> code. I still expect the Pecan focused to have code, but I already moved
>> the Plugin one out of Kilo at this point because there was no chance the
>> code would land.
>>
>> But I get your point here. I think this thread has highlighted the fact
>> that the BP/spec process worked to some extent, but for small things, the
>> core reviewer team should have the ability to say "Yes, we can easily merge
>> that, lets approve that spec" even if it's late in the cycle.
>>
>>
>>> Another problem is that currently milestones are used merely for
>>> targeting bugs and features, but no one really cares about whether the
>>> actual milestone shipment actually works. Again, a story from neutron
>>> world: Kilo-1 was shipped in the middle of advanced services split,
>>> with some essential patches around distutils setup missing (no proper
>>> migration plan applied, conflicting config files in neutron and *aas
>>> repos, etc.)
>>>
>>> This is true, the milestone release matter but are not given enough
>> focus and they release (for the most part) irregardless of items in them,
>> given they are not long-lived, etc.
>>
>> So I'm all for reforms around processes we apply.
>>>
>>> "If there's one thing OpenStack is good at, it's change."
>>
>>
>>> That said, I don't believe the real problem here is that we don't
>>> generate project tarballs frequently enough.
>>>
>>> Major problems I see as critical to tackle in our dev process are:
>>>
>>> - - enforced spec/dev mode. Solution: allow to propose (and approve) a
>>> reasonable spec at any moment in the cycle; allow code drops for
>>> approved specs at any moment in the cycle (except pre-release
>>> stabilization time); stop targeting specs: if it's sane, it's probably
>>> sane N+2 cycle from now too.
>>>
>>> I'd say this is fine for specs that are small and people agree can
>> easily be merged. I'd say this is not the case for large features near the
>> end of the release which are unlikely to gather enough review momentum to
>> actually merge.
>>
>>
>>> - - core team rubber stamping a random set of specs and putting -2 on
>>> all other specs due to "project priorities". Solution: stop pretending
>>
>> core team (or core drivers) can reasonably estimate review and coding
>>> resources for the next cycle. Instead allows community to decide
>>> what's worth the effort by approving all technically reasonable specs
>>> and allowing everyone to invest time and effort in specs (s)he seems
>>> worth it.
>>>
>>> If you're referring to Neutron here, I think you fail to estimate the
>> amount of time the neutron-drivers team (along with a handful of other
>> folks) spent reviewing specs and trying to allocate them for this release.
>> We're not just rubber stamping things, we're reviewing, providing comment,
>> and ensuring things fit in a consistent roadmap for the next release. In
>> the past, we've had this sort of "wild west" where all specs are approved,
>> everyone focuses on their own stuff, and a large chunk of things fail to
>> merge. Juno was a classic example of that. Now, we'll likely see a similar
>> amount of that happening in Kilo despite the changes to the process. But
>> I'd at least argue the process here around spec approvals is trying to
>> maintain some coherency release to release.
>>
>
> Same thing is true in nova as well.
>
>
>>
>>
>> - - no due stabilization process before dev milestones. Solution:
>>> introduce one in your team workflow, be more strict in what goes in
>>> during pre milestone stabilization time.
>>>
>>> If all above is properly applied, we would get into position similar
>>> to your proposal. The difference though is that upstream project would
>>> not call milestone tarballs 'Releases'. Still, if there are brave
>>> vendors to ship milestones, fine, they would have the same freedom as
>>> in your proposal.
>>>
>>>
> While I don't agree with some of your points above, I think what you state
> above is a good goal IMHO (In fact I proposed the same thing in a previous
> thread)
>
>
>> Note: all the steps mentioned above can be applied on *per team* basis
>>> without breaking existing release workflow.
>>>
>>> I think the ideas above are worth exploring further, and I look forward
>> to some discussions on this in Vancouver, as well as continued discussion
>> here on the ML before then.
>>
>
>
>> Thanks,
>> Kyle
>>
>>
>>> Some more comments from stable-maint/distribution side below.
>>>
>>> On 02/24/2015 10:53 AM, Daniel P. Berrange wrote:
>>> [...skip...]
>>> > The modest proposal ===================
>>> [...skip...]
>>> >
>>> > Stable branches ---------------
>>> >
>>> > The consequences of a 2 month release cycle appear fairly severe
>>> > for the stable branch maint teams at first sight. This is not,
>>> > however, an insurmountable problem. The linux kernel shows an easy
>>> > way forward with their approach of only maintaining stable branches
>>> > for a subset of major releases, based around user / vendor demand.
>>> > So it is still entirely conceivable that the stable team only
>>> > provide stable branch releases for 2 out of the 6 yearly releases.
>>> > ie no additional burden over what they face today. Of course they
>>> > might decide they want to do more stable branches, but maintain
>>> > each for a shorter time. So I could equally see them choosing todo
>>> > 3 or 4 stable branches a year. Whatever is most effective for those
>>> > involved and those consuming them is fine.
>>> >
>>>
>>> Since it's not only stable branches that are affected (translators,
>>> documentation writers, VMT were already mentioned), those affected
>>> will probably need to come up with some synchronized decision.
>>>
>>> Let's say we still decide to support two out of six releases (same
>>> scheme as is now). In that case no process that we usually attach to
>>> releases will be running after release dates. This makes me wonder how
>>> it's different from milestones we already have.
>>>
>>> Do you think any downstream vendors will actually ship and support
>>> upstream releases that upstream drops any guarantees for (no VMT, no
>>> stable branches, no gate runs, ...) right after the release date? I
>>> doubt there will be significant interest in it.
>>>
>>> > Downstream vendors ------------------
>>> >
>>> > Downstream product vendors would face a similar decision about how
>>> > they wished to produce. Some OS vendors may choose to ship all
>>> > stable releases in their distros as they become available, others
>>> > may choose to focus on a subset of the releases. In some cases more
>>> > frequent release may actually simplify life for the vendors. For
>>> > example, if there is a feature they need which doesn't make the cut
>>> > for a release today, they currently have the choice of waiting 6
>>> > months for the next release, or carrying non-upstream patches, or
>>> > backporting patches from master. With a 2 month release cycle, it
>>> > is more practical for vendors to simply wait 2 months for the next
>>> > release to ship. So this would probably reduce the amount of patch
>>> > backporting vendors have todo in general, giving them more time to
>>> > focus on useful upstream work.
>>> >
>>> > Design summit -------------
>>> >
>>> > Another consequence of 2 month release cycles would be that the
>>> > design summits become entirely detached from the release cycles,
>>> > as they'd be on different cycle lengths. So instead of having a
>>> > focus on release planning, design summits would likely become a
>>> > forum for general ongoing collaboration and discussions. Probably a
>>> > half & half merge between the current design summit format and a
>>> > more traditional open source conference. This also ought to result
>>> > in less pressure for all contributors to attend all summits, and so
>>> > be time efficient travelwise and beneficial in cost reduction for
>>> > vendors.
>>> >
>>> > More interestingly, is that by detaching them from release cycles,
>>> > the summit organizers could have more flexibility with scheduling
>>> > the summits. They could be placed on the dates that are most
>>> > suitable for desired travel locations, rather than dates suited to
>>> > release schedules. Likewise there is more freedom if there was ever
>>> > a wish to have design summits and the main track conference
>>> > separated.
>>> >
>>> > Election process ----------------
>>> >
>>> > I wouldn't suggest any changes to project elections or governance.
>>> > It is fine to have elections every 6 months. There is no need for
>>> > that to be tied to release cycle length in any way, particularly as
>>> > people often serve for more than one release already.
>>> >
>>> > Release naming --------------
>>> >
>>> > Last but not least, there would be some questions around what this
>>> > might mean for release naming. There are many options and I don't
>>> > intend to suggest a particular one is best, just highlight some
>>> > possibilities that could be considered.
>>> >
>>> > The easiest is to just drop the idea of release names entirely and
>>> > have everyone refer to the release versions instead. Another
>>> > option is to give each 2 month release its own name, so we'd have
>>> > to come up with 3x as many names during the year. The person
>>> > running the release naming elections will probably have a heart
>>> > attack at this idea :-) Another option is to have names that span a
>>> > couple of releases, perhaps all releases for an entire year get a
>>> > given name.
>>> >
>>> > Upgrades & deprecation ----------------------
>>> >
>>> > It is common right now for projects to say upgrades are only
>>> > supported between releases N-1 and N. ie to go from Icehouse to
>>> > Kilo, you need to first deploy Juno. This is passable when you're
>>> > talking 6 month gaps between cycles, but when there are 2 month
>>> > gaps it is not reasonable to expect everyone to be moving fast
>>> > enough to keep up with every release. If an organization's
>>> > beurocracy means they can't deploy more often than every 12 months,
>>> > forcing them to deploy the 5 intermediate releases to run upgrade
>>> > scripts is quite unpleasant. We would likely have to look at
>>> > officially allowing upgrades between any (N-3, N-2, N-1) to N. From
>>> > a database POV this should not be hard, since the DB migration
>>> > scripts don't have any built in assumptions about this. Similarly
>>> > the versioned objects used by Nova are quite flexible in this
>>> > regard too, as long as the compat code isn't deleted too soon.
>>>
>>> Let's say horizontal teams decided to stick to 2 supported releases
>>> per year scheme (the current one). What would grenade jobs for stable
>>> branches (and master) test in this case?
>>>
>>> >
>>> > Deprecation warnings would need similar consideration. It would not
>>> > be sufficient to deprecate in one release and delete in the next.
>>> > We'd likely want to say that depecations last for a given time
>>> > period rather than number of releases, eg 6 months. This could be
>>> > easily handled by simply including the date of initial deprecation
>>> > in the deprecation message. It would thus be clear when the feature
>>> > will be removed to all involved.
>>> >
>>> > Why do releases at all ? ------------------------
>>> >
>>> > Some people might wonder why we should do releases at all. We
>>> > already aim to support a continuous deployment model for users as
>>> > an option. Continuous deployment, however, is quite a conceptual
>>> > jump for many users and organizations in particular. Their internal
>>> > software approval processes and bureaucracy can make this
>>> > impractical to even consider in the forseeable future. So if
>>> > openstack did not do releases, then many downstream vendors would
>>> > still end up taking git snapshots and forming periodic releases
>>> > around them. The unsatisfactory result would be a balkanization of
>>> > OpenStack where each vendor duplicated the others work with their
>>> > own release cycles in slightly different ways. To some extent this
>>> > happens already as vendors backport features, due to inability to
>>> > wait 6 months for next major release. A two month release cycle
>>> > though would likely reduce the amount of difference between what
>>> > each vendor ships which would be a good thing in general. Releases
>>> > also serve as a useful hook for marketing activities by the project
>>> > and vendors alike, giving a good opportunity to promote the new
>>> > body of features now available to users. It might be difficult to
>>> > drum up media enthusiasm to promote arbitrary git snapshots, but
>>> > then I'm no marketing expert.
>>> >
>>>
>>> I think it's still easier for people to work in release/stability mode
>>> for some predefined time than "forever".
>>>
>>> >
>>> >
>>> >
>>> > Regards, Daniel
>>> >
>>> > [1]
>>> >
>>> http://lists.openstack.org/pipermail/openstack-dev/2014-September/044872.html
>>> >
>>> >
>>> [2] http://www.openstack.org/projects/openstack-faq/
>>> > [3] https://wiki.openstack.org/wiki/Kilo_Release_Schedule [4]
>>> >
>>> http://lists.openstack.org/pipermail/openstack-dev/2014-October/047922.html
>>> >
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1
>>>
>>> iQEbBAEBAgAGBQJU9IheAAoJEC5aWaUY1u57/M4H912jtcA8IP424DZnkDVm/6dF
>>> uXl02JqeUeFvne+M9/JoRIO+GwuY05kfmwFAXY5RNX5PmW1MA+eLpirKI5waMjzJ
>>> 6zAwViir2sLtrXjpHNKkUrQxzURWN72qvnU+idCJlExU4ZXCkKcTuGo0YC6j46WH
>>> mLuRheb3wRqmkMrZ7nlShaAiSkLBYpslP+NuKH4JuWzyAZqrQsTwcu9duOD9QiTR
>>> 294ojzqihCth7npMeXcdMtdkl+GoxP494ojouWqVTAYCpWDfTiZw3/Slb0VTYVIe
>>> uIS1gN2Hmur1vTbkM4kgvU3D9Uf+uLS9vH9GDg+zyDsjQQm/6RToCOTFiAYp8w==
>>> =nhsu
>>> -----END PGP SIGNATURE-----
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150302/57a96426/attachment.html>