Open Stack

On Mon, Mar 2, 2015 at 9:59 AM, Kyle Mestery <mestery at mestery.com> wrote:

> On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka <ihrachys at redhat.com>
> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi Daniel,
>>
>> thanks for a clear write-up of the matter and food for thought.
>>
>> I think the idea of having more smooth development mode that would not
>> make people to wait for 6+ months to release a new feature is great.
>>
>> ++
>
>
>> It's insane to expect that feature priorities won't ever slightly
>> shift in the next 6 months. Having a limited list of features targeted
>> for the next 6 months is prone to mistakes, since people behind some
>>
>
* Sure, we have had a few things that popped up, nova EC2 split for
example. But this is fairly rare.


> of approved features may need to postpone the effort for any type of
>> reasons (personal, job rush, bad resource allocation, ...), and we
>
> then end up with approved specs with no actual code drops, using
>> review 'slots' that would better be spent for other features that were
>> not that lucky to get a rubber stamp during spec phase. Prior resource
>>
>
* As stated below specs approval is very much not rubber stamping
* As stated below this doesn't even make sense, we are *not* using review
slots.


> allocation would probably work somehow if we were working for the same
>> company that would define priorities to us, but it's not the case.
>
>
>> It should be noted that even though Nova is using slots for reviews,
> Neutron is not. I've found that it's hard to try and "slot" people in to
> review specific things. During Juno I tried this for Neutron, and it failed
> miserably. For Kilo in Neutron, we're not using slots but instead I've
> tried to highlight the approved specs of "Essential" and "High" priority
> for review by all reviews, core and non-core included. It's gone ok, but
> the reality is you can't force people to review things. There are steps
> submitters can take to try and get timely review (lots of small, easy to
> digest patches, quick turnaround of comments, engagement in IRC and ML,
> etc.).
>

So this is a big fat lie, one that others believe as well. Nova is *not*
using slots for reviews. We discussed using slots for reviews but did not
adopt them.


>
>
>> Anecdotally, in neutron, we have two Essential blueprints for Kilo,
>> and there are no code drops or patches in review for any of those, so
>> I would expect them to fail to merge. At the same time, I will need to
>> wait for the end of Kilo to consider adding support for guru reports
>> to the project. Or in oslo world, I will need to wait for Liberty to
>> introduce features in oslo.policy that are needed by neutron to switch
>> to it, etc.
>>
>> To be fair, there are many reasons those to "Essential" BPs do not have
> code. I still expect the Pecan focused to have code, but I already moved
> the Plugin one out of Kilo at this point because there was no chance the
> code would land.
>
> But I get your point here. I think this thread has highlighted the fact
> that the BP/spec process worked to some extent, but for small things, the
> core reviewer team should have the ability to say "Yes, we can easily merge
> that, lets approve that spec" even if it's late in the cycle.
>
>
>> Another problem is that currently milestones are used merely for
>> targeting bugs and features, but no one really cares about whether the
>> actual milestone shipment actually works. Again, a story from neutron
>> world: Kilo-1 was shipped in the middle of advanced services split,
>> with some essential patches around distutils setup missing (no proper
>> migration plan applied, conflicting config files in neutron and *aas
>> repos, etc.)
>>
>> This is true, the milestone release matter but are not given enough focus
> and they release (for the most part) irregardless of items in them, given
> they are not long-lived, etc.
>
> So I'm all for reforms around processes we apply.
>>
>> "If there's one thing OpenStack is good at, it's change."
>
>
>> That said, I don't believe the real problem here is that we don't
>> generate project tarballs frequently enough.
>>
>> Major problems I see as critical to tackle in our dev process are:
>>
>> - - enforced spec/dev mode. Solution: allow to propose (and approve) a
>> reasonable spec at any moment in the cycle; allow code drops for
>> approved specs at any moment in the cycle (except pre-release
>> stabilization time); stop targeting specs: if it's sane, it's probably
>> sane N+2 cycle from now too.
>>
>> I'd say this is fine for specs that are small and people agree can easily
> be merged. I'd say this is not the case for large features near the end of
> the release which are unlikely to gather enough review momentum to actually
> merge.
>
>
>> - - core team rubber stamping a random set of specs and putting -2 on
>> all other specs due to "project priorities". Solution: stop pretending
>
> core team (or core drivers) can reasonably estimate review and coding
>> resources for the next cycle. Instead allows community to decide
>> what's worth the effort by approving all technically reasonable specs
>> and allowing everyone to invest time and effort in specs (s)he seems
>> worth it.
>>
>> If you're referring to Neutron here, I think you fail to estimate the
> amount of time the neutron-drivers team (along with a handful of other
> folks) spent reviewing specs and trying to allocate them for this release.
> We're not just rubber stamping things, we're reviewing, providing comment,
> and ensuring things fit in a consistent roadmap for the next release. In
> the past, we've had this sort of "wild west" where all specs are approved,
> everyone focuses on their own stuff, and a large chunk of things fail to
> merge. Juno was a classic example of that. Now, we'll likely see a similar
> amount of that happening in Kilo despite the changes to the process. But
> I'd at least argue the process here around spec approvals is trying to
> maintain some coherency release to release.
>

Same thing is true in nova as well.


>
>
> - - no due stabilization process before dev milestones. Solution:
>> introduce one in your team workflow, be more strict in what goes in
>> during pre milestone stabilization time.
>>
>> If all above is properly applied, we would get into position similar
>> to your proposal. The difference though is that upstream project would
>> not call milestone tarballs 'Releases'. Still, if there are brave
>> vendors to ship milestones, fine, they would have the same freedom as
>> in your proposal.
>>
>>
While I don't agree with some of your points above, I think what you state
above is a good goal IMHO (In fact I proposed the same thing in a previous
thread)


> Note: all the steps mentioned above can be applied on *per team* basis
>> without breaking existing release workflow.
>>
>> I think the ideas above are worth exploring further, and I look forward
> to some discussions on this in Vancouver, as well as continued discussion
> here on the ML before then.
>


> Thanks,
> Kyle
>
>
>> Some more comments from stable-maint/distribution side below.
>>
>> On 02/24/2015 10:53 AM, Daniel P. Berrange wrote:
>> [...skip...]
>> > The modest proposal ===================
>> [...skip...]
>> >
>> > Stable branches ---------------
>> >
>> > The consequences of a 2 month release cycle appear fairly severe
>> > for the stable branch maint teams at first sight. This is not,
>> > however, an insurmountable problem. The linux kernel shows an easy
>> > way forward with their approach of only maintaining stable branches
>> > for a subset of major releases, based around user / vendor demand.
>> > So it is still entirely conceivable that the stable team only
>> > provide stable branch releases for 2 out of the 6 yearly releases.
>> > ie no additional burden over what they face today. Of course they
>> > might decide they want to do more stable branches, but maintain
>> > each for a shorter time. So I could equally see them choosing todo
>> > 3 or 4 stable branches a year. Whatever is most effective for those
>> > involved and those consuming them is fine.
>> >
>>
>> Since it's not only stable branches that are affected (translators,
>> documentation writers, VMT were already mentioned), those affected
>> will probably need to come up with some synchronized decision.
>>
>> Let's say we still decide to support two out of six releases (same
>> scheme as is now). In that case no process that we usually attach to
>> releases will be running after release dates. This makes me wonder how
>> it's different from milestones we already have.
>>
>> Do you think any downstream vendors will actually ship and support
>> upstream releases that upstream drops any guarantees for (no VMT, no
>> stable branches, no gate runs, ...) right after the release date? I
>> doubt there will be significant interest in it.
>>
>> > Downstream vendors ------------------
>> >
>> > Downstream product vendors would face a similar decision about how
>> > they wished to produce. Some OS vendors may choose to ship all
>> > stable releases in their distros as they become available, others
>> > may choose to focus on a subset of the releases. In some cases more
>> > frequent release may actually simplify life for the vendors. For
>> > example, if there is a feature they need which doesn't make the cut
>> > for a release today, they currently have the choice of waiting 6
>> > months for the next release, or carrying non-upstream patches, or
>> > backporting patches from master. With a 2 month release cycle, it
>> > is more practical for vendors to simply wait 2 months for the next
>> > release to ship. So this would probably reduce the amount of patch
>> > backporting vendors have todo in general, giving them more time to
>> > focus on useful upstream work.
>> >
>> > Design summit -------------
>> >
>> > Another consequence of 2 month release cycles would be that the
>> > design summits become entirely detached from the release cycles,
>> > as they'd be on different cycle lengths. So instead of having a
>> > focus on release planning, design summits would likely become a
>> > forum for general ongoing collaboration and discussions. Probably a
>> > half & half merge between the current design summit format and a
>> > more traditional open source conference. This also ought to result
>> > in less pressure for all contributors to attend all summits, and so
>> > be time efficient travelwise and beneficial in cost reduction for
>> > vendors.
>> >
>> > More interestingly, is that by detaching them from release cycles,
>> > the summit organizers could have more flexibility with scheduling
>> > the summits. They could be placed on the dates that are most
>> > suitable for desired travel locations, rather than dates suited to
>> > release schedules. Likewise there is more freedom if there was ever
>> > a wish to have design summits and the main track conference
>> > separated.
>> >
>> > Election process ----------------
>> >
>> > I wouldn't suggest any changes to project elections or governance.
>> > It is fine to have elections every 6 months. There is no need for
>> > that to be tied to release cycle length in any way, particularly as
>> > people often serve for more than one release already.
>> >
>> > Release naming --------------
>> >
>> > Last but not least, there would be some questions around what this
>> > might mean for release naming. There are many options and I don't
>> > intend to suggest a particular one is best, just highlight some
>> > possibilities that could be considered.
>> >
>> > The easiest is to just drop the idea of release names entirely and
>> > have everyone refer to the release versions instead. Another
>> > option is to give each 2 month release its own name, so we'd have
>> > to come up with 3x as many names during the year. The person
>> > running the release naming elections will probably have a heart
>> > attack at this idea :-) Another option is to have names that span a
>> > couple of releases, perhaps all releases for an entire year get a
>> > given name.
>> >
>> > Upgrades & deprecation ----------------------
>> >
>> > It is common right now for projects to say upgrades are only
>> > supported between releases N-1 and N. ie to go from Icehouse to
>> > Kilo, you need to first deploy Juno. This is passable when you're
>> > talking 6 month gaps between cycles, but when there are 2 month
>> > gaps it is not reasonable to expect everyone to be moving fast
>> > enough to keep up with every release. If an organization's
>> > beurocracy means they can't deploy more often than every 12 months,
>> > forcing them to deploy the 5 intermediate releases to run upgrade
>> > scripts is quite unpleasant. We would likely have to look at
>> > officially allowing upgrades between any (N-3, N-2, N-1) to N. From
>> > a database POV this should not be hard, since the DB migration
>> > scripts don't have any built in assumptions about this. Similarly
>> > the versioned objects used by Nova are quite flexible in this
>> > regard too, as long as the compat code isn't deleted too soon.
>>
>> Let's say horizontal teams decided to stick to 2 supported releases
>> per year scheme (the current one). What would grenade jobs for stable
>> branches (and master) test in this case?
>>
>> >
>> > Deprecation warnings would need similar consideration. It would not
>> > be sufficient to deprecate in one release and delete in the next.
>> > We'd likely want to say that depecations last for a given time
>> > period rather than number of releases, eg 6 months. This could be
>> > easily handled by simply including the date of initial deprecation
>> > in the deprecation message. It would thus be clear when the feature
>> > will be removed to all involved.
>> >
>> > Why do releases at all ? ------------------------
>> >
>> > Some people might wonder why we should do releases at all. We
>> > already aim to support a continuous deployment model for users as
>> > an option. Continuous deployment, however, is quite a conceptual
>> > jump for many users and organizations in particular. Their internal
>> > software approval processes and bureaucracy can make this
>> > impractical to even consider in the forseeable future. So if
>> > openstack did not do releases, then many downstream vendors would
>> > still end up taking git snapshots and forming periodic releases
>> > around them. The unsatisfactory result would be a balkanization of
>> > OpenStack where each vendor duplicated the others work with their
>> > own release cycles in slightly different ways. To some extent this
>> > happens already as vendors backport features, due to inability to
>> > wait 6 months for next major release. A two month release cycle
>> > though would likely reduce the amount of difference between what
>> > each vendor ships which would be a good thing in general. Releases
>> > also serve as a useful hook for marketing activities by the project
>> > and vendors alike, giving a good opportunity to promote the new
>> > body of features now available to users. It might be difficult to
>> > drum up media enthusiasm to promote arbitrary git snapshots, but
>> > then I'm no marketing expert.
>> >
>>
>> I think it's still easier for people to work in release/stability mode
>> for some predefined time than "forever".
>>
>> >
>> >
>> >
>> > Regards, Daniel
>> >
>> > [1]
>> >
>> http://lists.openstack.org/pipermail/openstack-dev/2014-September/044872.html
>> >
>> >
>> [2] http://www.openstack.org/projects/openstack-faq/
>> > [3] https://wiki.openstack.org/wiki/Kilo_Release_Schedule [4]
>> >
>> http://lists.openstack.org/pipermail/openstack-dev/2014-October/047922.html
>> >
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1
>>
>> iQEbBAEBAgAGBQJU9IheAAoJEC5aWaUY1u57/M4H912jtcA8IP424DZnkDVm/6dF
>> uXl02JqeUeFvne+M9/JoRIO+GwuY05kfmwFAXY5RNX5PmW1MA+eLpirKI5waMjzJ
>> 6zAwViir2sLtrXjpHNKkUrQxzURWN72qvnU+idCJlExU4ZXCkKcTuGo0YC6j46WH
>> mLuRheb3wRqmkMrZ7nlShaAiSkLBYpslP+NuKH4JuWzyAZqrQsTwcu9duOD9QiTR
>> 294ojzqihCth7npMeXcdMtdkl+GoxP494ojouWqVTAYCpWDfTiZw3/Slb0VTYVIe
>> uIS1gN2Hmur1vTbkM4kgvU3D9Uf+uLS9vH9GDg+zyDsjQQm/6RToCOTFiAYp8w==
>> =nhsu
>> -----END PGP SIGNATURE-----
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150302/43784195/attachment.html>