Open Stack

On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka <ihrachys at redhat.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Daniel,
>
> thanks for a clear write-up of the matter and food for thought.
>
> I think the idea of having more smooth development mode that would not
> make people to wait for 6+ months to release a new feature is great.
>
> ++


> It's insane to expect that feature priorities won't ever slightly
> shift in the next 6 months. Having a limited list of features targeted
> for the next 6 months is prone to mistakes, since people behind some
> of approved features may need to postpone the effort for any type of
> reasons (personal, job rush, bad resource allocation, ...), and we
> then end up with approved specs with no actual code drops, using
> review 'slots' that would better be spent for other features that were
> not that lucky to get a rubber stamp during spec phase. Prior resource
> allocation would probably work somehow if we were working for the same
> company that would define priorities to us, but it's not the case.
>
> It should be noted that even though Nova is using slots for reviews,
Neutron is not. I've found that it's hard to try and "slot" people in to
review specific things. During Juno I tried this for Neutron, and it failed
miserably. For Kilo in Neutron, we're not using slots but instead I've
tried to highlight the approved specs of "Essential" and "High" priority
for review by all reviews, core and non-core included. It's gone ok, but
the reality is you can't force people to review things. There are steps
submitters can take to try and get timely review (lots of small, easy to
digest patches, quick turnaround of comments, engagement in IRC and ML,
etc.).


> Anecdotally, in neutron, we have two Essential blueprints for Kilo,
> and there are no code drops or patches in review for any of those, so
> I would expect them to fail to merge. At the same time, I will need to
> wait for the end of Kilo to consider adding support for guru reports
> to the project. Or in oslo world, I will need to wait for Liberty to
> introduce features in oslo.policy that are needed by neutron to switch
> to it, etc.
>
> To be fair, there are many reasons those to "Essential" BPs do not have
code. I still expect the Pecan focused to have code, but I already moved
the Plugin one out of Kilo at this point because there was no chance the
code would land.

But I get your point here. I think this thread has highlighted the fact
that the BP/spec process worked to some extent, but for small things, the
core reviewer team should have the ability to say "Yes, we can easily merge
that, lets approve that spec" even if it's late in the cycle.


> Another problem is that currently milestones are used merely for
> targeting bugs and features, but no one really cares about whether the
> actual milestone shipment actually works. Again, a story from neutron
> world: Kilo-1 was shipped in the middle of advanced services split,
> with some essential patches around distutils setup missing (no proper
> migration plan applied, conflicting config files in neutron and *aas
> repos, etc.)
>
> This is true, the milestone release matter but are not given enough focus
and they release (for the most part) irregardless of items in them, given
they are not long-lived, etc.

So I'm all for reforms around processes we apply.
>
> "If there's one thing OpenStack is good at, it's change."


> That said, I don't believe the real problem here is that we don't
> generate project tarballs frequently enough.
>
> Major problems I see as critical to tackle in our dev process are:
>
> - - enforced spec/dev mode. Solution: allow to propose (and approve) a
> reasonable spec at any moment in the cycle; allow code drops for
> approved specs at any moment in the cycle (except pre-release
> stabilization time); stop targeting specs: if it's sane, it's probably
> sane N+2 cycle from now too.
>
> I'd say this is fine for specs that are small and people agree can easily
be merged. I'd say this is not the case for large features near the end of
the release which are unlikely to gather enough review momentum to actually
merge.


> - - core team rubber stamping a random set of specs and putting -2 on
> all other specs due to "project priorities". Solution: stop pretending
> core team (or core drivers) can reasonably estimate review and coding
> resources for the next cycle. Instead allows community to decide
> what's worth the effort by approving all technically reasonable specs
> and allowing everyone to invest time and effort in specs (s)he seems
> worth it.
>
> If you're referring to Neutron here, I think you fail to estimate the
amount of time the neutron-drivers team (along with a handful of other
folks) spent reviewing specs and trying to allocate them for this release.
We're not just rubber stamping things, we're reviewing, providing comment,
and ensuring things fit in a consistent roadmap for the next release. In
the past, we've had this sort of "wild west" where all specs are approved,
everyone focuses on their own stuff, and a large chunk of things fail to
merge. Juno was a classic example of that. Now, we'll likely see a similar
amount of that happening in Kilo despite the changes to the process. But
I'd at least argue the process here around spec approvals is trying to
maintain some coherency release to release.


- - no due stabilization process before dev milestones. Solution:
> introduce one in your team workflow, be more strict in what goes in
> during pre milestone stabilization time.
>
> If all above is properly applied, we would get into position similar
> to your proposal. The difference though is that upstream project would
> not call milestone tarballs 'Releases'. Still, if there are brave
> vendors to ship milestones, fine, they would have the same freedom as
> in your proposal.
>
> Note: all the steps mentioned above can be applied on *per team* basis
> without breaking existing release workflow.
>
> I think the ideas above are worth exploring further, and I look forward to
some discussions on this in Vancouver, as well as continued discussion here
on the ML before then.

Thanks,
Kyle


> Some more comments from stable-maint/distribution side below.
>
> On 02/24/2015 10:53 AM, Daniel P. Berrange wrote:
> [...skip...]
> > The modest proposal ===================
> [...skip...]
> >
> > Stable branches ---------------
> >
> > The consequences of a 2 month release cycle appear fairly severe
> > for the stable branch maint teams at first sight. This is not,
> > however, an insurmountable problem. The linux kernel shows an easy
> > way forward with their approach of only maintaining stable branches
> > for a subset of major releases, based around user / vendor demand.
> > So it is still entirely conceivable that the stable team only
> > provide stable branch releases for 2 out of the 6 yearly releases.
> > ie no additional burden over what they face today. Of course they
> > might decide they want to do more stable branches, but maintain
> > each for a shorter time. So I could equally see them choosing todo
> > 3 or 4 stable branches a year. Whatever is most effective for those
> > involved and those consuming them is fine.
> >
>
> Since it's not only stable branches that are affected (translators,
> documentation writers, VMT were already mentioned), those affected
> will probably need to come up with some synchronized decision.
>
> Let's say we still decide to support two out of six releases (same
> scheme as is now). In that case no process that we usually attach to
> releases will be running after release dates. This makes me wonder how
> it's different from milestones we already have.
>
> Do you think any downstream vendors will actually ship and support
> upstream releases that upstream drops any guarantees for (no VMT, no
> stable branches, no gate runs, ...) right after the release date? I
> doubt there will be significant interest in it.
>
> > Downstream vendors ------------------
> >
> > Downstream product vendors would face a similar decision about how
> > they wished to produce. Some OS vendors may choose to ship all
> > stable releases in their distros as they become available, others
> > may choose to focus on a subset of the releases. In some cases more
> > frequent release may actually simplify life for the vendors. For
> > example, if there is a feature they need which doesn't make the cut
> > for a release today, they currently have the choice of waiting 6
> > months for the next release, or carrying non-upstream patches, or
> > backporting patches from master. With a 2 month release cycle, it
> > is more practical for vendors to simply wait 2 months for the next
> > release to ship. So this would probably reduce the amount of patch
> > backporting vendors have todo in general, giving them more time to
> > focus on useful upstream work.
> >
> > Design summit -------------
> >
> > Another consequence of 2 month release cycles would be that the
> > design summits become entirely detached from the release cycles,
> > as they'd be on different cycle lengths. So instead of having a
> > focus on release planning, design summits would likely become a
> > forum for general ongoing collaboration and discussions. Probably a
> > half & half merge between the current design summit format and a
> > more traditional open source conference. This also ought to result
> > in less pressure for all contributors to attend all summits, and so
> > be time efficient travelwise and beneficial in cost reduction for
> > vendors.
> >
> > More interestingly, is that by detaching them from release cycles,
> > the summit organizers could have more flexibility with scheduling
> > the summits. They could be placed on the dates that are most
> > suitable for desired travel locations, rather than dates suited to
> > release schedules. Likewise there is more freedom if there was ever
> > a wish to have design summits and the main track conference
> > separated.
> >
> > Election process ----------------
> >
> > I wouldn't suggest any changes to project elections or governance.
> > It is fine to have elections every 6 months. There is no need for
> > that to be tied to release cycle length in any way, particularly as
> > people often serve for more than one release already.
> >
> > Release naming --------------
> >
> > Last but not least, there would be some questions around what this
> > might mean for release naming. There are many options and I don't
> > intend to suggest a particular one is best, just highlight some
> > possibilities that could be considered.
> >
> > The easiest is to just drop the idea of release names entirely and
> > have everyone refer to the release versions instead. Another
> > option is to give each 2 month release its own name, so we'd have
> > to come up with 3x as many names during the year. The person
> > running the release naming elections will probably have a heart
> > attack at this idea :-) Another option is to have names that span a
> > couple of releases, perhaps all releases for an entire year get a
> > given name.
> >
> > Upgrades & deprecation ----------------------
> >
> > It is common right now for projects to say upgrades are only
> > supported between releases N-1 and N. ie to go from Icehouse to
> > Kilo, you need to first deploy Juno. This is passable when you're
> > talking 6 month gaps between cycles, but when there are 2 month
> > gaps it is not reasonable to expect everyone to be moving fast
> > enough to keep up with every release. If an organization's
> > beurocracy means they can't deploy more often than every 12 months,
> > forcing them to deploy the 5 intermediate releases to run upgrade
> > scripts is quite unpleasant. We would likely have to look at
> > officially allowing upgrades between any (N-3, N-2, N-1) to N. From
> > a database POV this should not be hard, since the DB migration
> > scripts don't have any built in assumptions about this. Similarly
> > the versioned objects used by Nova are quite flexible in this
> > regard too, as long as the compat code isn't deleted too soon.
>
> Let's say horizontal teams decided to stick to 2 supported releases
> per year scheme (the current one). What would grenade jobs for stable
> branches (and master) test in this case?
>
> >
> > Deprecation warnings would need similar consideration. It would not
> > be sufficient to deprecate in one release and delete in the next.
> > We'd likely want to say that depecations last for a given time
> > period rather than number of releases, eg 6 months. This could be
> > easily handled by simply including the date of initial deprecation
> > in the deprecation message. It would thus be clear when the feature
> > will be removed to all involved.
> >
> > Why do releases at all ? ------------------------
> >
> > Some people might wonder why we should do releases at all. We
> > already aim to support a continuous deployment model for users as
> > an option. Continuous deployment, however, is quite a conceptual
> > jump for many users and organizations in particular. Their internal
> > software approval processes and bureaucracy can make this
> > impractical to even consider in the forseeable future. So if
> > openstack did not do releases, then many downstream vendors would
> > still end up taking git snapshots and forming periodic releases
> > around them. The unsatisfactory result would be a balkanization of
> > OpenStack where each vendor duplicated the others work with their
> > own release cycles in slightly different ways. To some extent this
> > happens already as vendors backport features, due to inability to
> > wait 6 months for next major release. A two month release cycle
> > though would likely reduce the amount of difference between what
> > each vendor ships which would be a good thing in general. Releases
> > also serve as a useful hook for marketing activities by the project
> > and vendors alike, giving a good opportunity to promote the new
> > body of features now available to users. It might be difficult to
> > drum up media enthusiasm to promote arbitrary git snapshots, but
> > then I'm no marketing expert.
> >
>
> I think it's still easier for people to work in release/stability mode
> for some predefined time than "forever".
>
> >
> >
> >
> > Regards, Daniel
> >
> > [1]
> >
> http://lists.openstack.org/pipermail/openstack-dev/2014-September/044872.html
> >
> >
> [2] http://www.openstack.org/projects/openstack-faq/
> > [3] https://wiki.openstack.org/wiki/Kilo_Release_Schedule [4]
> >
> http://lists.openstack.org/pipermail/openstack-dev/2014-October/047922.html
> >
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQEbBAEBAgAGBQJU9IheAAoJEC5aWaUY1u57/M4H912jtcA8IP424DZnkDVm/6dF
> uXl02JqeUeFvne+M9/JoRIO+GwuY05kfmwFAXY5RNX5PmW1MA+eLpirKI5waMjzJ
> 6zAwViir2sLtrXjpHNKkUrQxzURWN72qvnU+idCJlExU4ZXCkKcTuGo0YC6j46WH
> mLuRheb3wRqmkMrZ7nlShaAiSkLBYpslP+NuKH4JuWzyAZqrQsTwcu9duOD9QiTR
> 294ojzqihCth7npMeXcdMtdkl+GoxP494ojouWqVTAYCpWDfTiZw3/Slb0VTYVIe
> uIS1gN2Hmur1vTbkM4kgvU3D9Uf+uLS9vH9GDg+zyDsjQQm/6RToCOTFiAYp8w==
> =nhsu
> -----END PGP SIGNATURE-----
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150302/20f95105/attachment.html>