Open Stack

On Aug 12, 2014, at 1:44 PM, Dolph Mathews <dolph.mathews at gmail.com> wrote:

> 
> On Tue, Aug 12, 2014 at 12:30 AM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
> 
> 
> 
> On Fri, Aug 8, 2014 at 6:58 AM, Kyle Mestery <mestery at mestery.com> wrote:
> On Thu, Aug 7, 2014 at 1:26 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
> >
> >
> >
> > On Tue, Aug 5, 2014 at 9:03 AM, Thierry Carrez <thierry at openstack.org>
> > wrote:
> >>
> >> Hi everyone,
> >>
> >> With the incredible growth of OpenStack, our development community is
> >> facing complex challenges. How we handle those might determine the
> >> ultimate success or failure of OpenStack.
> >>
> >> With this cycle we hit new limits in our processes, tools and cultural
> >> setup. This resulted in new limiting factors on our overall velocity,
> >> which is frustrating for developers. This resulted in the burnout of key
> >> firefighting resources. This resulted in tension between people who try
> >> to get specific work done and people who try to keep a handle on the big
> >> picture.
> >>
> >> It all boils down to an imbalance between strategic and tactical
> >> contributions. At the beginning of this project, we had a strong inner
> >> group of people dedicated to fixing all loose ends. Then a lot of
> >> companies got interested in OpenStack and there was a surge in tactical,
> >> short-term contributions. We put on a call for more resources to be
> >> dedicated to strategic contributions like critical bugfixing,
> >> vulnerability management, QA, infrastructure... and that call was
> >> answered by a lot of companies that are now key members of the OpenStack
> >> Foundation, and all was fine again. But OpenStack contributors kept on
> >> growing, and we grew the narrowly-focused population way faster than the
> >> cross-project population.
> >>
> >>
> >> At the same time, we kept on adding new projects to incubation and to
> >> the integrated release, which is great... but the new developers you get
> >> on board with this are much more likely to be tactical than strategic
> >> contributors. This also contributed to the imbalance. The penalty for
> >> that imbalance is twofold: we don't have enough resources available to
> >> solve old, known OpenStack-wide issues; but we also don't have enough
> >> resources to identify and fix new issues.
> >>
> >> We have several efforts under way, like calling for new strategic
> >> contributors, driving towards in-project functional testing, making
> >> solving rare issues a more attractive endeavor, or hiring resources
> >> directly at the Foundation level to help address those. But there is a
> >> topic we haven't raised yet: should we concentrate on fixing what is
> >> currently in the integrated release rather than adding new projects ?
> >
> >
> > TL;DR: Our development model is having growing pains. until we sort out the
> > growing pains adding more projects spreads us too thin.
> >
> +100
> 
> > In addition to the issues mentioned above, with the scale of OpenStack today
> > we have many major cross project issues to address and no good place to
> > discuss them.
> >
> We do have the ML, as well as the cross-project meeting every Tuesday
> [1], but we as a project need to do a better job of actually bringing
> up relevant issues here.
> 
> [1] https://wiki.openstack.org/wiki/Meetings/ProjectMeeting
> 
> >>
> >>
> >> We seem to be unable to address some key issues in the software we
> >> produce, and part of it is due to strategic contributors (and core
> >> reviewers) being overwhelmed just trying to stay afloat of what's
> >> happening. For such projects, is it time for a pause ? Is it time to
> >> define key cycle goals and defer everything else ?
> >
> >
> >
> > I really like this idea, as Michael and others alluded to in above, we are
> > attempting to set cycle goals for Kilo in Nova. but I think it is worth
> > doing for all of OpenStack. We would like to make a list of key goals before
> > the summit so that we can plan our summit sessions around the goals. On a
> > really high level one way to look at this is, in Kilo we need to pay down
> > our technical debt.
> >
> > The slots/runway idea is somewhat separate from defining key cycle goals; we
> > can be approve blueprints based on key cycle goals without doing slots.  But
> > with so many concurrent blueprints up for review at any given time, the
> > review teams are doing a lot of multitasking and humans are not very good at
> > multitasking. Hopefully slots can help address this issue, and hopefully
> > allow us to actually merge more blueprints in a given cycle.
> >
> I'm not 100% sold on what the slots idea buys us. What I've seen this
> cycle in Neutron is that we have a LOT of BPs proposed. We approve
> them after review. And then we hit one of two issues: Slow review
> cycles, and slow code turnaround issues. I don't think slots would
> help this, and in fact may cause more issues. If we approve a BP and
> give it a slot for which the eventual result is slow review and/or
> code review turnaround, we're right back where we started. Even worse,
> we may have not picked a BP for which the code submitter would have
> turned around reviews faster. So we've now doubly hurt ourselves. I
> have no idea how to solve this issue, but by over subscribing the
> slots (e.g. over approving), we allow for the submissions with faster
> turnaround a chance to merge quicker. With slots, we've removed this
> capability by limiting what is even allowed to be considered for
> review.
> 
> Slow review: by limiting the number of blueprints up we hope to focus our efforts on fewer concurrent things
> slow code turn around: when a blueprint is given a slot (runway) we will first make sure the author/owner is available for fast code turnaround.
> 
> If a blueprint review stalls out (slow code turnaround, stalemate in review discussions etc.) we will take the slot and give it to another blueprint.
> 
> How is that more efficient than today's do-the-best-we-can approach? It just sounds like bureaucracy to me.
> 
> Reading between the lines throughout this thread, it sounds like what we're lacking is a reliable method to communicate review prioritization to core reviewers.

It seems like this is exactly what the slots give us, though. The core review team picks a number of slots indicating how much work they think they can actually do (less than the available number of blueprints), and then blueprints queue up to get a slot based on priorities and turnaround time and other criteria that try to make slot allocation fair. By having the slots, not only is the review priority communicated to the review team, it is also communicated to anyone watching the project.

Doug

> 
> What the community wants/needs is just noise generated via IRC, email, etherpad, launchpad, etc. It's part of a PTL's job to help filter that noise and help communicate that back to core reviewers, but that just ends up creating more IRC/email/etc, rather than directly impacting anyone's experience with gerrit.
> 
>  
> 
> Thanks,
> Kyle
> 
> >>
> >>
> >> On the integrated release side, "more projects" means stretching our
> >> limited strategic resources more. Is it time for the Technical Committee
> >> to more aggressively define what is "in" and what is "out" ? If we go
> >> through such a redefinition, shall we push currently-integrated projects
> >> that fail to match that definition out of the "integrated release" inner
> >> circle ?
> >>
> >> The TC discussion on what the integrated release should or should not
> >> include has always been informally going on. Some people would like to
> >> strictly limit to end-user-facing projects. Some others suggest that
> >> "OpenStack" should just be about integrating/exposing/scaling smart
> >> functionality that lives in specialized external projects, rather than
> >> trying to outsmart those by writing our own implementation. Some others
> >> are advocates of carefully moving up the stack, and to resist from
> >> further addressing IaaS+ services until we "complete" the pure IaaS
> >> space in a satisfactory manner. Some others would like to build a
> >> roadmap based on AWS services. Some others would just add anything that
> >> fits the incubation/integration requirements.
> >>
> >>
> >> On one side this is a long-term discussion, but on the other we also
> >> need to make quick decisions. With 4 incubated projects, and 2 new ones
> >> currently being proposed, there are a lot of people knocking at the door.
> >
> >
> >
> > I have a slightly different short term opinion on what 'OpenStack' should
> > be: it should work really well.
> >
> > While we need to figure out howto increase our strategic resources,
> > realistically I think that will still be the limiting factor in Kilo. So in
> > order to better allocate our cross project strategic resources, I think we
> > should do a project level triage ('identify projects that are likely to die,
> > regardless of what care they receive' to paraphrase the medical definition
> > of the term), and tighten our focus until we get our development process
> > into a better state.
> >
> >
> >>
> >>
> >> Thanks for reading this braindump this far. I hope this will trigger the
> >> open discussions we need to have, as an open source project, to reach
> >> the next level.
> >
> >
> > Thank you for bringing this topic up.
> >
> >>
> >>
> >> Cheers,
> >>
> >> --
> >> Thierry Carrez (ttx)
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140812/4f6eefbf/attachment.html>