Open Stack

On Thu, Aug 7, 2014 at 1:26 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>
>
>
> On Tue, Aug 5, 2014 at 9:03 AM, Thierry Carrez <thierry at openstack.org>
> wrote:
>>
>> Hi everyone,
>>
>> With the incredible growth of OpenStack, our development community is
>> facing complex challenges. How we handle those might determine the
>> ultimate success or failure of OpenStack.
>>
>> With this cycle we hit new limits in our processes, tools and cultural
>> setup. This resulted in new limiting factors on our overall velocity,
>> which is frustrating for developers. This resulted in the burnout of key
>> firefighting resources. This resulted in tension between people who try
>> to get specific work done and people who try to keep a handle on the big
>> picture.
>>
>> It all boils down to an imbalance between strategic and tactical
>> contributions. At the beginning of this project, we had a strong inner
>> group of people dedicated to fixing all loose ends. Then a lot of
>> companies got interested in OpenStack and there was a surge in tactical,
>> short-term contributions. We put on a call for more resources to be
>> dedicated to strategic contributions like critical bugfixing,
>> vulnerability management, QA, infrastructure... and that call was
>> answered by a lot of companies that are now key members of the OpenStack
>> Foundation, and all was fine again. But OpenStack contributors kept on
>> growing, and we grew the narrowly-focused population way faster than the
>> cross-project population.
>>
>>
>> At the same time, we kept on adding new projects to incubation and to
>> the integrated release, which is great... but the new developers you get
>> on board with this are much more likely to be tactical than strategic
>> contributors. This also contributed to the imbalance. The penalty for
>> that imbalance is twofold: we don't have enough resources available to
>> solve old, known OpenStack-wide issues; but we also don't have enough
>> resources to identify and fix new issues.
>>
>> We have several efforts under way, like calling for new strategic
>> contributors, driving towards in-project functional testing, making
>> solving rare issues a more attractive endeavor, or hiring resources
>> directly at the Foundation level to help address those. But there is a
>> topic we haven't raised yet: should we concentrate on fixing what is
>> currently in the integrated release rather than adding new projects ?
>
>
> TL;DR: Our development model is having growing pains. until we sort out the
> growing pains adding more projects spreads us too thin.
>
+100

> In addition to the issues mentioned above, with the scale of OpenStack today
> we have many major cross project issues to address and no good place to
> discuss them.
>
We do have the ML, as well as the cross-project meeting every Tuesday
[1], but we as a project need to do a better job of actually bringing
up relevant issues here.

[1] https://wiki.openstack.org/wiki/Meetings/ProjectMeeting

>>
>>
>> We seem to be unable to address some key issues in the software we
>> produce, and part of it is due to strategic contributors (and core
>> reviewers) being overwhelmed just trying to stay afloat of what's
>> happening. For such projects, is it time for a pause ? Is it time to
>> define key cycle goals and defer everything else ?
>
>
>
> I really like this idea, as Michael and others alluded to in above, we are
> attempting to set cycle goals for Kilo in Nova. but I think it is worth
> doing for all of OpenStack. We would like to make a list of key goals before
> the summit so that we can plan our summit sessions around the goals. On a
> really high level one way to look at this is, in Kilo we need to pay down
> our technical debt.
>
> The slots/runway idea is somewhat separate from defining key cycle goals; we
> can be approve blueprints based on key cycle goals without doing slots.  But
> with so many concurrent blueprints up for review at any given time, the
> review teams are doing a lot of multitasking and humans are not very good at
> multitasking. Hopefully slots can help address this issue, and hopefully
> allow us to actually merge more blueprints in a given cycle.
>
I'm not 100% sold on what the slots idea buys us. What I've seen this
cycle in Neutron is that we have a LOT of BPs proposed. We approve
them after review. And then we hit one of two issues: Slow review
cycles, and slow code turnaround issues. I don't think slots would
help this, and in fact may cause more issues. If we approve a BP and
give it a slot for which the eventual result is slow review and/or
code review turnaround, we're right back where we started. Even worse,
we may have not picked a BP for which the code submitter would have
turned around reviews faster. So we've now doubly hurt ourselves. I
have no idea how to solve this issue, but by over subscribing the
slots (e.g. over approving), we allow for the submissions with faster
turnaround a chance to merge quicker. With slots, we've removed this
capability by limiting what is even allowed to be considered for
review.

Thanks,
Kyle

>>
>>
>> On the integrated release side, "more projects" means stretching our
>> limited strategic resources more. Is it time for the Technical Committee
>> to more aggressively define what is "in" and what is "out" ? If we go
>> through such a redefinition, shall we push currently-integrated projects
>> that fail to match that definition out of the "integrated release" inner
>> circle ?
>>
>> The TC discussion on what the integrated release should or should not
>> include has always been informally going on. Some people would like to
>> strictly limit to end-user-facing projects. Some others suggest that
>> "OpenStack" should just be about integrating/exposing/scaling smart
>> functionality that lives in specialized external projects, rather than
>> trying to outsmart those by writing our own implementation. Some others
>> are advocates of carefully moving up the stack, and to resist from
>> further addressing IaaS+ services until we "complete" the pure IaaS
>> space in a satisfactory manner. Some others would like to build a
>> roadmap based on AWS services. Some others would just add anything that
>> fits the incubation/integration requirements.
>>
>>
>> On one side this is a long-term discussion, but on the other we also
>> need to make quick decisions. With 4 incubated projects, and 2 new ones
>> currently being proposed, there are a lot of people knocking at the door.
>
>
>
> I have a slightly different short term opinion on what 'OpenStack' should
> be: it should work really well.
>
> While we need to figure out howto increase our strategic resources,
> realistically I think that will still be the limiting factor in Kilo. So in
> order to better allocate our cross project strategic resources, I think we
> should do a project level triage ('identify projects that are likely to die,
> regardless of what care they receive' to paraphrase the medical definition
> of the term), and tighten our focus until we get our development process
> into a better state.
>
>
>>
>>
>> Thanks for reading this braindump this far. I hope this will trigger the
>> open discussions we need to have, as an open source project, to reach
>> the next level.
>
>
> Thank you for bringing this topic up.
>
>>
>>
>> Cheers,
>>
>> --
>> Thierry Carrez (ttx)
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>