Open Stack

On 08/08/2014 07:58 AM, Kyle Mestery wrote:
> On Thu, Aug 7, 2014 at 1:26 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>>
>>
>>
>> On Tue, Aug 5, 2014 at 9:03 AM, Thierry Carrez <thierry at openstack.org>
>> wrote:
>>>
>>> Hi everyone,
>>>
>>> With the incredible growth of OpenStack, our development community is
>>> facing complex challenges. How we handle those might determine the
>>> ultimate success or failure of OpenStack.
>>>
>>> With this cycle we hit new limits in our processes, tools and cultural
>>> setup. This resulted in new limiting factors on our overall velocity,
>>> which is frustrating for developers. This resulted in the burnout of key
>>> firefighting resources. This resulted in tension between people who try
>>> to get specific work done and people who try to keep a handle on the big
>>> picture.
>>>
>>> It all boils down to an imbalance between strategic and tactical
>>> contributions. At the beginning of this project, we had a strong inner
>>> group of people dedicated to fixing all loose ends. Then a lot of
>>> companies got interested in OpenStack and there was a surge in tactical,
>>> short-term contributions. We put on a call for more resources to be
>>> dedicated to strategic contributions like critical bugfixing,
>>> vulnerability management, QA, infrastructure... and that call was
>>> answered by a lot of companies that are now key members of the OpenStack
>>> Foundation, and all was fine again. But OpenStack contributors kept on
>>> growing, and we grew the narrowly-focused population way faster than the
>>> cross-project population.
>>>
>>>
>>> At the same time, we kept on adding new projects to incubation and to
>>> the integrated release, which is great... but the new developers you get
>>> on board with this are much more likely to be tactical than strategic
>>> contributors. This also contributed to the imbalance. The penalty for
>>> that imbalance is twofold: we don't have enough resources available to
>>> solve old, known OpenStack-wide issues; but we also don't have enough
>>> resources to identify and fix new issues.
>>>
>>> We have several efforts under way, like calling for new strategic
>>> contributors, driving towards in-project functional testing, making
>>> solving rare issues a more attractive endeavor, or hiring resources
>>> directly at the Foundation level to help address those. But there is a
>>> topic we haven't raised yet: should we concentrate on fixing what is
>>> currently in the integrated release rather than adding new projects ?
>>
>>
>> TL;DR: Our development model is having growing pains. until we sort out the
>> growing pains adding more projects spreads us too thin.
>>
> +100
> 
>> In addition to the issues mentioned above, with the scale of OpenStack today
>> we have many major cross project issues to address and no good place to
>> discuss them.
>>
> We do have the ML, as well as the cross-project meeting every Tuesday
> [1], but we as a project need to do a better job of actually bringing
> up relevant issues here.
> 
> [1] https://wiki.openstack.org/wiki/Meetings/ProjectMeeting
> 
>>>
>>>
>>> We seem to be unable to address some key issues in the software we
>>> produce, and part of it is due to strategic contributors (and core
>>> reviewers) being overwhelmed just trying to stay afloat of what's
>>> happening. For such projects, is it time for a pause ? Is it time to
>>> define key cycle goals and defer everything else ?
>>
>>
>>
>> I really like this idea, as Michael and others alluded to in above, we are
>> attempting to set cycle goals for Kilo in Nova. but I think it is worth
>> doing for all of OpenStack. We would like to make a list of key goals before
>> the summit so that we can plan our summit sessions around the goals. On a
>> really high level one way to look at this is, in Kilo we need to pay down
>> our technical debt.
>>
>> The slots/runway idea is somewhat separate from defining key cycle goals; we
>> can be approve blueprints based on key cycle goals without doing slots.  But
>> with so many concurrent blueprints up for review at any given time, the
>> review teams are doing a lot of multitasking and humans are not very good at
>> multitasking. Hopefully slots can help address this issue, and hopefully
>> allow us to actually merge more blueprints in a given cycle.
>>
> I'm not 100% sold on what the slots idea buys us. What I've seen this
> cycle in Neutron is that we have a LOT of BPs proposed. We approve
> them after review. And then we hit one of two issues: Slow review
> cycles, and slow code turnaround issues. I don't think slots would
> help this, and in fact may cause more issues. If we approve a BP and
> give it a slot for which the eventual result is slow review and/or
> code review turnaround, we're right back where we started.
My reading of Michael's original proposal addressed this situation. The
slow code review turnaround grants that BP a spot back in the queue
freeing up that slot/runway for a BP that the team feels has a better
chance to use the runway for its intended purpose than the recently
re-queued BP. The model inherently addresses the slow review concern by
ensuring that all reviewers (core, not-yet-core and non-core reviewers)
have their focus on the runways for review.

> Even worse,
> we may have not picked a BP for which the code submitter would have
> turned around reviews faster.
True, so the model addresses this by ensuring constant movement in the
runways to clear a space for the BP that has not yet been assigned a runway.

> So we've now doubly hurt ourselves. I
> have no idea how to solve this issue, but by over subscribing the
> slots (e.g. over approving), we allow for the submissions with faster
> turnaround a chance to merge quicker.
Unfortunately this approach increases the amount of time spent mentally
transitioning from one BP to another for the reviewers.

> With slots, we've removed this
> capability by limiting what is even allowed to be considered for
> review.
Yes, which by design focuses the review process and if the runways are
well maintained, helps to increase the frequency (and quality) of what
gets landed.

I can see your concerns, Kyle, I do think the model presented addresses
the concerns about BP and review.

Thanks,
Anita.
> 
> Thanks,
> Kyle
> 
>>>
>>>
>>> On the integrated release side, "more projects" means stretching our
>>> limited strategic resources more. Is it time for the Technical Committee
>>> to more aggressively define what is "in" and what is "out" ? If we go
>>> through such a redefinition, shall we push currently-integrated projects
>>> that fail to match that definition out of the "integrated release" inner
>>> circle ?
>>>
>>> The TC discussion on what the integrated release should or should not
>>> include has always been informally going on. Some people would like to
>>> strictly limit to end-user-facing projects. Some others suggest that
>>> "OpenStack" should just be about integrating/exposing/scaling smart
>>> functionality that lives in specialized external projects, rather than
>>> trying to outsmart those by writing our own implementation. Some others
>>> are advocates of carefully moving up the stack, and to resist from
>>> further addressing IaaS+ services until we "complete" the pure IaaS
>>> space in a satisfactory manner. Some others would like to build a
>>> roadmap based on AWS services. Some others would just add anything that
>>> fits the incubation/integration requirements.
>>>
>>>
>>> On one side this is a long-term discussion, but on the other we also
>>> need to make quick decisions. With 4 incubated projects, and 2 new ones
>>> currently being proposed, there are a lot of people knocking at the door.
>>
>>
>>
>> I have a slightly different short term opinion on what 'OpenStack' should
>> be: it should work really well.
>>
>> While we need to figure out howto increase our strategic resources,
>> realistically I think that will still be the limiting factor in Kilo. So in
>> order to better allocate our cross project strategic resources, I think we
>> should do a project level triage ('identify projects that are likely to die,
>> regardless of what care they receive' to paraphrase the medical definition
>> of the term), and tighten our focus until we get our development process
>> into a better state.
>>
>>
>>>
>>>
>>> Thanks for reading this braindump this far. I hope this will trigger the
>>> open discussions we need to have, as an open source project, to reach
>>> the next level.
>>
>>
>> Thank you for bringing this topic up.
>>
>>>
>>>
>>> Cheers,
>>>
>>> --
>>> Thierry Carrez (ttx)
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>