Open Stack

On 08/06/2014 11:51 AM, Eoghan Glynn wrote:
> 
> 
>> Hi everyone,
>>
>> With the incredible growth of OpenStack, our development community is
>> facing complex challenges. How we handle those might determine the
>> ultimate success or failure of OpenStack.
>>
>> With this cycle we hit new limits in our processes, tools and cultural
>> setup. This resulted in new limiting factors on our overall velocity,
>> which is frustrating for developers. This resulted in the burnout of key
>> firefighting resources. This resulted in tension between people who try
>> to get specific work done and people who try to keep a handle on the big
>> picture.
>>
>> It all boils down to an imbalance between strategic and tactical
>> contributions. At the beginning of this project, we had a strong inner
>> group of people dedicated to fixing all loose ends. Then a lot of
>> companies got interested in OpenStack and there was a surge in tactical,
>> short-term contributions. We put on a call for more resources to be
>> dedicated to strategic contributions like critical bugfixing,
>> vulnerability management, QA, infrastructure... and that call was
>> answered by a lot of companies that are now key members of the OpenStack
>> Foundation, and all was fine again. But OpenStack contributors kept on
>> growing, and we grew the narrowly-focused population way faster than the
>> cross-project population.
>>
>> At the same time, we kept on adding new projects to incubation and to
>> the integrated release, which is great... but the new developers you get
>> on board with this are much more likely to be tactical than strategic
>> contributors. This also contributed to the imbalance. The penalty for
>> that imbalance is twofold: we don't have enough resources available to
>> solve old, known OpenStack-wide issues; but we also don't have enough
>> resources to identify and fix new issues.
>>
>> We have several efforts under way, like calling for new strategic
>> contributors, driving towards in-project functional testing, making
>> solving rare issues a more attractive endeavor, or hiring resources
>> directly at the Foundation level to help address those. But there is a
>> topic we haven't raised yet: should we concentrate on fixing what is
>> currently in the integrated release rather than adding new projects ?
>>
>> We seem to be unable to address some key issues in the software we
>> produce, and part of it is due to strategic contributors (and core
>> reviewers) being overwhelmed just trying to stay afloat of what's
>> happening. For such projects, is it time for a pause ? Is it time to
>> define key cycle goals and defer everything else ?
>>
>> On the integrated release side, "more projects" means stretching our
>> limited strategic resources more. Is it time for the Technical Committee
>> to more aggressively define what is "in" and what is "out" ? If we go
>> through such a redefinition, shall we push currently-integrated projects
>> that fail to match that definition out of the "integrated release" inner
>> circle ?
>>
>> The TC discussion on what the integrated release should or should not
>> include has always been informally going on. Some people would like to
>> strictly limit to end-user-facing projects. Some others suggest that
>> "OpenStack" should just be about integrating/exposing/scaling smart
>> functionality that lives in specialized external projects, rather than
>> trying to outsmart those by writing our own implementation. Some others
>> are advocates of carefully moving up the stack, and to resist from
>> further addressing IaaS+ services until we "complete" the pure IaaS
>> space in a satisfactory manner. Some others would like to build a
>> roadmap based on AWS services. Some others would just add anything that
>> fits the incubation/integration requirements.
>>
>> On one side this is a long-term discussion, but on the other we also
>> need to make quick decisions. With 4 incubated projects, and 2 new ones
>> currently being proposed, there are a lot of people knocking at the door.
>>
>> Thanks for reading this braindump this far. I hope this will trigger the
>> open discussions we need to have, as an open source project, to reach
>> the next level.
> 
> 
> Thanks Thierry, for this timely post.
> 
> You've touched on multiple trains-of-thought that could indeed
> justify separate threads of their own.
> 
> I agree with your read on the diverging growth rates in the
> strategic versus the tactical elements of the community.
> 
> I would also be supportive of the notion of taking a cycle out to
> fully concentrate on solving existing quality/scaling/performance
> issues, if that's what you meant by pausing to define key cycle
> goals while deferring everything else.
> 
> Though FWIW I think scaling back the set of currently integrated
> projects is not the appropriate solution to the problem of over-
> stretched strategic resources on the QA/infra side of the house.
> 
> Rather, I think the proposed move to in-project functional
> testing, in place of throwing the kitchen sink into Tempest,
> is far more likely to pay dividends in terms of making the job
> facing the QA Trojans more tractable and sustainable.
> 
> Just my $0.02 ...

While I definitely think re-balancing our quality responsibilities back
into the projects will provide an overall better release, I think it's
going to take a long time before it lightens our load to the point where
we get more breathing room again.

This isn't just QA issues, it's a coordination issue on overall
consistency across projects. Something that worked fine at 5 integrated
projects, got strained at 9, and I think is completely untenable at 15.

I think one of the big issues with a large number of projects is that
implications of implementation of one project impact others, but people
don't always realize. Locally correct decisions for each project may not
be globally correct for OpenStack. The GBP discussion, the Rally
discussion, all are flavors of this.

People are frustrated in infra load, for instance. It's probably worth
noting that the 'config' repo currently has more commits landed than any
other project in OpenStack besides 'nova' in this release. It has 30%
the core team size as Nova (http://stackalytics.com/?metric=commits).

So I do think we need to really think about what *must* be in OpenStack
for it to be successful, and ensure that story is well thought out, and
that the pieces which provide those features in OpenStack are clearly
best of breed, so they are deployed in all OpenStack deployments, and
can be counted on by users of OpenStack. Because if every version of
OpenStack deploys with a different Auth API (an example that's current
but going away), we can't grow an ecosystem of tools around it.

This is organic definition of OpenStack through feedback with operators
and developers on what's minimum needed and currently working well
enough that people are happy to maintain it. And make that solid.

Having a TC that is independently selected separate from the PTLs allows
that group to try to make some holistic calls here.

At the end of the day, that's probably going to mean saying No to more
things. Everytime I turn around everyone wants the TC to say No to
things, just not to their particular thing. :) Which is human nature.
But I think if we don't start saying No to more things we're going to
end up with a pile of mud that no one is happy with.

	-Sean

-- 
Sean Dague
http://dague.net