Open Stack

On Tue, Aug 5, 2014 at 10:02 AM, Monty Taylor <mordred at inaugust.com> wrote:
> On 08/05/2014 09:03 AM, Thierry Carrez wrote:
>>
>> Hi everyone,
>>
>> With the incredible growth of OpenStack, our development community is
>> facing complex challenges. How we handle those might determine the
>> ultimate success or failure of OpenStack.
>>
>> With this cycle we hit new limits in our processes, tools and cultural
>> setup. This resulted in new limiting factors on our overall velocity,
>> which is frustrating for developers. This resulted in the burnout of key
>> firefighting resources. This resulted in tension between people who try
>> to get specific work done and people who try to keep a handle on the big
>> picture.
>>
>> It all boils down to an imbalance between strategic and tactical
>> contributions. At the beginning of this project, we had a strong inner
>> group of people dedicated to fixing all loose ends. Then a lot of
>> companies got interested in OpenStack and there was a surge in tactical,
>> short-term contributions. We put on a call for more resources to be
>> dedicated to strategic contributions like critical bugfixing,
>> vulnerability management, QA, infrastructure... and that call was
>> answered by a lot of companies that are now key members of the OpenStack
>> Foundation, and all was fine again. But OpenStack contributors kept on
>> growing, and we grew the narrowly-focused population way faster than the
>> cross-project population.
>>
>> At the same time, we kept on adding new projects to incubation and to
>> the integrated release, which is great... but the new developers you get
>> on board with this are much more likely to be tactical than strategic
>> contributors. This also contributed to the imbalance. The penalty for
>> that imbalance is twofold: we don't have enough resources available to
>> solve old, known OpenStack-wide issues; but we also don't have enough
>> resources to identify and fix new issues.
>>
>> We have several efforts under way, like calling for new strategic
>> contributors, driving towards in-project functional testing, making
>> solving rare issues a more attractive endeavor, or hiring resources
>> directly at the Foundation level to help address those. But there is a
>> topic we haven't raised yet: should we concentrate on fixing what is
>> currently in the integrated release rather than adding new projects ?
>>
>> We seem to be unable to address some key issues in the software we
>> produce, and part of it is due to strategic contributors (and core
>> reviewers) being overwhelmed just trying to stay afloat of what's
>> happening. For such projects, is it time for a pause ? Is it time to
>> define key cycle goals and defer everything else ?
>>
>> On the integrated release side, "more projects" means stretching our
>> limited strategic resources more. Is it time for the Technical Committee
>> to more aggressively define what is "in" and what is "out" ? If we go
>> through such a redefinition, shall we push currently-integrated projects
>> that fail to match that definition out of the "integrated release" inner
>> circle ?
>>
>> The TC discussion on what the integrated release should or should not
>> include has always been informally going on. Some people would like to
>> strictly limit to end-user-facing projects. Some others suggest that
>> "OpenStack" should just be about integrating/exposing/scaling smart
>> functionality that lives in specialized external projects, rather than
>> trying to outsmart those by writing our own implementation. Some others
>> are advocates of carefully moving up the stack, and to resist from
>> further addressing IaaS+ services until we "complete" the pure IaaS
>> space in a satisfactory manner. Some others would like to build a
>> roadmap based on AWS services. Some others would just add anything that
>> fits the incubation/integration requirements.
>>
>> On one side this is a long-term discussion, but on the other we also
>> need to make quick decisions. With 4 incubated projects, and 2 new ones
>> currently being proposed, there are a lot of people knocking at the door.
>>
>> Thanks for reading this braindump this far. I hope this will trigger the
>> open discussions we need to have, as an open source project, to reach
>> the next level.
>
>
> Yes.
>
> Additionally, and I think we've been getting better at this in the 2 cycles
> that we've had an all-elected TC, I think we need to learn how to say no on
> technical merit - and we need to learn how to say "thank you for your
> effort, but this isn't working out" Breaking up with someone is hard to do,
> but sometimes it's best for everyone involved.
>

I agree.

The challenge is scaling the technical assessment of projects. We're
all busy, and digging deeply enough into a new project to make an
accurate assessment of it is time consuming. Some times, there are
impartial subject-matter experts who can spot problems very quickly,
but how do we actually gauge fitness?

Letting the industry field-test a project and feed their experience
back into the community is a slow process, but that is the best
measure of a project's success. I seem to recall this being an
implicit expectation a few years ago, but haven't seen it discussed in
a while. I'm not suggesting we make a policy of it, but if, after a
few cycles, a project is still not meeting the needs of users, I think
that's a very good reason to free up the hold on that role within the
stack so other projects can try and fill it (assuming that is even a
role we would want filled).

-Deva


> I'm wary of explicit answers in the form of policy at this point - I think
> we've spent far too long using policy as a shield from hard questions. The
> questions of "Is OpenStack IaaS, or should it also be PaaS" I think are
> useless questions that exist only to further the lives of analysts,
> journalists and bloggers. Instead, I'd love to focus more on "what is
> software that solves problems" and "what is software that solves problems
> that we're in a good position to solve" So if Savanna solves problems for
> users well in a way that makes sense, I'd hate to exclude it because it
> didn't meet a policy's pre-conceived notion that Hadoop is not "IaaS" or
> even "IaaS+" You know what I NEVER do when I'm working on deploying
> services? I never say "Hey Jim, we should go operate our IaaS resources"
> Literally, it has never happened.
>
> Finally, as it pertains to technical merits, and at the risk of magicking
> myself into a corner of defining a policy after saying I don't like them ...
> I'd really like to think a bit more about the nature of what we do and what
> we excel at. I'll be the first to admit I've been involved in transgressions
> of this - but I believe it's becoming clear to me that where we as a general
> community and as a software project excel are the places where we're
> orchestrating and/or gluing things together that are existing technologies.
> Nova orchestrates hypervisors, cinder provides iscsi, keystone has long
> resisted being a user management system and instead is a glue layer between
> openstack services and existing user management systems. Trove orchestrates
> databases - it did not attempt to write an RDBMS from scratch in python.
> Simply turning that into a rule of thumb is perhaps overly simplistic and
> could lead into the dangerous land of adhering to policy in the face of
> sanity. But for myself, I'm going to be thinking about all of our projects
> through that lens over the coming months. Please don't be surprised or take
> it personally if I start suggesting that we've been too liberal in
> suggesting projects that sit on the other side of that line, and that
> perhaps its time to bring that chapter in our evolution to a close.
>
> Monty
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev