Open Stack

On Tue, Aug 12, 2014 at 11:08 AM, Doug Hellmann <doug at doughellmann.com>
wrote:

>
> On Aug 12, 2014, at 1:44 PM, Dolph Mathews <dolph.mathews at gmail.com>
> wrote:
>
>
> On Tue, Aug 12, 2014 at 12:30 AM, Joe Gordon <joe.gordon0 at gmail.com>
> wrote:
>
>>
>>
>>
>> On Fri, Aug 8, 2014 at 6:58 AM, Kyle Mestery <mestery at mestery.com> wrote:
>>
>>> On Thu, Aug 7, 2014 at 1:26 PM, Joe Gordon <joe.gordon0 at gmail.com>
>>> wrote:
>>> >
>>> >
>>> >
>>> > On Tue, Aug 5, 2014 at 9:03 AM, Thierry Carrez <thierry at openstack.org>
>>> > wrote:
>>> >>
>>> >> Hi everyone,
>>> >>
>>> >> With the incredible growth of OpenStack, our development community is
>>> >> facing complex challenges. How we handle those might determine the
>>> >> ultimate success or failure of OpenStack.
>>> >>
>>> >> With this cycle we hit new limits in our processes, tools and cultural
>>> >> setup. This resulted in new limiting factors on our overall velocity,
>>> >> which is frustrating for developers. This resulted in the burnout of
>>> key
>>> >> firefighting resources. This resulted in tension between people who
>>> try
>>> >> to get specific work done and people who try to keep a handle on the
>>> big
>>> >> picture.
>>> >>
>>> >> It all boils down to an imbalance between strategic and tactical
>>> >> contributions. At the beginning of this project, we had a strong inner
>>> >> group of people dedicated to fixing all loose ends. Then a lot of
>>> >> companies got interested in OpenStack and there was a surge in
>>> tactical,
>>> >> short-term contributions. We put on a call for more resources to be
>>> >> dedicated to strategic contributions like critical bugfixing,
>>> >> vulnerability management, QA, infrastructure... and that call was
>>> >> answered by a lot of companies that are now key members of the
>>> OpenStack
>>> >> Foundation, and all was fine again. But OpenStack contributors kept on
>>> >> growing, and we grew the narrowly-focused population way faster than
>>> the
>>> >> cross-project population.
>>> >>
>>> >>
>>> >> At the same time, we kept on adding new projects to incubation and to
>>> >> the integrated release, which is great... but the new developers you
>>> get
>>> >> on board with this are much more likely to be tactical than strategic
>>> >> contributors. This also contributed to the imbalance. The penalty for
>>> >> that imbalance is twofold: we don't have enough resources available to
>>> >> solve old, known OpenStack-wide issues; but we also don't have enough
>>> >> resources to identify and fix new issues.
>>> >>
>>> >> We have several efforts under way, like calling for new strategic
>>> >> contributors, driving towards in-project functional testing, making
>>> >> solving rare issues a more attractive endeavor, or hiring resources
>>> >> directly at the Foundation level to help address those. But there is a
>>> >> topic we haven't raised yet: should we concentrate on fixing what is
>>> >> currently in the integrated release rather than adding new projects ?
>>> >
>>> >
>>> > TL;DR: Our development model is having growing pains. until we sort
>>> out the
>>> > growing pains adding more projects spreads us too thin.
>>> >
>>> +100
>>>
>>> > In addition to the issues mentioned above, with the scale of OpenStack
>>> today
>>> > we have many major cross project issues to address and no good place to
>>> > discuss them.
>>> >
>>> We do have the ML, as well as the cross-project meeting every Tuesday
>>> [1], but we as a project need to do a better job of actually bringing
>>> up relevant issues here.
>>>
>>> [1] https://wiki.openstack.org/wiki/Meetings/ProjectMeeting
>>>
>>> >>
>>> >>
>>> >> We seem to be unable to address some key issues in the software we
>>> >> produce, and part of it is due to strategic contributors (and core
>>> >> reviewers) being overwhelmed just trying to stay afloat of what's
>>> >> happening. For such projects, is it time for a pause ? Is it time to
>>> >> define key cycle goals and defer everything else ?
>>> >
>>> >
>>> >
>>> > I really like this idea, as Michael and others alluded to in above, we
>>> are
>>> > attempting to set cycle goals for Kilo in Nova. but I think it is worth
>>> > doing for all of OpenStack. We would like to make a list of key goals
>>> before
>>> > the summit so that we can plan our summit sessions around the goals.
>>> On a
>>> > really high level one way to look at this is, in Kilo we need to pay
>>> down
>>> > our technical debt.
>>> >
>>> > The slots/runway idea is somewhat separate from defining key cycle
>>> goals; we
>>> > can be approve blueprints based on key cycle goals without doing
>>> slots.  But
>>> > with so many concurrent blueprints up for review at any given time, the
>>> > review teams are doing a lot of multitasking and humans are not very
>>> good at
>>> > multitasking. Hopefully slots can help address this issue, and
>>> hopefully
>>> > allow us to actually merge more blueprints in a given cycle.
>>> >
>>> I'm not 100% sold on what the slots idea buys us. What I've seen this
>>> cycle in Neutron is that we have a LOT of BPs proposed. We approve
>>> them after review. And then we hit one of two issues: Slow review
>>> cycles, and slow code turnaround issues. I don't think slots would
>>> help this, and in fact may cause more issues. If we approve a BP and
>>> give it a slot for which the eventual result is slow review and/or
>>> code review turnaround, we're right back where we started. Even worse,
>>> we may have not picked a BP for which the code submitter would have
>>> turned around reviews faster. So we've now doubly hurt ourselves. I
>>> have no idea how to solve this issue, but by over subscribing the
>>> slots (e.g. over approving), we allow for the submissions with faster
>>> turnaround a chance to merge quicker. With slots, we've removed this
>>> capability by limiting what is even allowed to be considered for
>>> review.
>>>
>>
>> Slow review: by limiting the number of blueprints up we hope to focus our
>> efforts on fewer concurrent things
>> slow code turn around: when a blueprint is given a slot (runway) we will
>> first make sure the author/owner is available for fast code turnaround.
>>
>> If a blueprint review stalls out (slow code turnaround, stalemate in
>> review discussions etc.) we will take the slot and give it to another
>> blueprint.
>>
>
> How is that more efficient than today's do-the-best-we-can approach? It
> just sounds like bureaucracy to me.
>
> Reading between the lines throughout this thread, it sounds like what
> we're lacking is a reliable method to communicate review prioritization to
> core reviewers.
>
>
> It seems like this is exactly what the slots give us, though. The core
> review team picks a number of slots indicating how much work they think
> they can actually do (less than the available number of blueprints), and
> then blueprints queue up to get a slot based on priorities and turnaround
> time and other criteria that try to make slot allocation fair. By having
> the slots, not only is the review priority communicated to the review team,
> it is also communicated to anyone watching the project.
>

FYI:

Here is the full nova proposal on  "Blueprint in Kilo: Runways and Project
Priorities"

https://review.openstack.org/#/c/112733/

http://docs-draft.openstack.org/33/112733/4/check/gate-nova-docs/5f38603/doc/build/html/devref/runways.html



> Doug
>
>
> What the community wants/needs is just noise generated via IRC, email,
> etherpad, launchpad, etc. It's part of a PTL's job to help filter that
> noise and help communicate that back to core reviewers, but that just ends
> up creating more IRC/email/etc, rather than directly impacting anyone's
> experience with gerrit.
>
>
>>
>>>
>>> Thanks,
>>> Kyle
>>>
>>> >>
>>> >>
>>> >> On the integrated release side, "more projects" means stretching our
>>> >> limited strategic resources more. Is it time for the Technical
>>> Committee
>>> >> to more aggressively define what is "in" and what is "out" ? If we go
>>> >> through such a redefinition, shall we push currently-integrated
>>> projects
>>> >> that fail to match that definition out of the "integrated release"
>>> inner
>>> >> circle ?
>>> >>
>>> >> The TC discussion on what the integrated release should or should not
>>> >> include has always been informally going on. Some people would like to
>>> >> strictly limit to end-user-facing projects. Some others suggest that
>>> >> "OpenStack" should just be about integrating/exposing/scaling smart
>>> >> functionality that lives in specialized external projects, rather than
>>> >> trying to outsmart those by writing our own implementation. Some
>>> others
>>> >> are advocates of carefully moving up the stack, and to resist from
>>> >> further addressing IaaS+ services until we "complete" the pure IaaS
>>> >> space in a satisfactory manner. Some others would like to build a
>>> >> roadmap based on AWS services. Some others would just add anything
>>> that
>>> >> fits the incubation/integration requirements.
>>> >>
>>> >>
>>> >> On one side this is a long-term discussion, but on the other we also
>>> >> need to make quick decisions. With 4 incubated projects, and 2 new
>>> ones
>>> >> currently being proposed, there are a lot of people knocking at the
>>> door.
>>> >
>>> >
>>> >
>>> > I have a slightly different short term opinion on what 'OpenStack'
>>> should
>>> > be: it should work really well.
>>> >
>>> > While we need to figure out howto increase our strategic resources,
>>> > realistically I think that will still be the limiting factor in Kilo.
>>> So in
>>> > order to better allocate our cross project strategic resources, I
>>> think we
>>> > should do a project level triage ('identify projects that are likely
>>> to die,
>>> > regardless of what care they receive' to paraphrase the medical
>>> definition
>>> > of the term), and tighten our focus until we get our development
>>> process
>>> > into a better state.
>>> >
>>> >
>>> >>
>>> >>
>>> >> Thanks for reading this braindump this far. I hope this will trigger
>>> the
>>> >> open discussions we need to have, as an open source project, to reach
>>> >> the next level.
>>> >
>>> >
>>> > Thank you for bringing this topic up.
>>> >
>>> >>
>>> >>
>>> >> Cheers,
>>> >>
>>> >> --
>>> >> Thierry Carrez (ttx)
>>> >>
>>> >> _______________________________________________
>>> >> OpenStack-dev mailing list
>>> >> OpenStack-dev at lists.openstack.org
>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > OpenStack-dev mailing list
>>> > OpenStack-dev at lists.openstack.org
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> >
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140812/8a0e9c82/attachment.html>