<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 12, 2014 at 12:30 AM, Joe Gordon <span dir="ltr"><<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">
<div><div class="h5">On Fri, Aug 8, 2014 at 6:58 AM, Kyle Mestery <span dir="ltr"><<a href="mailto:mestery@mestery.com" target="_blank">mestery@mestery.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div>On Thu, Aug 7, 2014 at 1:26 PM, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>> wrote:<br>
><br>
><br>
><br>
> On Tue, Aug 5, 2014 at 9:03 AM, Thierry Carrez <<a href="mailto:thierry@openstack.org" target="_blank">thierry@openstack.org</a>><br>
> wrote:<br>
>><br>
>> Hi everyone,<br>
>><br>
>> With the incredible growth of OpenStack, our development community is<br>
>> facing complex challenges. How we handle those might determine the<br>
>> ultimate success or failure of OpenStack.<br>
>><br>
>> With this cycle we hit new limits in our processes, tools and cultural<br>
>> setup. This resulted in new limiting factors on our overall velocity,<br>
>> which is frustrating for developers. This resulted in the burnout of key<br>
>> firefighting resources. This resulted in tension between people who try<br>
>> to get specific work done and people who try to keep a handle on the big<br>
>> picture.<br>
>><br>
>> It all boils down to an imbalance between strategic and tactical<br>
>> contributions. At the beginning of this project, we had a strong inner<br>
>> group of people dedicated to fixing all loose ends. Then a lot of<br>
>> companies got interested in OpenStack and there was a surge in tactical,<br>
>> short-term contributions. We put on a call for more resources to be<br>
>> dedicated to strategic contributions like critical bugfixing,<br>
>> vulnerability management, QA, infrastructure... and that call was<br>
>> answered by a lot of companies that are now key members of the OpenStack<br>
>> Foundation, and all was fine again. But OpenStack contributors kept on<br>
>> growing, and we grew the narrowly-focused population way faster than the<br>
>> cross-project population.<br>
>><br>
>><br>
>> At the same time, we kept on adding new projects to incubation and to<br>
>> the integrated release, which is great... but the new developers you get<br>
>> on board with this are much more likely to be tactical than strategic<br>
>> contributors. This also contributed to the imbalance. The penalty for<br>
>> that imbalance is twofold: we don't have enough resources available to<br>
>> solve old, known OpenStack-wide issues; but we also don't have enough<br>
>> resources to identify and fix new issues.<br>
>><br>
>> We have several efforts under way, like calling for new strategic<br>
>> contributors, driving towards in-project functional testing, making<br>
>> solving rare issues a more attractive endeavor, or hiring resources<br>
>> directly at the Foundation level to help address those. But there is a<br>
>> topic we haven't raised yet: should we concentrate on fixing what is<br>
>> currently in the integrated release rather than adding new projects ?<br>
><br>
><br>
> TL;DR: Our development model is having growing pains. until we sort out the<br>
> growing pains adding more projects spreads us too thin.<br>
><br>
</div></div>+100<br>
<div><br>
> In addition to the issues mentioned above, with the scale of OpenStack today<br>
> we have many major cross project issues to address and no good place to<br>
> discuss them.<br>
><br>
</div>We do have the ML, as well as the cross-project meeting every Tuesday<br>
[1], but we as a project need to do a better job of actually bringing<br>
up relevant issues here.<br>
<br>
[1] <a href="https://wiki.openstack.org/wiki/Meetings/ProjectMeeting" target="_blank">https://wiki.openstack.org/wiki/Meetings/ProjectMeeting</a><br>
<div><br>
>><br>
>><br>
>> We seem to be unable to address some key issues in the software we<br>
>> produce, and part of it is due to strategic contributors (and core<br>
>> reviewers) being overwhelmed just trying to stay afloat of what's<br>
>> happening. For such projects, is it time for a pause ? Is it time to<br>
>> define key cycle goals and defer everything else ?<br>
><br>
><br>
><br>
> I really like this idea, as Michael and others alluded to in above, we are<br>
> attempting to set cycle goals for Kilo in Nova. but I think it is worth<br>
> doing for all of OpenStack. We would like to make a list of key goals before<br>
> the summit so that we can plan our summit sessions around the goals. On a<br>
> really high level one way to look at this is, in Kilo we need to pay down<br>
> our technical debt.<br>
><br>
> The slots/runway idea is somewhat separate from defining key cycle goals; we<br>
> can be approve blueprints based on key cycle goals without doing slots. But<br>
> with so many concurrent blueprints up for review at any given time, the<br>
> review teams are doing a lot of multitasking and humans are not very good at<br>
> multitasking. Hopefully slots can help address this issue, and hopefully<br>
> allow us to actually merge more blueprints in a given cycle.<br>
><br>
</div>I'm not 100% sold on what the slots idea buys us. What I've seen this<br>
cycle in Neutron is that we have a LOT of BPs proposed. We approve<br>
them after review. And then we hit one of two issues: Slow review<br>
cycles, and slow code turnaround issues. I don't think slots would<br>
help this, and in fact may cause more issues. If we approve a BP and<br>
give it a slot for which the eventual result is slow review and/or<br>
code review turnaround, we're right back where we started. Even worse,<br>
we may have not picked a BP for which the code submitter would have<br>
turned around reviews faster. So we've now doubly hurt ourselves. I<br>
have no idea how to solve this issue, but by over subscribing the<br>
slots (e.g. over approving), we allow for the submissions with faster<br>
turnaround a chance to merge quicker. With slots, we've removed this<br>
capability by limiting what is even allowed to be considered for<br>
review.<br></blockquote><div><br></div></div></div><div>Slow review: by limiting the number of blueprints up we hope to focus our efforts on fewer concurrent things</div><div>slow code turn around: when a blueprint is given a slot (runway) we will first make sure the author/owner is available for fast code turnaround.</div>
<div><br></div><div>If a blueprint review stalls out (slow code turnaround, stalemate in review discussions etc.) we will take the slot and give it to another blueprint.</div></div></div></div></blockquote><div><br></div>
<div>How is that more efficient than today's do-the-best-we-can approach? It just sounds like bureaucracy to me.</div><div><br></div><div>Reading between the lines throughout this thread, it sounds like what we're lacking is a reliable method to communicate review prioritization to core reviewers.</div>
<div><br></div><div>What the community wants/needs is just noise generated via IRC, email, etherpad, launchpad, etc. It's part of a PTL's job to help filter that noise and help communicate that back to core reviewers, but that just ends up creating more IRC/email/etc, rather than directly impacting anyone's experience with gerrit.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Thanks,<br>
Kyle<br>
<div><div><br>
>><br>
>><br>
>> On the integrated release side, "more projects" means stretching our<br>
>> limited strategic resources more. Is it time for the Technical Committee<br>
>> to more aggressively define what is "in" and what is "out" ? If we go<br>
>> through such a redefinition, shall we push currently-integrated projects<br>
>> that fail to match that definition out of the "integrated release" inner<br>
>> circle ?<br>
>><br>
>> The TC discussion on what the integrated release should or should not<br>
>> include has always been informally going on. Some people would like to<br>
>> strictly limit to end-user-facing projects. Some others suggest that<br>
>> "OpenStack" should just be about integrating/exposing/scaling smart<br>
>> functionality that lives in specialized external projects, rather than<br>
>> trying to outsmart those by writing our own implementation. Some others<br>
>> are advocates of carefully moving up the stack, and to resist from<br>
>> further addressing IaaS+ services until we "complete" the pure IaaS<br>
>> space in a satisfactory manner. Some others would like to build a<br>
>> roadmap based on AWS services. Some others would just add anything that<br>
>> fits the incubation/integration requirements.<br>
>><br>
>><br>
>> On one side this is a long-term discussion, but on the other we also<br>
>> need to make quick decisions. With 4 incubated projects, and 2 new ones<br>
>> currently being proposed, there are a lot of people knocking at the door.<br>
><br>
><br>
><br>
> I have a slightly different short term opinion on what 'OpenStack' should<br>
> be: it should work really well.<br>
><br>
> While we need to figure out howto increase our strategic resources,<br>
> realistically I think that will still be the limiting factor in Kilo. So in<br>
> order to better allocate our cross project strategic resources, I think we<br>
> should do a project level triage ('identify projects that are likely to die,<br>
> regardless of what care they receive' to paraphrase the medical definition<br>
> of the term), and tighten our focus until we get our development process<br>
> into a better state.<br>
><br>
><br>
>><br>
>><br>
>> Thanks for reading this braindump this far. I hope this will trigger the<br>
>> open discussions we need to have, as an open source project, to reach<br>
>> the next level.<br>
><br>
><br>
> Thank you for bringing this topic up.<br>
><br>
>><br>
>><br>
>> Cheers,<br>
>><br>
>> --<br>
>> Thierry Carrez (ttx)<br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div></div></div><br></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>