<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="Section1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;
font-family:"Tahoma","sans-serif"">Devananda van der Veen, on January 21, 2016 5:14 PM wrote:</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">On Wed, Jan 20, 2016 at 9:53 AM, Flavio Percoco <<a href="mailto:flavio@redhat.com" target="_blank">flavio@redhat.com</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:
12.0pt;margin-left:.5in">
Greetings,<br>
<br>
At the Tokyo summit, we discussed OpenStack's development themes in a<br>
cross-project session. In this session a group of folks started discussing what<br>
topics the overall community could focus on as a shared effort. One of the<br>
things that was raised during this session is the need of having cycles to<br>
stabilize projects. This was brought up by Robert Collins again in a meeting[0]<br>
the TC had right after the summit and no much has been done ever since.<br>
<br>
Now, "stabilization Cycles" are easy to dream about but really hard to do and<br>
enforce. Nonetheless, they are still worth a try or, at the very least, a<br>
thought. I'll try to go through some of the issues and benefits a stabilization<br>
cycle could bring but bear in mind that the lists below are not exhaustive. In<br>
fact, I'd love for other folks to chime in and help building a case in favor or<br>
against this.<br>
<br>
Negative(?) effects<br>
===================<br>
<br>
- Project won't get new features for a period of time Economic impact on<br>
developers(?)<br>
- It was mentioned that some folks receive bonuses for landed features<br>
- Economic impact on companies/market because no new features were added (?)<br>
- (?)<br>
<br>
Positive effects<br>
================<br>
<br>
- Focus on bug fixing<br>
- Reduce review backlog<br>
- Refactor *existing* code/features with cleanups<br>
- Focus on multi-cycle features (if any) and complete those<br>
- (?)<br>
<br>
A stabilization cycle, as it was also discussed in the aforementioned<br>
meeting[0], doesn't need to be all or nothing. For instance, it should be<br>
perfectly fine for a project to say that a project would dedicate 50% of the<br>
cycle to stabilization and the rest to complete some pending features. Moreover,<br>
each project is free to choose when/if a stabilization cycle would be good for<br>
it or not.<br>
<br>
For example, the Glance team is currently working on refactoring the image<br>
import workflow. This is a long term effort that will require at least 2 cycles<br>
to be completed. Furthermore, it's very likely these changes will introduce bugs<br>
and that will require further work. If the Glance team would decide (this is not<br>
an actual proposal... yet :) to use Newton as a stabilization cycle, the team<br>
would be able to focus all its forces on fixing those bugs, completing the<br>
feature and tackling other, long-term, pending issues. In the case of Glance,<br>
this would impact *only glance* and not other projects under the Glance team<br>
umbrella like glanceclient and glance_store. In fact, this would be a perfect<br>
time for the glance team to dedicate time to improving glanceclient and catch up<br>
with the server side latest changes.<br>
<br>
So, the above sounds quite vague, still but that's the idea. This email is not a<br>
formal proposal but a starting point to move this conversation forward. Is this<br>
something other teams would be interested in? Is this something some teams would<br>
be entirely against? Why?<br>
<br>
From a governance perspective, projects are already empowered to do this and<br>
they don't (and won't) need to be granted permission to have stabilization<br>
cycles. However, the TC could work on formalizing this process so that teams<br>
have a reference to follow when they want to have one. For example, we would<br>
have to formalize how projects announce they want to have a stabilization cycle<br>
(I believe it should be done before the mid-term of the ongoing cycle).<br>
<br>
Thoughts? Feedback?<br>
Flavio<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Thanks for writing this up, Flavio.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">The topic's come up in smaller discussion groups several times over the last few years, mostly with a nod to "that would be great, except the corporations won't let it happen".<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">To everyone who's replied with shock to this thread, the reality is that nearly all of the developer-hours which fuel OpenStack's progress are funded directly by corporations, whether big or small. Even those folks
who have worked in open source for a long time, and are working on OpenStack by choice, are being paid by companies deeply invested in the success of this project. Some developers are adept at separating the demands of their employer from the best interests
of the community. Some are not. I don't have hard data, but I suspect that most of the nearly-2000 developers who have contributed to OpenStack during the Mitaka cycle are working on what ever they're working on BECAUSE IT MATTERS TO THEIR EMPLOYER.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Every project experiences pressure from companies who are trying to land very specific features. Why? Because they're all chasing first-leader advantage in the market. Lots of features are in flight right now, and,
even though it sounds unbelievable, many companies announce those features in their products BEFORE they actually land upstream. Crazy, right? Except... IT WORKS. Other companies buy their product because they are buying a PRODUCT from some company. It happens
to contain OpenStack. And it has a bunch of unmerged features.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">With my open source hat on, and with my community hat on, I think this is terrible.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Halting the acceptance of features upstream will encourage this "bad behavior" -- because, today, it is what so many of the companies' finances are counting on for revenue. They're going to do it anyway, because
they have to.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">We'll see the pressure on review teams and PTLs temporarily increase near the end of the cycle before the "stabilization cycle" -- and then we'll see a lot of developers get pulled downstream for a while. Probably
the whole cycle. And then at the start of the next cycle, a bunch of code will come "flying over the wall" from companies who have already been shipping it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Yes, this is crazy. Yes, I've been rallying against it within my past (and current) employer. But it's also how SO MANY OTHER companies are operating, too.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">And just to be clear, I'm all for a stabilization cycle.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">--devananda <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Thanks for this, Devananda.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Yes, most of our employers put pressure on us to get features our customers want into OpenStack. And this makes stabilization cycles extremely hard. But,
let's look at it a bit more broadly, as Devananda and others have already written about.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Stabilization is more than just bug fixes. In fact, if all it were was just bug fixes, it's unlikely that the code would get all that much more stable.
Stabilization *<b>should</b>* encompass all of those "nonfunctional" requirements that play such a large role in helping users decide whether the code/distro/public cloud is good or bad. And don't be mistaken, most of these pain points hit our corporate patrons
as hard or harder than our endusers. Remember, if someone is getting a distro from a company, they expect the company to jump when they have a problem. The more bugs, races, inconsistencies, install issues, etc. those customers have, the more the company
has to spend on technical support engineers, escalation engineers, etc. And, for every patch on a private branch they have to manage, that's QA engineers and developers, and documentation and trainers. So, certain "stability" fixes, when providing significant
reduction of private support expenses, will be considered valuable by our corporate sponsors.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Secondly, it's true about not having to have the whole cycle focus on stability. Just as features are tradeoffs, nonfunctional code sets are tradeoffs.
You evaluate both on return on the effort. How do you get to the tradeoff point? You take a user pain point in your project -- a racy section, a performance issue, a section of code with lots of bugs filed against it -- and you write a spec on how you
would fix the issue, what the alternatives are, risks, advantages, downsides, etc. Just like we are doing in this email thread with regard to stability cycles. Once you have the spec, or a collection of specs, that address user pain points, or even developer
pain points ("I am so sick of patching that section of code"), then you can judge all the specs, features and non-features, on their value and whether you have developers willing to tackle them.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Now, with a focus on bugs, those who are closest to the existing bugs, and have been fixing, triaging, reviewing fixes, etc. will have a deeper level of understanding
of where the deeper issues within the code base are. They will see the areas that keep generating bugs. They will see the collection of bugs that are all different manifestations of the same problems. Those developers who have been deep in the bugs and
bug systems, if they get together and analyze what they know about the projects bugs, can categorize and focus specs on areas that will eliminate bugs through rearchitecting and refactoring (replacement), or just refactoring, or clarification, documentation
and fixing of cross project interactions. A stabilization effort should not simply be a massive bug swatting endeavor.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">So, go back to your product managers, or your corporate QA teams, or the support teams and ask them what their biggest pain points and gripes are with OpenStack,
or with an OpenStack project. Then let's gather all this info up, have an upstream session that is cross-project to get everyone understanding where the pain is, and break the points down into approachable chunks so that you can spec them (or start the spec
process) during the summit and present them on an even level with the feature specs. Then you can decide how much of a development cycle is focused on "stabilization" and how much on new features.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">And be thankful we have companies willing to support the Robert Collins annd Doug Hellmanns and Davinum (Dims) (and we have the developers willing to do the
yeoman's work) of OpenStack as they have been slogging away at making this whole OpenSource project continue to work and get better, without any "monetizing features" in their contributions.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"></rant><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">--Rocky<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>