Open Stack

Devananda van der Veen, on  January 21, 2016 5:14 PM wrote:

On Wed, Jan 20, 2016 at 9:53 AM, Flavio Percoco <flavio at redhat.com<mailto:flavio at redhat.com>> wrote:
Greetings,

At the Tokyo summit, we discussed OpenStack's development themes in a
cross-project session. In this session a group of folks started discussing what
topics the overall community could focus on as a shared effort. One of the
things that was raised during this session is the need of having cycles to
stabilize projects. This was brought up by Robert Collins again in a meeting[0]
the TC had right after the summit and no much has been done ever since.

Now, "stabilization Cycles" are easy to dream about but really hard to do and
enforce. Nonetheless, they are still worth a try or, at the very least, a
thought. I'll try to go through some of the issues and benefits a stabilization
cycle could bring but bear in mind that the lists below are not exhaustive. In
fact, I'd love for other folks to chime in and help building a case in favor or
against this.

Negative(?) effects
===================

- Project won't get new features for a period of time Economic impact on
 developers(?)
- It was mentioned that some folks receive bonuses for landed features
- Economic impact on companies/market because no new features were added (?)
- (?)

Positive effects
================

- Focus on bug fixing
- Reduce review backlog
- Refactor *existing* code/features with cleanups
- Focus on multi-cycle features (if any) and complete those
- (?)

A stabilization cycle, as it was also discussed in the aforementioned
meeting[0], doesn't need to be all or nothing. For instance, it should be
perfectly fine for a project to say that a project would dedicate 50% of the
cycle to stabilization and the rest to complete some pending features. Moreover,
each project is free to choose when/if a stabilization cycle would be good for
it or not.

For example, the Glance team is currently working on refactoring the image
import workflow. This is a long term effort that will require at least 2 cycles
to be completed. Furthermore, it's very likely these changes will introduce bugs
and that will require further work. If the Glance team would decide (this is not
an actual proposal... yet :) to use Newton as a stabilization cycle, the team
would be able to focus all its forces on fixing those bugs, completing the
feature and tackling other, long-term, pending issues. In the case of Glance,
this would impact *only glance* and not other projects under the Glance team
umbrella like glanceclient and glance_store. In fact, this would be a perfect
time for the glance team to dedicate time to improving glanceclient and catch up
with the server side latest changes.

So, the above sounds quite vague, still but that's the idea. This email is not a
formal proposal but a starting point to move this conversation forward. Is this
something other teams would be interested in? Is this something some teams would
be entirely against? Why?

From a governance perspective, projects are already empowered to do this and
they don't (and won't) need to be granted permission to have stabilization
cycles. However, the TC could work on formalizing this process so that teams
have a reference to follow when they want to have one. For example, we would
have to formalize how projects announce they want to have a stabilization cycle
(I believe it should be done before the mid-term of the ongoing cycle).

Thoughts? Feedback?
Flavio


Thanks for writing this up, Flavio.

The topic's come up in smaller discussion groups several times over the last few years, mostly with a nod to "that would be great, except the corporations won't let it happen".

To everyone who's replied with shock to this thread, the reality is that nearly all of the developer-hours which fuel OpenStack's progress are funded directly by corporations, whether big or small. Even those folks who have worked in open source for a long time, and are working on OpenStack by choice, are being paid by companies deeply invested in the success of this project. Some developers are adept at separating the demands of their employer from the best interests of the community. Some are not. I don't have hard data, but I suspect that most of the nearly-2000 developers who have contributed to OpenStack during the Mitaka cycle are working on what ever they're working on BECAUSE IT MATTERS TO THEIR EMPLOYER.

Every project experiences pressure from companies who are trying to land very specific features. Why? Because they're all chasing first-leader advantage in the market. Lots of features are in flight right now, and, even though it sounds unbelievable, many companies announce those features in their products BEFORE they actually land upstream. Crazy, right? Except... IT WORKS. Other companies buy their product because they are buying a PRODUCT from some company. It happens to contain OpenStack. And it has a bunch of unmerged features.

With my open source hat on, and with my community hat on, I think this is terrible.

Halting the acceptance of features upstream will encourage this "bad behavior" -- because, today, it is what so many of the companies' finances are counting on for revenue. They're going to do it anyway, because they have to.

We'll see the pressure on review teams and PTLs temporarily increase near the end of the cycle before the "stabilization cycle" -- and then we'll see a lot of developers get pulled downstream for a while. Probably the whole cycle. And then at the start of the next cycle, a bunch of code will come "flying over the wall" from companies who have already been shipping it.

Yes, this is crazy. Yes, I've been rallying against it within my past (and current) employer. But it's also how SO MANY OTHER companies are operating, too.

And just to be clear, I'm all for a stabilization cycle.

--devananda

Thanks for this, Devananda.

Yes, most of our employers put pressure on us to get features our customers want into OpenStack.  And this makes stabilization cycles extremely hard.  But, let's look at it a bit more broadly, as Devananda and others have already written about.

Stabilization is more than just bug fixes.  In fact, if all it were was just bug fixes, it's unlikely that the code would get all that much more stable.  Stabilization *should* encompass all of those "nonfunctional" requirements that play such a large role in helping users decide whether the code/distro/public cloud is good or bad.  And don't be mistaken, most of these pain points hit our corporate patrons as hard or harder than our endusers.  Remember, if someone is getting a distro from a company, they expect the company to jump when they have a problem.  The more bugs, races, inconsistencies, install issues, etc. those customers have, the more the company has to spend on technical support engineers, escalation engineers, etc.  And, for every patch on a private branch they have to manage, that's QA engineers and developers, and documentation and trainers.  So, certain "stability" fixes, when providing significant reduction of private support expenses, will be considered valuable by our corporate sponsors.

Secondly, it's true about not having to have the whole cycle focus on stability.  Just as features are tradeoffs, nonfunctional code sets are tradeoffs.  You evaluate both on return on the effort.    How do you get to the tradeoff point?  You take a user pain point in your project -- a racy section, a performance issue,  a section of code with lots of bugs filed against it -- and you write a spec on how you would fix the issue, what the alternatives are, risks, advantages, downsides, etc.  Just like we are doing in this email thread with regard to stability cycles.    Once you have the spec, or a collection of specs, that address user pain points, or even developer pain points ("I am so sick of patching that section of code"), then you can judge all the specs, features and non-features, on their value and whether you have developers willing to tackle them.

Now, with a focus on bugs, those who are closest to the existing bugs, and have been fixing, triaging, reviewing fixes, etc. will have a deeper level of understanding of where the deeper issues within the code base are.  They will see the areas that keep generating bugs.  They will see the collection of bugs that are all different manifestations of the same problems.  Those developers who have been deep in the bugs and bug systems, if they get together and analyze what they know about the projects bugs, can categorize and focus specs on areas that will eliminate bugs through rearchitecting and refactoring (replacement), or just refactoring, or clarification, documentation and fixing of cross project interactions.  A stabilization effort should not simply be a massive bug swatting endeavor.

So, go back to your product managers, or your corporate QA teams, or the support teams and ask them what their biggest pain points and gripes are with OpenStack, or with an OpenStack project.  Then let's gather all this info up, have an upstream session that is cross-project to get everyone understanding where the pain is, and break the points down into approachable chunks so that you can spec them (or start the spec process) during the summit and present them on an even level with the feature specs.  Then you can decide how much of a development cycle is focused on "stabilization" and how much on new features.

And be thankful we have companies willing to support the Robert Collins annd Doug Hellmanns and Davinum (Dims) (and we have the developers willing to do the yeoman's work) of OpenStack as they have been slogging away at making this whole OpenSource project continue to work and get better, without any "monetizing features" in their contributions.

</rant>

--Rocky


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160122/a581a3c9/attachment.html>