Open Stack

Hi Sean,

I'm going to top post because my response is general. I totally agree that we need people that understand the code base and we should encourage new people to be cross-functional. I guess my main issue is with how we get there. I believe in encouragment over punishment. In my mind giving people autonomy and control encourages them to contribute more. 

In my opinion giving the driver developers control over their own code will lead to higher quality drivers. Yes, we risk integration issues, lack of test coverage, and buggy implementations, but it is my opinion that the increased velocity that the developers will enjoy will mean faster bug fixes and more opportunity to improve the drivers.

I also think lowering the amount of code that nova-core has to keep an eye on will improve the review velocity of the rest of the code as well.

Vish

On Oct 15, 2013, at 4:36 PM, Sean Dague <sean at dague.net> wrote:

> On 10/15/2013 04:54 PM, Vishvananda Ishaya wrote:
>> Hi Everyone,
>> 
>> I've been following this conversation and weighing the different sides. This is a tricky issue but I think it is important to decouple further and extend our circle of trust.
>> 
>> When nova started it was very easy to do feature development. As it has matured the pace has slowed. This is expected and necessary, but we periodically must make decoupling decisions or we will become mired in overhead. We did this already with cinder and neutron, and we have discussed doing this with virt drivers in the past.
>> 
>> We have a large number of people attempting to contribute to small sections of nova and getting frustrated with the process.  The perception of developers is much more important than the actual numbers here. If people are frustrated they are disincentivized to help and it hurts everyone. Suggesting that these contributors need to learn all of nova and help with the review queue is silly and makes us seem elitist. We should make it as easy as possible for new contributors to help.
>> 
>> I think our current model is breaking down at our current size and we need to adopt something more similar to the linux model when dealing with subsystems. The hyper-v team is the only one suggesting changes, but there have been similar concerns from the vmware team. I have no doubt that there are similar issues with the PowerVM, Xen, Docker, lxc and even kvm driver contributors.
> 
> The Linux kernel process works for a couple of reasons...
> 
> 1) the subsystem maintainers have known each other for a solid decade (i.e. 3x the lifespan of the OpenStack project), over a history of 10 years, of people doing the right things, you build trust in their judgment.
> 
> *no one* in the Linux tree was given trust first, under the hope that it would work out. They had to earn it, hard, by doing community work, and not just playing in their corner of the world.
> 
> 2) This http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ is completely acceptable behavior. So when someone has bad code, they are flamed to within an inch of their life, repeatedly, until they never ever do that again. This is actually a time saving measure in code review. It's a lot faster to just call people idiots then to help them with line by line improvements in their code, 10, 20, 30, or 40 iterations in gerrit.
> 
> We, as a community have decided, I think rightly, that #2 really isn't in our culture. But you can't start cherry picking parts of the Linux kernel community without considering how all the parts work together. The good and the bad are part of why the whole system works.
> 
>> In my opinion, nova-core needs to be willing to trust the subsystem developers and let go of a little bit of control. I frankly don't see the drawbacks.
> 
> I actually see huge draw backs. Culture matters. Having people active and willing to work on real core issues matter. The long term health of Nova matters.
> 
> As the QA PTL I can tell you that when you look at Nova vs. Cinder vs. Neutron, you'll see some very clear lines about how long it takes to get to the bottom of a race condition, and how many deep races are in each of them. I find this directly related to the stance each project has taken on whether it's socially acceptable to only work on your own vendor code. Nova's insistence up until this point that if you only play in your corner, you don't get the same attention is important incentive for people to integrate and work beyond just their boundaries. I think diluting this part of the culture would be hugely detrimental to Nova.
> 
> Let's take an example that came up today, the compute_diagnostics API. This is an area where we've left it completely to the virt drivers to vomit up a random dictionary of the day for debugging reasons, and stamped it as an API. With a model where we let virt driver authors go hide in a corner, that's never going to become an API with any kind of contract, and given how much effort we've spent on ensuring RPC versioning and message formats, the idea that we are exposing a public rest endpoint that's randomly fluctuating data based on date and underlying implementation, is a bit saddening.
> 
>> I'm leaning towards giving control of the subtree to the team as the best option because it is simple and works with our current QA system. Alternatively, we could split out the driver into a nova subproject (2 below) or we could allow them to have a separate branch and do a trusted merge of all changes at the end of the cycle (similar to the linux model).
>> 
>> I hope we can come to a solution to the summit that makes all of our contributors want to participate more. I believe that giving people more responsibility inspires them to participate more fully.
> 
> I would like nothing more than all our contributors to participate more. But more has to mean caring about not only your stuff.
> 
> I was called out today in the hyper-v meeting because I had the audacity to -1 a hyper-v patch because I wanted some reference in the code somewhere to format references so why we had some new random seek call would be understood by people down the road - http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-10-15-16.03.log.html
> 
> As OpenStack grows, the single biggest factor in it's success isn't going to be a feature in a driver, it's going to be if this crazy complicated system holds together. Whether or not we've got a handle on the emergent behavior that happens in an asynchronous message based system, with 10s of integrated projects, and many dozens of daemons cross talking with each other.
> 
> I mean seriously, one of the only reasons we made it through to Havana RC phase is because we built a search engine based system to build statistical frequency analysis of unique failures on our gate resets to fully expose the top race conditions that had gotten so bad the gate basically locked up. And a bunch of people went all hands on deck to drive these out. People jumped across normal project lines to help on some of these top bugs, because that's what makes OpenStack a whole system.
> 
> Things actually looked *really* bleak for release for a while. All the people that helped out and got us through this deserve a huge pat on the back. That's what OpenStack is about.
> 
> So I feel pretty strongly that optimizing the contribution process for people that aren't helping with that larger problem, is the tragedy of the commons, and I think entirely the wrong optimization to be made.
> 
>    -Sean
> 
> -- 
> Sean Dague
> http://dague.net