Open Stack

On August 26, 2014, Anne Gentle wrote:
On Mon, Aug 25, 2014 at 8:36 AM, Sean Dague <sean at dague.net<mailto:sean at dague.net>> wrote:
On 08/20/2014 12:37 PM, Zane Bitter wrote:
> On 11/08/14 05:24, Thierry Carrez wrote:
>> So the idea that being (and remaining) in the integrated release should
>> also be judged on technical merit is a slightly different effort. It's
>> always been a factor in our choices, but like Devananda says, it's more
>> difficult than just checking a number of QA/integration checkboxes. In
>> some cases, blessing one project in a problem space stifles competition,
>> innovation and alternate approaches. In some other cases, we reinvent
>> domain-specific solutions rather than standing on the shoulders of
>> domain-specific giants in neighboring open source projects.
>
> I totally agree that these are the things we need to be vigilant about.
>
> Stifling competition is a big worry, but it appears to me that a lot of
> the stifling is happening even before incubation. Everyone's time is
> limited, so if you happen to notice a new project on the incubation
> trajectory doing things in what you think is the Wrong Way, you're most
> likely to either leave some drive-by feedback or to just ignore it and
> carry on with your life. What you're most likely *not* to do is to start
> a competing project to prove them wrong, or to jump in full time to the
> existing project and show them the light. It's really hard to argue
> against the domain experts too - when you're acutely aware of how
> shallow your knowledge is in a particular area it's very hard to know
> how hard to push. (Perhaps ironically, since becoming a PTL I feel I
> have to be much more cautious in what I say too, because people are
> inclined to read too much into my opinion - I wonder if TC members feel
> the same pressure.) I speak from first-hand instances of guilt here -
> for example, I gave some feedback to the Mistral folks just before the
> last design summit[1], but I haven't had time to follow it up at all. I
> wouldn't be a bit surprised if they showed up with an incubation
> request, a largely-unchanged user interface and an expectation that I
> would support it.
>
> The result is that projects often don't hear the feedback they need
> until far too late - often when they get to the incubation review (maybe
> not even their first incubation review). In the particularly unfortunate
> case of Marconi, it wasn't until the graduation review. (More about that
> in a second.) My best advice to new projects here is that you must be
> like a ferret up the pant-leg of any negative feedback. Grab hold of any
> criticism and don't let go until you have either converted the person
> giving it into your biggest supporter, been converted by them, or
> provoked them to start a competing project. (Any of those is a win as
> far as the community is concerned.)
>
> Perhaps we could consider a space like a separate mailing list
> (openstack-future?) reserved just for announcements of Related projects,
> their architectural principles, and discussions of the same?  They
> certainly tend to get drowned out amidst the noise of openstack-dev.
> (Project management, meeting announcements, and internal project
> discussion would all be out of scope for this list.)
>
> As for reinventing domain-specific solutions, I'm not sure that happens
> as often as is being made out. IMO the defining feature of IaaS that
> makes the cloud the cloud is on-demand (i.e. real-time) self-service.
> Everything else more or less falls out of that requirement, but the very
> first thing to fall out is multi-tenancy and there just aren't that many
> multi-tenant services floating around out there. There are a couple of
> obvious strategies to deal with that: one is to run existing software
> within a tenant-local resource provisioned by OpenStack (Trove and
> Sahara are examples of this), and the other is to wrap a multi-tenancy
> framework around an existing piece of software (Nova and Cinder are
> examples of this). (BTW the former is usually inherently less
> satisfying, because it scales at a much coarser granularity.) The answer
> to a question of the form:
>
> "Why do we need OpenStack project $X, when open source project $Y
> already exists?"
>
> is almost always:
>
> "Because $Y is not multi-tenant aware; we need to wrap it with a
> multi-tenancy layer with OpenStack-native authentication, metering and
> quota management. That even allows us to set up an abstraction layer so
> that you can substitute $Z as the back end too."
>
> This is completely uncontroversial when you substitute X, Y, Z = Nova,
> libvirt, Xen. However, when you instead substitute X, Y, Z =
> Zaqar/Marconi, Qpid, MongoDB it suddenly becomes *highly* controversial.
> I'm all in favour of a healthy scepticism, but I think we've passed that
> point now. (How would *you* make an AMQP bus multi-tenant?)
>
> To be clear, Marconi did made a mistake. The Marconi API presented
> semantics to the user that excluded many otherwise-obvious choices of
> back-end plugin (i.e. Qpid/RabbitMQ). It seems to be a common thing (see
> also: Mistral) to want to design for every feature an existing
> Enterprisey application might use, which IMHO kind of ignores the fact
> that applications need to be rewritten to use these new APIs anyway, and
> we would be better off presenting _simple_ interfaces that attract
> developers, lead to good design for new applications and provide
> flexibility on the back-end side. I digress though, because that wasn't
> the mistake. The mistake was failing to educate the entire community
> (but in particular the TC) on the relative merits of this feature enough
> get either buy-in or rejection on this critical detail much earlier in
> the process.
>
> By the way, I claimed without justification earlier that "there just
> aren't that many multi-tenant services floating around out there", but
> there is actually a very good reason for that. Before there were open
> source IaaS platforms out there, there wasn't any reason to build such a
> service. And now that there are, _the obvious place to build such a
> service in in OpenStack_. We have the users, we have the authentication
> API that already ties in to their existing directories (Keystone is
> well-named) and we have the community.
I think this is a very crisp view of what OpenStack is trying to do. I
like it. :)

>> This all has created a world where you need to be*in*  OpenStack to
>> matter, or to justify the investment. This has created a world where
>> everything and everyone wants to be in the "OpenStack" integrated
>> release. This has created more pressure to add new projects, and less
>> pressure to fix and make the existing projects perfect. 4 years in, we
>> might want to inflect that trajectory and take steps to fix this world.
>
> We should certainly consider this possibility, that we've set up
> perverse incentives leading to failure. But what if it's just because we
> haven't yet come even close to satisfying all of our users' needs? I
> mean, AWS has more than 30 services that could be considered equivalent
> in scope to an OpenStack project... if anything our scope is increasing
> more _slowly_ than the industry at large. I'm slightly shocked that
> nobody in this thread appears to have even entertained the idea that
> *this is what success looks like*.
That's a fair point of view. However I'm sure the AWS team is staffed a
bit differently where the feature implementers don't outnumber the
QA/Docs/Infra folks by a factor for 10 to 1.


With 1200 overall OpenStack unique contributors, we had 195 docs contributors. What's good about that metric is it shows over 16% contributors are involved in docs. However, about 10 individual contributors wrote half the docs patches for Icehouse. So we still struggle as a core team.

Scoping the docs efforts very tightly is the current method for maintaining docs. That said, I spent about a day last week writing up incubating and integrating docs requirements. [1] After that exercise I sense that our doc scaling struggles will continue but that the program's teams will step up. My efforts around simplifying the docs contribution path are stalled a bit, but that's the second method to work on keeping up.

All that said, my current line of thinking about adding programs is that we need to enable more options in each program, even if I personally have no idea how to keep up with the level of docs quality, accuracy, reviews, and so on. :)

By having the TC enable innovation and competition within programs we can prove we are not locking anyone in to one integration grouping. With our open source ethos, we need to work harder to enable projects that are optional. Yes, this stance flies in the face of "but how will we support, test, document unless we pick a happy path?" Believe me, I know this conundrum well.

I like to think of projects as LEGO bricks that interlock optionally. I want us to think like we're making LEGO Creator sets -- you get to build three different LEGO animals (eagle, scorpion, beaver [2]) from one kit. For an OpenStack example, we need a way for TripleO, chef, and puppet experts to be willing to work in a common way to further a Deployment program's goals. Same for monitoring and probably other programs. Then the TC isn't picking the "only" way to meet those goals.

[Rocky Grober]
Thank you, Ann for saying so concisely (and with a good analogy) how I view OpenStack should be.  OpenStack defines the rules for interactions and presents none, one or many ways to compose the stack that is your cloud, but it doesn’t pick winners or losers, just actively developed code that meets its standards, or helps those developing the code to understand how to meet its standards.

Yay, Ann!

And thank you,
--Rocky

Thanks for reading this lonnng thread. At least you get LEGO links at the very end.
Anne

1. https://wiki.openstack.org/wiki/Documentation/IncubateIntegrate
2. http://shop.lego.com/en-US/Fierce-Flyer-31004



I'm also sure there are a lot more dedicated folks to cross project
effort than we see today.

If we weren't so feature biased in our effort, 30 services might be a
sensible target. But the lack of folks working on some of the holistic /
cross project efforts means I don't think it's really viable.

> The world is not going to stop because we want to get off, take a
> breather, do a "consolidation cycle".
Most deployers I've talked to would actually love for OpenStack to slow
down a bit. They are at least a year behind on their deployments, and a
stabalization cycle is something they'd lvove to see.

> I think we can all agree that any cross-project function program that
> aims to _be_ that function for all of OpenStack, as opposed to enabling
> that function in the individual projects, will not scale. It won't scale
> even if we never accept another new project. It won't scale even if we
> de-integrate some projects pour encourager les autres. I think that the
> work that Doug has been doing with Oslo points the way here. Every
> cross-project function (including release management - get rid of PTLs)
> should have a designated co-ordinator/liason in each project.
This becomes an N^2 communication problem and eventually consistency
problem.

> The move of functional tests out of Tempest and into the individual
> projects' repositories is a great step in the right direction. Hopefully
> that will also address the growth of CI testing capacity requirements -
> while projects like Nova, Cinder and Neutron all depend on and benefit
> from being gated against each other, that is not at all true for many
> projects, and in particular the kinds of new projects that are now
> appearing. Let's take actual dependencies into account when we choose
> what to gate on - the resulting growth might not be sublinear, but it
> needn't be n^2 either.
>
> Simply put, I don't think we're anywhere near so close to running out of
> ideas that we have to stop and freeze the world. We have scaling
> challenges, and we'll have to make changes, and we'll have to do it -
> sometimes under pressure - even as the world changes around us because
> that's just how life works.
>
> The question that is not being asked enough IMO is how exactly does it
> benefit our *users* (it is still all about the users, right?) to be
> telling developers "you arrived too late, so you're not real OpenStack
> developers, with free conference tickets and ATC badges and stuff".
> Because if it's just that the support functions within OpenStack can't
> handle them yet, that's on us not them. If it's that we don't have a
> clear way of communicating to users the information they need to make
> the right decision (for them) on whether to deploy some code, then we
> need to communicate that information more clearly. And if we believe
> that somehow companies will magically divert all of the resources they
> would otherwise have committed to new projects into bug fixing on
> existing projects, I think we're being incredibly naive. If companies
> are not seeing the value in strategically contributing to the core of
> projects then it's because we have failed to demonstrate the value, and
> simply blocking off other avenues for them to create value (or blocking
> recognition of those avenues) will absolutely not solve it.
I also think it's completely fair to think about this as small tent,
large ecosystem. The Linux Kernel stops at a certain function point for
a reason. That didn't prevent a ton of exceptionally useful function to
exist above it. But it also mean that things like c library / init
system / packaging / compilers were things where innovation could play
out. And where things could evolve over time.

I don't think small test / big ecosystem prevents anyone from working on
what they are passionate about. It does bring the value of n in n^2
relationships to something tractable.

Maybe there are better people to handle the cross project interactions
that can handle the value of n we currently have, or the value of n we
might have in the future. But I'm watching the strain on the existing
teams doing this and I think under current course and speed we mostly
will lose key leadership (under burn out), and not grow more of it. And
that's a really important thing to consider in keeping something like
OpenStack a functioning base. If the replacement rate of key leadership
does not exceed the fall off rate we won't end up in a happy place in a
few more cycles.

        -Sean

--
Sean Dague
http://dague.net
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140827/080645fd/attachment.html>