Open Stack

On 08/20/2014 12:37 PM, Zane Bitter wrote:
> On 11/08/14 05:24, Thierry Carrez wrote:
>> So the idea that being (and remaining) in the integrated release should
>> also be judged on technical merit is a slightly different effort. It's
>> always been a factor in our choices, but like Devananda says, it's more
>> difficult than just checking a number of QA/integration checkboxes. In
>> some cases, blessing one project in a problem space stifles competition,
>> innovation and alternate approaches. In some other cases, we reinvent
>> domain-specific solutions rather than standing on the shoulders of
>> domain-specific giants in neighboring open source projects.
> 
> I totally agree that these are the things we need to be vigilant about.
> 
> Stifling competition is a big worry, but it appears to me that a lot of
> the stifling is happening even before incubation. Everyone's time is
> limited, so if you happen to notice a new project on the incubation
> trajectory doing things in what you think is the Wrong Way, you're most
> likely to either leave some drive-by feedback or to just ignore it and
> carry on with your life. What you're most likely *not* to do is to start
> a competing project to prove them wrong, or to jump in full time to the
> existing project and show them the light. It's really hard to argue
> against the domain experts too - when you're acutely aware of how
> shallow your knowledge is in a particular area it's very hard to know
> how hard to push. (Perhaps ironically, since becoming a PTL I feel I
> have to be much more cautious in what I say too, because people are
> inclined to read too much into my opinion - I wonder if TC members feel
> the same pressure.) I speak from first-hand instances of guilt here -
> for example, I gave some feedback to the Mistral folks just before the
> last design summit[1], but I haven't had time to follow it up at all. I
> wouldn't be a bit surprised if they showed up with an incubation
> request, a largely-unchanged user interface and an expectation that I
> would support it.
> 
> The result is that projects often don't hear the feedback they need
> until far too late - often when they get to the incubation review (maybe
> not even their first incubation review). In the particularly unfortunate
> case of Marconi, it wasn't until the graduation review. (More about that
> in a second.) My best advice to new projects here is that you must be
> like a ferret up the pant-leg of any negative feedback. Grab hold of any
> criticism and don't let go until you have either converted the person
> giving it into your biggest supporter, been converted by them, or
> provoked them to start a competing project. (Any of those is a win as
> far as the community is concerned.)
> 
> Perhaps we could consider a space like a separate mailing list
> (openstack-future?) reserved just for announcements of Related projects,
> their architectural principles, and discussions of the same?  They
> certainly tend to get drowned out amidst the noise of openstack-dev.
> (Project management, meeting announcements, and internal project
> discussion would all be out of scope for this list.)
> 
> As for reinventing domain-specific solutions, I'm not sure that happens
> as often as is being made out. IMO the defining feature of IaaS that
> makes the cloud the cloud is on-demand (i.e. real-time) self-service.
> Everything else more or less falls out of that requirement, but the very
> first thing to fall out is multi-tenancy and there just aren't that many
> multi-tenant services floating around out there. There are a couple of
> obvious strategies to deal with that: one is to run existing software
> within a tenant-local resource provisioned by OpenStack (Trove and
> Sahara are examples of this), and the other is to wrap a multi-tenancy
> framework around an existing piece of software (Nova and Cinder are
> examples of this). (BTW the former is usually inherently less
> satisfying, because it scales at a much coarser granularity.) The answer
> to a question of the form:
> 
> "Why do we need OpenStack project $X, when open source project $Y
> already exists?"
> 
> is almost always:
> 
> "Because $Y is not multi-tenant aware; we need to wrap it with a
> multi-tenancy layer with OpenStack-native authentication, metering and
> quota management. That even allows us to set up an abstraction layer so
> that you can substitute $Z as the back end too."
> 
> This is completely uncontroversial when you substitute X, Y, Z = Nova,
> libvirt, Xen. However, when you instead substitute X, Y, Z =
> Zaqar/Marconi, Qpid, MongoDB it suddenly becomes *highly* controversial.
> I'm all in favour of a healthy scepticism, but I think we've passed that
> point now. (How would *you* make an AMQP bus multi-tenant?)
> 
> To be clear, Marconi did made a mistake. The Marconi API presented
> semantics to the user that excluded many otherwise-obvious choices of
> back-end plugin (i.e. Qpid/RabbitMQ). It seems to be a common thing (see
> also: Mistral) to want to design for every feature an existing
> Enterprisey application might use, which IMHO kind of ignores the fact
> that applications need to be rewritten to use these new APIs anyway, and
> we would be better off presenting _simple_ interfaces that attract
> developers, lead to good design for new applications and provide
> flexibility on the back-end side. I digress though, because that wasn't
> the mistake. The mistake was failing to educate the entire community
> (but in particular the TC) on the relative merits of this feature enough
> get either buy-in or rejection on this critical detail much earlier in
> the process.
> 
> By the way, I claimed without justification earlier that "there just
> aren't that many multi-tenant services floating around out there", but
> there is actually a very good reason for that. Before there were open
> source IaaS platforms out there, there wasn't any reason to build such a
> service. And now that there are, _the obvious place to build such a
> service in in OpenStack_. We have the users, we have the authentication
> API that already ties in to their existing directories (Keystone is
> well-named) and we have the community.

I think this is a very crisp view of what OpenStack is trying to do. I
like it. :)

>> This all has created a world where you need to be*in*  OpenStack to
>> matter, or to justify the investment. This has created a world where
>> everything and everyone wants to be in the "OpenStack" integrated
>> release. This has created more pressure to add new projects, and less
>> pressure to fix and make the existing projects perfect. 4 years in, we
>> might want to inflect that trajectory and take steps to fix this world.
> 
> We should certainly consider this possibility, that we've set up
> perverse incentives leading to failure. But what if it's just because we
> haven't yet come even close to satisfying all of our users' needs? I
> mean, AWS has more than 30 services that could be considered equivalent
> in scope to an OpenStack project... if anything our scope is increasing
> more _slowly_ than the industry at large. I'm slightly shocked that
> nobody in this thread appears to have even entertained the idea that
> *this is what success looks like*.

That's a fair point of view. However I'm sure the AWS team is staffed a
bit differently where the feature implementers don't outnumber the
QA/Docs/Infra folks by a factor for 10 to 1.

I'm also sure there are a lot more dedicated folks to cross project
effort than we see today.

If we weren't so feature biased in our effort, 30 services might be a
sensible target. But the lack of folks working on some of the holistic /
cross project efforts means I don't think it's really viable.

> The world is not going to stop because we want to get off, take a
> breather, do a "consolidation cycle".

Most deployers I've talked to would actually love for OpenStack to slow
down a bit. They are at least a year behind on their deployments, and a
stabalization cycle is something they'd lvove to see.

> I think we can all agree that any cross-project function program that
> aims to _be_ that function for all of OpenStack, as opposed to enabling
> that function in the individual projects, will not scale. It won't scale
> even if we never accept another new project. It won't scale even if we
> de-integrate some projects pour encourager les autres. I think that the
> work that Doug has been doing with Oslo points the way here. Every
> cross-project function (including release management - get rid of PTLs)
> should have a designated co-ordinator/liason in each project.

This becomes an N^2 communication problem and eventually consistency
problem.

> The move of functional tests out of Tempest and into the individual
> projects' repositories is a great step in the right direction. Hopefully
> that will also address the growth of CI testing capacity requirements -
> while projects like Nova, Cinder and Neutron all depend on and benefit
> from being gated against each other, that is not at all true for many
> projects, and in particular the kinds of new projects that are now
> appearing. Let's take actual dependencies into account when we choose
> what to gate on - the resulting growth might not be sublinear, but it
> needn't be n^2 either.
> 
> Simply put, I don't think we're anywhere near so close to running out of
> ideas that we have to stop and freeze the world. We have scaling
> challenges, and we'll have to make changes, and we'll have to do it -
> sometimes under pressure - even as the world changes around us because
> that's just how life works.
> 
> The question that is not being asked enough IMO is how exactly does it
> benefit our *users* (it is still all about the users, right?) to be
> telling developers "you arrived too late, so you're not real OpenStack
> developers, with free conference tickets and ATC badges and stuff".
> Because if it's just that the support functions within OpenStack can't
> handle them yet, that's on us not them. If it's that we don't have a
> clear way of communicating to users the information they need to make
> the right decision (for them) on whether to deploy some code, then we
> need to communicate that information more clearly. And if we believe
> that somehow companies will magically divert all of the resources they
> would otherwise have committed to new projects into bug fixing on
> existing projects, I think we're being incredibly naive. If companies
> are not seeing the value in strategically contributing to the core of
> projects then it's because we have failed to demonstrate the value, and
> simply blocking off other avenues for them to create value (or blocking
> recognition of those avenues) will absolutely not solve it.

I also think it's completely fair to think about this as small tent,
large ecosystem. The Linux Kernel stops at a certain function point for
a reason. That didn't prevent a ton of exceptionally useful function to
exist above it. But it also mean that things like c library / init
system / packaging / compilers were things where innovation could play
out. And where things could evolve over time.

I don't think small test / big ecosystem prevents anyone from working on
what they are passionate about. It does bring the value of n in n^2
relationships to something tractable.

Maybe there are better people to handle the cross project interactions
that can handle the value of n we currently have, or the value of n we
might have in the future. But I'm watching the strain on the existing
teams doing this and I think under current course and speed we mostly
will lose key leadership (under burn out), and not grow more of it. And
that's a really important thing to consider in keeping something like
OpenStack a functioning base. If the replacement rate of key leadership
does not exceed the fall off rate we won't end up in a happy place in a
few more cycles.

	-Sean

-- 
Sean Dague
http://dague.net