Open Stack

On 11/08/14 05:24, Thierry Carrez wrote:
> So the idea that being (and remaining) in the integrated release should
> also be judged on technical merit is a slightly different effort. It's
> always been a factor in our choices, but like Devananda says, it's more
> difficult than just checking a number of QA/integration checkboxes. In
> some cases, blessing one project in a problem space stifles competition,
> innovation and alternate approaches. In some other cases, we reinvent
> domain-specific solutions rather than standing on the shoulders of
> domain-specific giants in neighboring open source projects.

I totally agree that these are the things we need to be vigilant about.

Stifling competition is a big worry, but it appears to me that a lot of 
the stifling is happening even before incubation. Everyone's time is 
limited, so if you happen to notice a new project on the incubation 
trajectory doing things in what you think is the Wrong Way, you're most 
likely to either leave some drive-by feedback or to just ignore it and 
carry on with your life. What you're most likely *not* to do is to start 
a competing project to prove them wrong, or to jump in full time to the 
existing project and show them the light. It's really hard to argue 
against the domain experts too - when you're acutely aware of how 
shallow your knowledge is in a particular area it's very hard to know 
how hard to push. (Perhaps ironically, since becoming a PTL I feel I 
have to be much more cautious in what I say too, because people are 
inclined to read too much into my opinion - I wonder if TC members feel 
the same pressure.) I speak from first-hand instances of guilt here - 
for example, I gave some feedback to the Mistral folks just before the 
last design summit[1], but I haven't had time to follow it up at all. I 
wouldn't be a bit surprised if they showed up with an incubation 
request, a largely-unchanged user interface and an expectation that I 
would support it.

The result is that projects often don't hear the feedback they need 
until far too late - often when they get to the incubation review (maybe 
not even their first incubation review). In the particularly unfortunate 
case of Marconi, it wasn't until the graduation review. (More about that 
in a second.) My best advice to new projects here is that you must be 
like a ferret up the pant-leg of any negative feedback. Grab hold of any 
criticism and don't let go until you have either converted the person 
giving it into your biggest supporter, been converted by them, or 
provoked them to start a competing project. (Any of those is a win as 
far as the community is concerned.)

Perhaps we could consider a space like a separate mailing list 
(openstack-future?) reserved just for announcements of Related projects, 
their architectural principles, and discussions of the same?  They 
certainly tend to get drowned out amidst the noise of openstack-dev. 
(Project management, meeting announcements, and internal project 
discussion would all be out of scope for this list.)

As for reinventing domain-specific solutions, I'm not sure that happens 
as often as is being made out. IMO the defining feature of IaaS that 
makes the cloud the cloud is on-demand (i.e. real-time) self-service. 
Everything else more or less falls out of that requirement, but the very 
first thing to fall out is multi-tenancy and there just aren't that many 
multi-tenant services floating around out there. There are a couple of 
obvious strategies to deal with that: one is to run existing software 
within a tenant-local resource provisioned by OpenStack (Trove and 
Sahara are examples of this), and the other is to wrap a multi-tenancy 
framework around an existing piece of software (Nova and Cinder are 
examples of this). (BTW the former is usually inherently less 
satisfying, because it scales at a much coarser granularity.) The answer 
to a question of the form:

"Why do we need OpenStack project $X, when open source project $Y 
already exists?"

is almost always:

"Because $Y is not multi-tenant aware; we need to wrap it with a 
multi-tenancy layer with OpenStack-native authentication, metering and 
quota management. That even allows us to set up an abstraction layer so 
that you can substitute $Z as the back end too."

This is completely uncontroversial when you substitute X, Y, Z = Nova, 
libvirt, Xen. However, when you instead substitute X, Y, Z = 
Zaqar/Marconi, Qpid, MongoDB it suddenly becomes *highly* controversial. 
I'm all in favour of a healthy scepticism, but I think we've passed that 
point now. (How would *you* make an AMQP bus multi-tenant?)

To be clear, Marconi did made a mistake. The Marconi API presented 
semantics to the user that excluded many otherwise-obvious choices of 
back-end plugin (i.e. Qpid/RabbitMQ). It seems to be a common thing (see 
also: Mistral) to want to design for every feature an existing 
Enterprisey application might use, which IMHO kind of ignores the fact 
that applications need to be rewritten to use these new APIs anyway, and 
we would be better off presenting _simple_ interfaces that attract 
developers, lead to good design for new applications and provide 
flexibility on the back-end side. I digress though, because that wasn't 
the mistake. The mistake was failing to educate the entire community 
(but in particular the TC) on the relative merits of this feature enough 
get either buy-in or rejection on this critical detail much earlier in 
the process.

By the way, I claimed without justification earlier that "there just 
aren't that many multi-tenant services floating around out there", but 
there is actually a very good reason for that. Before there were open 
source IaaS platforms out there, there wasn't any reason to build such a 
service. And now that there are, _the obvious place to build such a 
service in in OpenStack_. We have the users, we have the authentication 
API that already ties in to their existing directories (Keystone is 
well-named) and we have the community.

> This all has created a world where you need to be*in*  OpenStack to
> matter, or to justify the investment. This has created a world where
> everything and everyone wants to be in the "OpenStack" integrated
> release. This has created more pressure to add new projects, and less
> pressure to fix and make the existing projects perfect. 4 years in, we
> might want to inflect that trajectory and take steps to fix this world.

We should certainly consider this possibility, that we've set up 
perverse incentives leading to failure. But what if it's just because we 
haven't yet come even close to satisfying all of our users' needs? I 
mean, AWS has more than 30 services that could be considered equivalent 
in scope to an OpenStack project... if anything our scope is increasing 
more _slowly_ than the industry at large. I'm slightly shocked that 
nobody in this thread appears to have even entertained the idea that 
*this is what success looks like*.

The world is not going to stop because we want to get off, take a 
breather, do a "consolidation cycle".

I think we can all agree that any cross-project function program that 
aims to _be_ that function for all of OpenStack, as opposed to enabling 
that function in the individual projects, will not scale. It won't scale 
even if we never accept another new project. It won't scale even if we 
de-integrate some projects pour encourager les autres. I think that the 
work that Doug has been doing with Oslo points the way here. Every 
cross-project function (including release management - get rid of PTLs) 
should have a designated co-ordinator/liason in each project.

The move of functional tests out of Tempest and into the individual 
projects' repositories is a great step in the right direction. Hopefully 
that will also address the growth of CI testing capacity requirements - 
while projects like Nova, Cinder and Neutron all depend on and benefit 
from being gated against each other, that is not at all true for many 
projects, and in particular the kinds of new projects that are now 
appearing. Let's take actual dependencies into account when we choose 
what to gate on - the resulting growth might not be sublinear, but it 
needn't be n^2 either.

Simply put, I don't think we're anywhere near so close to running out of 
ideas that we have to stop and freeze the world. We have scaling 
challenges, and we'll have to make changes, and we'll have to do it - 
sometimes under pressure - even as the world changes around us because 
that's just how life works.

The question that is not being asked enough IMO is how exactly does it 
benefit our *users* (it is still all about the users, right?) to be 
telling developers "you arrived too late, so you're not real OpenStack 
developers, with free conference tickets and ATC badges and stuff". 
Because if it's just that the support functions within OpenStack can't 
handle them yet, that's on us not them. If it's that we don't have a 
clear way of communicating to users the information they need to make 
the right decision (for them) on whether to deploy some code, then we 
need to communicate that information more clearly. And if we believe 
that somehow companies will magically divert all of the resources they 
would otherwise have committed to new projects into bug fixing on 
existing projects, I think we're being incredibly naive. If companies 
are not seeing the value in strategically contributing to the core of 
projects then it's because we have failed to demonstrate the value, and 
simply blocking off other avenues for them to create value (or blocking 
recognition of those avenues) will absolutely not solve it.

cheers,
Zane.


[1] http://lists.openstack.org/pipermail/openstack-dev/2014-May/034641.html