Open Stack

On Thu, Aug 21, 2014 at 2:37 AM, Zane Bitter <zbitter at redhat.com> wrote:

> On 11/08/14 05:24, Thierry Carrez wrote:
>
>> So the idea that being (and remaining) in the integrated release should
>> also be judged on technical merit is a slightly different effort. It's
>> always been a factor in our choices, but like Devananda says, it's more
>> difficult than just checking a number of QA/integration checkboxes. In
>> some cases, blessing one project in a problem space stifles competition,
>> innovation and alternate approaches. In some other cases, we reinvent
>> domain-specific solutions rather than standing on the shoulders of
>> domain-specific giants in neighboring open source projects.
>>
>
> I totally agree that these are the things we need to be vigilant about.
>
> Stifling competition is a big worry, but it appears to me that a lot of
> the stifling is happening even before incubation. Everyone's time is
> limited, so if you happen to notice a new project on the incubation
> trajectory doing things in what you think is the Wrong Way, you're most
> likely to either leave some drive-by feedback or to just ignore it and
> carry on with your life. What you're most likely *not* to do is to start a
> competing project to prove them wrong, or to jump in full time to the
> existing project and show them the light. It's really hard to argue against
> the domain experts too - when you're acutely aware of how shallow your
> knowledge is in a particular area it's very hard to know how hard to push.
> (Perhaps ironically, since becoming a PTL I feel I have to be much more
> cautious in what I say too, because people are inclined to read too much
> into my opinion - I wonder if TC members feel the same pressure.) I speak
> from first-hand instances of guilt here - for example, I gave some feedback
> to the Mistral folks just before the last design summit[1], but I haven't
> had time to follow it up at all. I wouldn't be a bit surprised if they
> showed up with an incubation request, a largely-unchanged user interface
> and an expectation that I would support it.
>
> The result is that projects often don't hear the feedback they need until
> far too late - often when they get to the incubation review (maybe not even
> their first incubation review). In the particularly unfortunate case of
> Marconi, it wasn't until the graduation review. (More about that in a
> second.) My best advice to new projects here is that you must be like a
> ferret up the pant-leg of any negative feedback. Grab hold of any criticism
> and don't let go until you have either converted the person giving it into
> your biggest supporter, been converted by them, or provoked them to start a
> competing project. (Any of those is a win as far as the community is
> concerned.)
>
> Perhaps we could consider a space like a separate mailing list
> (openstack-future?) reserved just for announcements of Related projects,
> their architectural principles, and discussions of the same?  They
> certainly tend to get drowned out amidst the noise of openstack-dev.
> (Project management, meeting announcements, and internal project discussion
> would all be out of scope for this list.)
>
> As for reinventing domain-specific solutions, I'm not sure that happens as
> often as is being made out. IMO the defining feature of IaaS that makes the
> cloud the cloud is on-demand (i.e. real-time) self-service. Everything else
> more or less falls out of that requirement, but the very first thing to
> fall out is multi-tenancy and there just aren't that many multi-tenant
> services floating around out there. There are a couple of obvious
> strategies to deal with that: one is to run existing software within a
> tenant-local resource provisioned by OpenStack (Trove and Sahara are
> examples of this), and the other is to wrap a multi-tenancy framework
> around an existing piece of software (Nova and Cinder are examples of
> this). (BTW the former is usually inherently less satisfying, because it
> scales at a much coarser granularity.) The answer to a question of the form:
>
> "Why do we need OpenStack project $X, when open source project $Y already
> exists?"
>
> is almost always:
>
> "Because $Y is not multi-tenant aware; we need to wrap it with a
> multi-tenancy layer with OpenStack-native authentication, metering and
> quota management. That even allows us to set up an abstraction layer so
> that you can substitute $Z as the back end too."
>
> This is completely uncontroversial when you substitute X, Y, Z = Nova,
> libvirt, Xen. However, when you instead substitute X, Y, Z = Zaqar/Marconi,
> Qpid, MongoDB it suddenly becomes *highly* controversial. I'm all in favour
> of a healthy scepticism, but I think we've passed that point now. (How
> would *you* make an AMQP bus multi-tenant?)
>
> To be clear, Marconi did made a mistake. The Marconi API presented
> semantics to the user that excluded many otherwise-obvious choices of
> back-end plugin (i.e. Qpid/RabbitMQ). It seems to be a common thing (see
> also: Mistral) to want to design for every feature an existing Enterprisey
> application might use, which IMHO kind of ignores the fact that
> applications need to be rewritten to use these new APIs anyway, and we
> would be better off presenting _simple_ interfaces that attract developers,
> lead to good design for new applications and provide flexibility on the
> back-end side. I digress though, because that wasn't the mistake. The
> mistake was failing to educate the entire community (but in particular the
> TC) on the relative merits of this feature enough get either buy-in or
> rejection on this critical detail much earlier in the process.
>
> By the way, I claimed without justification earlier that "there just
> aren't that many multi-tenant services floating around out there", but
> there is actually a very good reason for that. Before there were open
> source IaaS platforms out there, there wasn't any reason to build such a
> service. And now that there are, _the obvious place to build such a service
> in in OpenStack_. We have the users, we have the authentication API that
> already ties in to their existing directories (Keystone is well-named) and
> we have the community.
>
>  This all has created a world where you need to be*in*  OpenStack to
>> matter, or to justify the investment. This has created a world where
>> everything and everyone wants to be in the "OpenStack" integrated
>> release. This has created more pressure to add new projects, and less
>> pressure to fix and make the existing projects perfect. 4 years in, we
>> might want to inflect that trajectory and take steps to fix this world.
>>
>
> We should certainly consider this possibility, that we've set up perverse
> incentives leading to failure. But what if it's just because we haven't yet
> come even close to satisfying all of our users' needs? I mean, AWS has more
> than 30 services that could be considered equivalent in scope to an
> OpenStack project... if anything our scope is increasing more _slowly_ than
> the industry at large. I'm slightly shocked that nobody in this thread
> appears to have even entertained the idea that *this is what success looks
> like*.
>
> The world is not going to stop because we want to get off, take a
> breather, do a "consolidation cycle".
>

+1

Well said, we need figure out (and quickly) how to scale our systems and
*encourage* innovation. If we can't people will move on to other things.

-Angus


> I think we can all agree that any cross-project function program that aims
> to _be_ that function for all of OpenStack, as opposed to enabling that
> function in the individual projects, will not scale. It won't scale even if
> we never accept another new project. It won't scale even if we de-integrate
> some projects pour encourager les autres. I think that the work that Doug
> has been doing with Oslo points the way here. Every cross-project function
> (including release management - get rid of PTLs) should have a designated
> co-ordinator/liason in each project.
>
> The move of functional tests out of Tempest and into the individual
> projects' repositories is a great step in the right direction. Hopefully
> that will also address the growth of CI testing capacity requirements -
> while projects like Nova, Cinder and Neutron all depend on and benefit from
> being gated against each other, that is not at all true for many projects,
> and in particular the kinds of new projects that are now appearing. Let's
> take actual dependencies into account when we choose what to gate on - the
> resulting growth might not be sublinear, but it needn't be n^2 either.
>
> Simply put, I don't think we're anywhere near so close to running out of
> ideas that we have to stop and freeze the world. We have scaling
> challenges, and we'll have to make changes, and we'll have to do it -
> sometimes under pressure - even as the world changes around us because
> that's just how life works.
>
> The question that is not being asked enough IMO is how exactly does it
> benefit our *users* (it is still all about the users, right?) to be telling
> developers "you arrived too late, so you're not real OpenStack developers,
> with free conference tickets and ATC badges and stuff". Because if it's
> just that the support functions within OpenStack can't handle them yet,
> that's on us not them. If it's that we don't have a clear way of
> communicating to users the information they need to make the right decision
> (for them) on whether to deploy some code, then we need to communicate that
> information more clearly. And if we believe that somehow companies will
> magically divert all of the resources they would otherwise have committed
> to new projects into bug fixing on existing projects, I think we're being
> incredibly naive. If companies are not seeing the value in strategically
> contributing to the core of projects then it's because we have failed to
> demonstrate the value, and simply blocking off other avenues for them to
> create value (or blocking recognition of those avenues) will absolutely not
> solve it.
>
> cheers,
> Zane.
>
>
> [1] http://lists.openstack.org/pipermail/openstack-dev/2014-
> May/034641.html
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140821/86d363fa/attachment.html>