Open Stack

On Aug 27, 2014, at 8:47 AM, Sean Dague <sean at dague.net> wrote:

> On 08/26/2014 11:40 AM, Anne Gentle wrote:
>> 
>> 
>> 
>> On Mon, Aug 25, 2014 at 8:36 AM, Sean Dague <sean at dague.net
>> <mailto:sean at dague.net>> wrote:
>> 
>>    On 08/20/2014 12:37 PM, Zane Bitter wrote:
>>> On 11/08/14 05:24, Thierry Carrez wrote:
>>>> So the idea that being (and remaining) in the integrated release
>>    should
>>>> also be judged on technical merit is a slightly different effort.
>>    It's
>>>> always been a factor in our choices, but like Devananda says,
>>    it's more
>>>> difficult than just checking a number of QA/integration
>>    checkboxes. In
>>>> some cases, blessing one project in a problem space stifles
>>    competition,
>>>> innovation and alternate approaches. In some other cases, we reinvent
>>>> domain-specific solutions rather than standing on the shoulders of
>>>> domain-specific giants in neighboring open source projects.
>>> 
>>> I totally agree that these are the things we need to be vigilant
>>    about.
>>> 
>>> Stifling competition is a big worry, but it appears to me that a
>>    lot of
>>> the stifling is happening even before incubation. Everyone's time is
>>> limited, so if you happen to notice a new project on the incubation
>>> trajectory doing things in what you think is the Wrong Way, you're
>>    most
>>> likely to either leave some drive-by feedback or to just ignore it and
>>> carry on with your life. What you're most likely *not* to do is to
>>    start
>>> a competing project to prove them wrong, or to jump in full time
>>    to the
>>> existing project and show them the light. It's really hard to argue
>>> against the domain experts too - when you're acutely aware of how
>>> shallow your knowledge is in a particular area it's very hard to know
>>> how hard to push. (Perhaps ironically, since becoming a PTL I feel I
>>> have to be much more cautious in what I say too, because people are
>>> inclined to read too much into my opinion - I wonder if TC members
>>    feel
>>> the same pressure.) I speak from first-hand instances of guilt here -
>>> for example, I gave some feedback to the Mistral folks just before the
>>> last design summit[1], but I haven't had time to follow it up at
>>    all. I
>>> wouldn't be a bit surprised if they showed up with an incubation
>>> request, a largely-unchanged user interface and an expectation that I
>>> would support it.
>>> 
>>> The result is that projects often don't hear the feedback they need
>>> until far too late - often when they get to the incubation review
>>    (maybe
>>> not even their first incubation review). In the particularly
>>    unfortunate
>>> case of Marconi, it wasn't until the graduation review. (More
>>    about that
>>> in a second.) My best advice to new projects here is that you must be
>>> like a ferret up the pant-leg of any negative feedback. Grab hold
>>    of any
>>> criticism and don't let go until you have either converted the person
>>> giving it into your biggest supporter, been converted by them, or
>>> provoked them to start a competing project. (Any of those is a win as
>>> far as the community is concerned.)
>>> 
>>> Perhaps we could consider a space like a separate mailing list
>>> (openstack-future?) reserved just for announcements of Related
>>    projects,
>>> their architectural principles, and discussions of the same?  They
>>> certainly tend to get drowned out amidst the noise of openstack-dev.
>>> (Project management, meeting announcements, and internal project
>>> discussion would all be out of scope for this list.)
>>> 
>>> As for reinventing domain-specific solutions, I'm not sure that
>>    happens
>>> as often as is being made out. IMO the defining feature of IaaS that
>>> makes the cloud the cloud is on-demand (i.e. real-time) self-service.
>>> Everything else more or less falls out of that requirement, but
>>    the very
>>> first thing to fall out is multi-tenancy and there just aren't
>>    that many
>>> multi-tenant services floating around out there. There are a couple of
>>> obvious strategies to deal with that: one is to run existing software
>>> within a tenant-local resource provisioned by OpenStack (Trove and
>>> Sahara are examples of this), and the other is to wrap a multi-tenancy
>>> framework around an existing piece of software (Nova and Cinder are
>>> examples of this). (BTW the former is usually inherently less
>>> satisfying, because it scales at a much coarser granularity.) The
>>    answer
>>> to a question of the form:
>>> 
>>> "Why do we need OpenStack project $X, when open source project $Y
>>> already exists?"
>>> 
>>> is almost always:
>>> 
>>> "Because $Y is not multi-tenant aware; we need to wrap it with a
>>> multi-tenancy layer with OpenStack-native authentication, metering and
>>> quota management. That even allows us to set up an abstraction
>>    layer so
>>> that you can substitute $Z as the back end too."
>>> 
>>> This is completely uncontroversial when you substitute X, Y, Z = Nova,
>>> libvirt, Xen. However, when you instead substitute X, Y, Z =
>>> Zaqar/Marconi, Qpid, MongoDB it suddenly becomes *highly*
>>    controversial.
>>> I'm all in favour of a healthy scepticism, but I think we've
>>    passed that
>>> point now. (How would *you* make an AMQP bus multi-tenant?)
>>> 
>>> To be clear, Marconi did made a mistake. The Marconi API presented
>>> semantics to the user that excluded many otherwise-obvious choices of
>>> back-end plugin (i.e. Qpid/RabbitMQ). It seems to be a common
>>    thing (see
>>> also: Mistral) to want to design for every feature an existing
>>> Enterprisey application might use, which IMHO kind of ignores the fact
>>> that applications need to be rewritten to use these new APIs
>>    anyway, and
>>> we would be better off presenting _simple_ interfaces that attract
>>> developers, lead to good design for new applications and provide
>>> flexibility on the back-end side. I digress though, because that
>>    wasn't
>>> the mistake. The mistake was failing to educate the entire community
>>> (but in particular the TC) on the relative merits of this feature
>>    enough
>>> get either buy-in or rejection on this critical detail much earlier in
>>> the process.
>>> 
>>> By the way, I claimed without justification earlier that "there just
>>> aren't that many multi-tenant services floating around out there", but
>>> there is actually a very good reason for that. Before there were open
>>> source IaaS platforms out there, there wasn't any reason to build
>>    such a
>>> service. And now that there are, _the obvious place to build such a
>>> service in in OpenStack_. We have the users, we have the
>>    authentication
>>> API that already ties in to their existing directories (Keystone is
>>> well-named) and we have the community.
>> 
>>    I think this is a very crisp view of what OpenStack is trying to do. I
>>    like it. :)
>> 
>>>> This all has created a world where you need to be*in*  OpenStack to
>>>> matter, or to justify the investment. This has created a world where
>>>> everything and everyone wants to be in the "OpenStack" integrated
>>>> release. This has created more pressure to add new projects, and less
>>>> pressure to fix and make the existing projects perfect. 4 years
>>    in, we
>>>> might want to inflect that trajectory and take steps to fix this
>>    world.
>>> 
>>> We should certainly consider this possibility, that we've set up
>>> perverse incentives leading to failure. But what if it's just
>>    because we
>>> haven't yet come even close to satisfying all of our users' needs? I
>>> mean, AWS has more than 30 services that could be considered
>>    equivalent
>>> in scope to an OpenStack project... if anything our scope is
>>    increasing
>>> more _slowly_ than the industry at large. I'm slightly shocked that
>>> nobody in this thread appears to have even entertained the idea that
>>> *this is what success looks like*.
>> 
>>    That's a fair point of view. However I'm sure the AWS team is staffed a
>>    bit differently where the feature implementers don't outnumber the
>>    QA/Docs/Infra folks by a factor for 10 to 1.
>> 
>> 
>> 
>> With 1200 overall OpenStack unique contributors, we had 195 docs
>> contributors. What's good about that metric is it shows over 16%
>> contributors are involved in docs. However, about 10 individual
>> contributors wrote half the docs patches for Icehouse. So we still
>> struggle as a core team.
>> 
>> Scoping the docs efforts very tightly is the current method for
>> maintaining docs. That said, I spent about a day last week writing up
>> incubating and integrating docs requirements. [1] After that exercise I
>> sense that our doc scaling struggles will continue but that the
>> program's teams will step up. My efforts around simplifying the docs
>> contribution path are stalled a bit, but that's the second method to
>> work on keeping up. 
>> 
>> All that said, my current line of thinking about adding programs is that
>> we need to enable more options in each program, even if I personally
>> have no idea how to keep up with the level of docs quality, accuracy,
>> reviews, and so on. :) 
>> 
>> By having the TC enable innovation and competition within programs we
>> can prove we are not locking anyone in to one integration grouping. With
>> our open source ethos, we need to work harder to enable projects that
>> are optional. Yes, this stance flies in the face of "but how will we
>> support, test, document unless we pick a happy path?" Believe me, I know
>> this conundrum well.
>> 
>> I like to think of projects as LEGO bricks that interlock optionally. I
>> want us to think like we're making LEGO Creator sets -- you get to build
>> three different LEGO animals (eagle, scorpion, beaver [2]) from one kit.
>> For an OpenStack example, we need a way for TripleO, chef, and puppet
>> experts to be willing to work in a common way to further a Deployment
>> program's goals. Same for monitoring and probably other programs. Then
>> the TC isn't picking the "only" way to meet those goals.
>> 
>> Thanks for reading this lonnng thread. At least you get LEGO links at
>> the very end. 
>> Anne
>> 
>> 1. https://wiki.openstack.org/wiki/Documentation/IncubateIntegrate
>> 2. http://shop.lego.com/en-US/Fierce-Flyer-31004
> 
> So I think we all want the future where OpenStack is a really nice set
> of composable services that let you easily create the cloud you want.
> They are all stable, upgradeable, and easy to understand.
> 
> I think the challenge is, personally, I don't see how we get there on
> current course and speed.
> 
> I just got back from effectively 2 weeks out (1 on vacation, 1 at Linux
> con). And basically the moment I was seen active on IRC I got slammed
> with pings of 'this kind of job now seems wedged, help!' 'this thing
> between two projects is blocking and we can't figure out how to land the
> code, help!'
> 
> As someone who spends a ton of time unwinding the completely crazy ways
> OpenStack fails to work with OpenStack given the complexity of
> interaction, it's exhausting. And, honestly, I'm feeling pretty strongly
> that it's time for me to step away from the gate. Because I actually
> think that the fact that I built this mental model and have been able to
> smooth over so many things is actually making the situation worse,
> because people largely don't realize how *incredibly* manual the
> assembly of these parts are, and that it's not sustainable.

You do an amazing job with gate debugging. There are a lot of other people, myself included, who would like to be able to help but do not yet have the expertise to do it the way you do. What can we do to shorten the “training” period? Is there a way to bootstrap us?

For example, Matt helped me with an issue yesterday, and afterwards I asked him to write up a few details about how he reached his conclusion because he was moving fast enough that I wasn’t actually learning anything from what he was saying to me on IRC. Having an example with some logs and then even stream of consciousness notes like “I noticed the out of memory error, and then I found the first instance of that and looked at the oom-killer report in syslog to see which process was killed and it was X which might mean Y” would help.

Doug

> 
> It's from this perspective, and seeing different community models that
> work with a smaller base, larger ecosystem, that I think we need to
> really think about a smaller tent model for what the minimum viable
> units are in OpenStack and how we layer up from there -
> https://dague.net/2014/08/26/openstack-as-layers/
> 
> 	-Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev