Open Stack

On 8/18/2014 9:27 AM, Thierry Carrez wrote:
> Clint Byrum wrote:
>> Here's why folk are questioning Ceilometer:
>>
>> Nova is a set of tools to abstract virtualization implementations.
>> Neutron is a set of tools to abstract SDN/NFV implementations.
>> Cinder is a set of tools to abstract block-device implementations.
>> Trove is a set of tools to simplify consumption of existing databases.
>> Sahara is a set of tools to simplify Hadoop consumption.
>> Swift is a feature-complete implementation of object storage, none of
>> which existed when it was started.
>> Keystone supports all of the above, unifying their auth.
>> Horizon supports all of the above, unifying their GUI.
>>
>> Ceilometer is a complete implementation of data collection and alerting.
>> There is no shortage of implementations that exist already.
>>
>> I'm also core on two projects that are getting some push back these
>> days:
>>
>> Heat is a complete implementation of orchestration. There are at least a
>> few of these already in existence, though not as many as their are data
>> collection and alerting systems.
>>
>> TripleO is an attempt to deploy OpenStack using tools that OpenStack
>> provides. There are already quite a few other tools that _can_ deploy
>> OpenStack, so it stands to reason that people will question why we
>> don't just use those. It is my hope we'll push more into the "unifying
>> the implementations" space and withdraw a bit from the "implementing
>> stuff" space.
>>
>> So, you see, people are happy to unify around a single abstraction, but
>> not so much around a brand new implementation of things that already
>> exist.
> Right, most projects focus on providing abstraction above
> implementations, and that abstraction is where the real "domain
> expertise" of OpenStack should be (because no one else is going to do it
> for us). Every time we reinvent something, we are at larger risk because
> we are out of our common specialty, and we just may not be as good as
> the domain specialists. That doesn't mean we should never reinvent
> something, but we need to be damn sure it's a good idea before we do.
> It's sometimes less fun to piggyback on existing implementations, but if
> they exist that's probably what we should do.
>
> While Ceilometer is far from alone in that space, what sets it apart is
> that even after it was blessed by the TC as "the one we should all
> converge on", we keep on seeing competing implementations for some (if
> not all) of its scope. Convergence did not happen, and without
> convergence we struggle in adoption. We need to understand why, and if
> this is fixable.
>

So, here's what happened with StackTach ...

We had two teams working on StackTach, one group working on the original
program (v2) and another working on Ceilometer integration of our new
design. The problem was, there was no way we could compete with the
speed of the v2 team. Every little thing we needed to do in OpenStack
was a herculean effort. Submit a branch in one place, it needs to go
somewhere else. Spend weeks trying to land a branch. Endlessly debate
about minutia. It goes on.

I know that's the nature of running a large project. And I know everyone
is feeling it.

We quickly came to realize that, if the stars aligned and we did what we
needed to do, we'd only be playing catch-up to the other StackTach team.
And StackTach had growing pains. We needed this new architecture to
solve real business problems *today*. This isn't "built it and they will
come", this is "we know it's valuable ... when can I have the new one?"
Like everyone, we have incredible pressure to deliver and we can't
accurately forecast with so many uncontrollable factors.

Much of what is now StackTach.v3 is (R)esearch not (D)evelopment. With
R, we need to be able to run a little fast-and-loose. Not every pull
request is a masterpiece. Our plans are going to change. We need to have
room to experiment. If it was all just D, yes, we could be more formal.
But we frequently go down a road to find a dead end and need to adjust.

We started on StackTach.v3 outside of formal OpenStack. It's still open
source. We still talk with interested parties (including ceilo) about
the design and how we're going to fulfill their needs, but we're mostly
head-down trying to get a production ready release in place. In the
process, we're making all of StackTach.v3 as tiny repos that other
groups (like Ceilo and Monasca) can adopt if they find them useful. Even
our impending move to StackForge is going to be a big productivity hit,
but it's necessary for some of our potential contributors.

Will we later revisit integration with Ceilometer? Possibly, but it's
not a priority. We have to serve the customers that are screaming for
v3. Arguably this is more of a BDFL model, but in order to innovate
quickly, get to large-scale production and remain competitive it may be
necessary.

This is why I'm pushing for an API-first model in OpenStack. Alternative
implementations shouldn't have to live outside the tribe.

(as always, my view only)