Open Stack

hI can't speak to any of the concerns below from an engineering
standpoint, so I won't try. I do have the benefit of an earful from
various tripleo users, and also of watching a lot of the history of
tripleo development, so I will comment from that standpoint.

On Tue, Jan 26, 2016 at 12:45 AM, Ben Nemec <openstack at nemebean.com> wrote:
> On 01/25/2016 03:56 PM, Steven Hardy wrote:
>> On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec wrote:
>>> So I haven't weighed in on this yet, in part because I was on vacation
>>> when it was first proposed and missed a lot of the initial discussion,
>>> and also because I wanted to take some time to order my thoughts on it.
>>>  Also because my initial reaction...was not conducive to calm and
>>> rational discussion. ;-)
>>>
>>> The tldr is that I don't like it.  To explain why, I'm going to make a
>>> list (everyone loves lists, right? Top $NUMBER reasons we should stop
>>> expecting other people to write our API for us):
>>>
>>> 1) We've been down this road before.  Except last time it was with Heat.
>>>  I'm being somewhat tongue-in-cheek here, but expecting a general
>>> service to provide us a user-friendly API for our specific use case just
>>> doesn't make sense to me.
>>
>> Actually, we've been down this road before with Tuskar, and discovered that
>> designing and maintaining a bespoke API for TripleO is really hard.
>
> My takeaway from Tuskar was that designing an API that none of the
> developers on the project use is doomed to fail.  Tuskar also suffered
> from a lack of some features in Heat that the new API is explicitly
> depending on in an attempt to avoid many of the problems Tuskar had.
>
> Problem #1 is still developer apathy IMHO though.

Tuskar API v2's primary function was to eliminate the need for
merge.py by providing a sane method to combine bits of heat templates
into a coherent whole.

Later, Heat (sensibly, I think) gained the ability to combine its own
templates and manage parameter inheritance. Once that happened Tuskar
was really just getting in the way, even more so when we started
needing relatively large environment files. It was a good engineering
effort that was rendered obsolete by developments around it. We should
not regret it; nor should we mourn it.

>>
>> I somewhat agree that heat as an API is insufficient, but that doesn't
>> necessarily imply you have to have a TripleO specific abstraction, just
>> that *an* abstraction is required.
>>
>>> 2) The TripleO API is not a workflow API.  I also largely missed this
>>> discussion, but the TripleO API is a _Deployment_ API.  In some cases
>>> there also happens to be a workflow going on behind the scenes, but
>>> honestly that's not something I want our users to have to care about.
>>
>> How would you differentiate between "deployment" in a generic sense in
>> contrast to a generic workflow?
>>
>> Every deployment I can think of involves a series of steps, involving some
>> choices and interactions with other services.  That *is* a workflow?
>
> Well, I mean if we want to take this to extremes then pretty much every
> API is a workflow API.  You make a REST call, a "workflow" happens in
> the service, and you get back a result.
>
> Let me turn this around: Would you implement Heat's API on Mistral?  All
> that happens when I call Heat is that a series of OpenStack calls are
> made from heat-engine, after all.  Or is that a gross oversimplification
> of what's happening?  I could argue that the same is true of this
> discussion. :-)

It is fairly clear to me from sitting next to the guy who has had to
implement a big chunk of the update and upgrade code, that Heat is not
good at workflow. This is well understood by everyone and should not
surprise us. It does not make any sense to talk about Heat "making a
series of calls" because "a series" is precisely what Heat does not do
-- it does "a graph" of things, if anything. There is nothing wrong
with this. If you want to do a series of things, however, you need a
different tool.

>>
>>> 3) It ties us 100% to a given implementation.  If Mistral proves to be a
>>> poor choice for some reason, or insufficient for a particular use case,
>>> we have no alternative.  If we have an API and decide to change our
>>> implementation, nobody has to know or care.  This is kind of the whole
>>> point of having an API - it shields users from all the nasty
>>> implementation details under the surface.
>>
>> This is a valid argument, but building (and maintining, forever) a bespoke
>> API is a high cost to pay for this additional degree of abstraction, and
>> when you think of the target audience, I'm not certain it's entirely
>> justified (or, honestly, if our community can bear that overhead);
>>
>> For example, take other single-purpose "deployment" projects, such as
>> Sahara, Magnum, perhaps Trove.  These are designed primarily as user-facing
>> API's, where the services will ultimately be consumed by public and private
>> cloud customers.
>>
>> Contrast with TripleO, where our target audience is, for the most part,
>> sysadmins who deploy and maintain an openstack deployment over a long
>> period of time.  There are two main groups here:
>>
>> 1. PoC "getting started" folks who need a very simple on-ramp (generalizing
>> somewhat, the audience for the opinionated deployments driven via UI's)
>>
>> 2. Seasoned sysadmins who want plugability, control and flexibility above
>> all else (and, simplicity and lack of extraneous abstractions)
>>
>> A bespoke API potentially has a fairly high value to (1), but a very low or
>> even negative value to (2).  Which is why this is turning out to be a tough
>> and polarized discussion, unfortunately.
>
> Well, to be honest I'm not sure we can satisfy the second type of user
> with what we have today anyway.  Our Heat-driven puppet is hardly
> lightweight or simple, and there are extraneous abstractions all over
> the place (see also every place that we have a Heat template param that
> exists solely to get plugged into a puppet hiera file :-).
>
> To me, this is mostly an artifact of the original intent of the Heat
> templates being _the_ abstraction that would then be translated into
> os-*-config, puppet, or [insert deployment tool of choice] by the
> templates, and I have to admit I'm not sure how to fix it for these users.
>
> So I guess the question is, how does having an API hurt those power
> users?  They'll still be able/have to edit Heat templates to deploy
> additional services.  They'll still have all the usual openstack clients
> to customize their Ironic or Nova setups.  They're already using an API
> today, it's just one written entirely in the client.
>
> On the other hand, an API that can help guide a user through the deploy
> process (You say you want network isolation enabled?  Well here are the
> parameters you need to configure...) could make a huge difference for
> the first type of user, as would _any_ API usable by the GUI (people
> just like pretty GUIs, whether it's actually better than the CLI or not :-).
>
> I guess it is somewhat subjective as you say, but to me the API doesn't
> significantly affect the power user experience, but it would massively
> improve the newbie experience.  That makes it a win in my book.

Let's talk for a minute about the community we hope to build around
the work we're doing here.

TripleO has gained a lot by adopting and using the OpenStack Puppet
Modules project, a busy project with a lot of contributors that stays
very well up to date with OpenStack itself. Unfortunately, the layer
on top of the puppet modules -- t-h-t, as currently constituted -- is
something carried entirely by this small project. If TripleO is going
to succeed, we need to start attracting operators who want to actively
contribute to the composition layer (a point well made by Clint
already in this thread). Whether that means we embrace Mistral, or
Ansible, or Mistral with an Ansible front end, or something else
entirely doesn't matter, as long as it is likely to attract a
significant number of contributors who actually deploy OpenStack in
the field and want a place where their accumulated knowledge of how
and how *not* to do that can be shared and encoded.

So, like I said, I can't and won't speak to the correct technical
choice here. I'm just very hopeful that we can take this opportunity
to think about what the deployer/operator community will want to get
involved with, and leverage their knowledge and their energy wherever
possible. If we can do this we will improve TripleO, and we will
improve OpenStack into the bargain.

--Hugh


>>
>>> 4) It raises the bar even further for both new deployers and developers.
>>>  You already need to have a pretty firm grasp of Puppet and Heat
>>> templates to understand how our stuff works, not to mention a decent
>>> understanding of quite a number of OpenStack services.
>>
>> I'm not really sure if a bespoke WSGI app vs an existing one (mistral)
>> really makes much difference at all wrt raising the bar.  I see it
>> primarily as in implementation detail tbh.
>
> I guess that depends.  Most people in OpenStack already know at least
> some Python, and if you've done any work in the other projects there's a
> fair chance you are familiar with the Python clients.  How many people
> know Mistral YAML?
>
> Maybe I'm overestimating the Python knowledge in the community, and
> underestimating the Mistral knowledge, but I would bet we're talking
> order(s?) of magnitude in terms of the difference.  And I'm not saying
> learning Mistral is a huge task on its own, but it's one more thing in a
> project full of one more things.
>
>>
>> Looking at the prototypes Dan has done, it *is* possible to define a much
>> more strongly versioned API with Mistral than it is with Heat, which is the
>> main requirement in terms of lowering the bar (stable, relatively
>> opinionated by default and easy to use).
>>
>> Steve
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
* Hugh Brock, hbrock at redhat.com
* Senior Engineering Manager, Cloud Engineering
* RDO Manager: Install, configure, and scale OpenStack.
* http://rdoproject.org

"I know that you believe you understand what you think I said, but I'm
not sure you realize that what you heard is not what I meant."
--Robert McCloskey