[openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

Dan Prince dprince at redhat.com
Tue Jan 26 15:27:28 UTC 2016


On Tue, 2016-01-26 at 09:56 -0500, James Slagle wrote:
> 
> 
> On Tue, Jan 26, 2016 at 9:08 AM, Steven Hardy <shardy at redhat.com>
> wrote:
> > On Tue, Jan 26, 2016 at 07:40:05AM -0500, James Slagle wrote:
> > >    On Tue, Jan 26, 2016 at 4:46 AM, Steven Hardy <shardy at redhat.c
> > om> wrote:
> > >
> > >      On Mon, Jan 25, 2016 at 05:45:30PM -0600, Ben Nemec wrote:
> > >      > On 01/25/2016 03:56 PM, Steven Hardy wrote:
> > >      > > On Fri, Jan 22, 2016 at 11:24:20AM -0600, Ben Nemec
> > wrote:
> > >      > >> So I haven't weighed in on this yet, in part because I
> > was on
> > >      vacation
> > >      > >> when it was first proposed and missed a lot of the
> > initial
> > >      discussion,
> > >      > >> and also because I wanted to take some time to order my
> > thoughts on
> > >      it.
> > >      > >>  Also because my initial reaction...was not conducive
> > to calm and
> > >      > >> rational discussion. ;-)
> > >      > >>
> > >      > >> The tldr is that I don't like it.  To explain why, I'm
> > going to
> > >      make a
> > >      > >> list (everyone loves lists, right? Top $NUMBER reasons
> > we should
> > >      stop
> > >      > >> expecting other people to write our API for us):
> > >      > >>
> > >      > >> 1) We've been down this road before.  Except last time
> > it was with
> > >      Heat.
> > >      > >>  I'm being somewhat tongue-in-cheek here, but
> > expecting a general
> > >      > >> service to provide us a user-friendly API for our
> > specific use case
> > >      just
> > >      > >> doesn't make sense to me.
> > >      > >
> > >      > > Actually, we've been down this road before with Tuskar,
> > and
> > >      discovered that
> > >      > > designing and maintaining a bespoke API for TripleO is
> > really hard.
> > >      >
> > >      > My takeaway from Tuskar was that designing an API that
> > none of the
> > >      > developers on the project use is doomed to fail.  Tuskar
> > also
> > >      suffered
> > >      > from a lack of some features in Heat that the new API is
> > explicitly
> > >      > depending on in an attempt to avoid many of the problems
> > Tuskar had.
> > >      >
> > >      > Problem #1 is still developer apathy IMHO though.
> > >
> > >      I think the main issue is developer capacity - we're a small
> > community
> > >      and
> > >      I for one am worried about the effort involved with building
> > and
> > >      maintaining a bespoke API - thus this whole discussion is
> > essentially
> > >      about
> > >      finding a quicker and easier way to meet the needs of those
> > needing an
> > >      API.
> > >
> > >      In terms of apathy, I think as a developer I don't need an
> > abstraction
> > >      between me, my templates and heat.  Some advanced operators
> > will feel
> > >      likewise, others won't.  What I would find useful sometimes
> > is a
> > >      general
> > >      purpose workflow engine, which is where I think the more
> > pluggable
> > >      mistral
> > >      based solution may have advantages in terms of developer and
> > advanced
> > >      operator uptake.
> > >      > > I somewhat agree that heat as an API is insufficient,
> > but that
> > >      doesn't
> > >      > > necessarily imply you have to have a TripleO specific
> > abstraction,
> > >      just
> > >      > > that *an* abstraction is required.
> > >      > >
> > >      > >> 2) The TripleO API is not a workflow API.  I also
> > largely missed
> > >      this
> > >      > >> discussion, but the TripleO API is a _Deployment_
> > API.  In some
> > >      cases
> > >      > >> there also happens to be a workflow going on behind the
> > scenes, but
> > >      > >> honestly that's not something I want our users to have
> > to care
> > >      about.
> > >      > >
> > >      > > How would you differentiate between "deployment" in a
> > generic sense
> > >      in
> > >      > > contrast to a generic workflow?
> > >      > >
> > >      > > Every deployment I can think of involves a series of
> > steps,
> > >      involving some
> > >      > > choices and interactions with other services.  That
> > *is* a
> > >      workflow?
> > >      >
> > >      > Well, I mean if we want to take this to extremes then
> > pretty much
> > >      every
> > >      > API is a workflow API.  You make a REST call, a
> > "workflow" happens in
> > >      > the service, and you get back a result.
> > >      >
> > >      > Let me turn this around: Would you implement Heat's API on
> > Mistral? 
> > >      All
> > >      > that happens when I call Heat is that a series of
> > OpenStack calls are
> > >      > made from heat-engine, after all.  Or is that a gross
> > >      oversimplification
> > >      > of what's happening?  I could argue that the same is true
> > of this
> > >      > discussion. :-)
> > >
> > >      As Hugh has mentioned the main thing Heat does is actually
> > manage
> > >      dependencies.  It processes the templates, builds a graph,
> > then walks
> > >      the
> > >      graph running a "workflow" to create/update/delete/etc each
> > resource.
> > >
> > >      I could imagine a future where we interface to some external
> > workflow
> > >      tool to
> > >      e.g do each resource action (e.g create a nova server, poll
> > until it's
> > >      active),
> > >      however that's actually a pretty high overhead approach, and
> > it'd
> > >      probably
> > >      be better to move towards better use of notifications
> > instead (e.g less
> > >      internal workflow)
> > >      > >> 3) It ties us 100% to a given implementation.  If
> > Mistral proves
> > >      to be a
> > >      > >> poor choice for some reason, or insufficient for a
> > particular use
> > >      case,
> > >      > >> we have no alternative.  If we have an API and decide
> > to change
> > >      our
> > >      > >> implementation, nobody has to know or care.  This is
> > kind of the
> > >      whole
> > >      > >> point of having an API - it shields users from all the
> > nasty
> > >      > >> implementation details under the surface.
> > >      > >
> > >      > > This is a valid argument, but building (and maintining,
> > forever) a
> > >      bespoke
> > >      > > API is a high cost to pay for this additional degree of
> > abstraction,
> > >      and
> > >      > > when you think of the target audience, I'm not certain
> > it's entirely
> > >      > > justified (or, honestly, if our community can bear that
> > overhead);
> > >      > >
> > >      > > For example, take other single-purpose "deployment"
> > projects, such
> > >      as
> > >      > > Sahara, Magnum, perhaps Trove.  These are designed
> > primarily as
> > >      user-facing
> > >      > > API's, where the services will ultimately be consumed by
> > public and
> > >      private
> > >      > > cloud customers.
> > >      > >
> > >      > > Contrast with TripleO, where our target audience is, for
> > the most
> > >      part,
> > >      > > sysadmins who deploy and maintain an openstack
> > deployment over a
> > >      long
> > >      > > period of time.  There are two main groups here:
> > >      > >
> > >      > > 1. PoC "getting started" folks who need a very simple
> > on-ramp
> > >      (generalizing
> > >      > > somewhat, the audience for the opinionated deployments
> > driven via
> > >      UI's)
> > >      > >
> > >      > > 2. Seasoned sysadmins who want plugability, control and
> > flexibility
> > >      above
> > >      > > all else (and, simplicity and lack of extraneous
> > abstractions)
> > >      > >
> > >      > > A bespoke API potentially has a fairly high value to
> > (1), but a very
> > >      low or
> > >      > > even negative value to (2).  Which is why this is
> > turning out to be
> > >      a tough
> > >      > > and polarized discussion, unfortunately.
> > >      >
> > >      > Well, to be honest I'm not sure we can satisfy the second
> > type of user
> > >      > with what we have today anyway.  Our Heat-driven puppet
> > is hardly
> > >      > lightweight or simple, and there are extraneous
> > abstractions all over
> > >      > the place (see also every place that we have a Heat
> > template param
> > >      that
> > >      > exists solely to get plugged into a puppet hiera file :-).
> > >      >
> > >      > To me, this is mostly an artifact of the original intent
> > of the Heat
> > >      > templates being _the_ abstraction that would then be
> > translated into
> > >      > os-*-config, puppet, or [insert deployment tool of choice]
> > by the
> > >      > templates, and I have to admit I'm not sure how to fix it
> > for these
> > >      users.
> > >
> > >      I think we fix it by giving them a choice.  E.g along the
> > lines of the
> > >      "split stack" approach discussed at summit - allow operators
> > to choose
> > >      either pre-defined roles with known interfaces (parameters),
> > or deploy
> > >      just
> > >      the infrastructure (servers, networking, maybe storage) then
> > drive
> > >      configuration tooling with a much thinner interface.
> > >      > So I guess the question is, how does having an API hurt
> > those power
> > >      > users?  They'll still be able/have to edit Heat templates
> > to deploy
> > >      > additional services.  They'll still have all the usual
> > openstack
> > >      clients
> > >      > to customize their Ironic or Nova setups.  They're
> > already using an
> > >      API
> > >      > today, it's just one written entirely in the client.
> > >
> > >      There's already a bunch of opaque complexity inside the
> > client and
> > >      TripleO
> > >      common, adding a very rigid API makes it more opaque, and
> > harder to
> > >      modify.
> > >      > On the other hand, an API that can help guide a user
> > through the
> > >      deploy
> > >      > process (You say you want network isolation enabled? 
> > Well here are
> > >      the
> > >      > parameters you need to configure...) could make a huge
> > difference for
> > >      > the first type of user, as would _any_ API usable by the
> > GUI (people
> > >      > just like pretty GUIs, whether it's actually better than
> > the CLI or
> > >      not :-).
> > >      >
> > >      > I guess it is somewhat subjective as you say, but to me
> > the API
> > >      doesn't
> > >      > significantly affect the power user experience, but it
> > would massively
> > >      > improve the newbie experience.  That makes it a win in my
> > book.
> > >
> > >      I agree 100% that we need to massively improve the newbie
> > experience - I
> > >      think everybody does.  I also think we also all agree there
> > must be a
> > >      stable, versioned API that a UI/CLI can interact with.
> > >
> > >      The question in my mind is, can we address that requirement
> > *and*
> > >      provide
> > >      something of non-negative value for developers and advanced
> > operators.
> > >
> > >      Ryan already commented earlier in this thread (and I agree
> > having seen
> > >      Dan's most recent PoC in action) that it doesn't make a lot
> > of
> > >      difference
> > >      from a consumer-of-api perspective which choice we make in
> > terms of APi
> > >      impelementation, either approach can help provide the stable
> > API surface
> > >      that is needed.
> > >
> > >      The main difference is, only one choice provides any
> > flexibility at all
> > >      wrt
> > >      operator customization (unless we reinvent a similar action
> > plugin
> > >      mechanism
> > >      inside a TripleO API).
> > >      > >> 4) It raises the bar even further for both new
> > deployers and
> > >      developers.
> > >      > >>  You already need to have a pretty firm grasp of
> > Puppet and Heat
> > >      > >> templates to understand how our stuff works, not to
> > mention a
> > >      decent
> > >      > >> understanding of quite a number of OpenStack services.
> > >      > >
> > >      > > I'm not really sure if a bespoke WSGI app vs an existing
> > one
> > >      (mistral)
> > >      > > really makes much difference at all wrt raising the
> > bar.  I see it
> > >      > > primarily as in implementation detail tbh.
> > >      >
> > >      > I guess that depends.  Most people in OpenStack already
> > know at least
> > >      > some Python, and if you've done any work in the other
> > projects there's
> > >      a
> > >      > fair chance you are familiar with the Python clients. 
> > How many
> > >      people
> > >      > know Mistral YAML?
> > >
> > >      So, I think you're conflating the OpenStack developer
> > community (who,
> > >      mostly, know python), with end-users and Operators, where
> > IME the same
> > >      is
> > >      often not true.
> > >
> > >      Think of more traditional enterprise environments - how many
> > sysadmins
> > >      on
> > >      the unix team are hardcore python hackers?  Not that many
> > IME (ignoring
> > >      more devops style environments here).
> > >      > Maybe I'm overestimating the Python knowledge in the
> > community, and
> > >      > underestimating the Mistral knowledge, but I would bet
> > we're talking
> > >      > order(s?) of magnitude in terms of the difference.  And
> > I'm not
> > >      saying
> > >      > learning Mistral is a huge task on its own, but it's one
> > more thing in
> > >      a
> > >      > project full of one more things.
> > >
> > >      It's one more thing, which is already maintained and has an
> > active
> > >      community, vs yet-another-bespoke-special-to-tripleo-
> > thing.  IMHO we
> > >      have
> > >      *way* too many tripleo specific things already.
> > >
> > >      However, lets look at the "python knowledge" thing in a bit
> > more detail.
> > >
> > >      Let's say, as an operator I want to wire in a HTTP call to
> > an internal
> > >      asset
> > >      management system.  The requirement is to log an HTTP call
> > with some
> > >      content every time an overcloud is deployed or updated. 
> > (This sort of
> > >      requirement is *very* common in enterprise environments IME)
> > >
> > >      In the mistral case[1], the modification would look
> > something like:
> > >
> > >      http_task:
> > >        action: std.http url='assets.foo.com' <some arguments>
> > >
> > >      You'd simply add two lines to your TripleO deployment
> > workflow yaml[2]:
> > >
> > >    â**This is where the argument for Mistral really breaks down
> > for me. One
> > >    of the advantages of Mistral shouldn't be that it makes it
> > easier for
> > >    operators to modify TripleO delivered workflows. If that
> > becomes
> > >    necessary, we haven't implemented the solution in a flexible
> > enough way.
> > >    Maybe you're just illustrating an example here of someone who
> > is
> > >    completely set on forking TripleO. But in that case, then the
> > example
> > >    isn't really all that relevant since we shouldn't be making a
> > technical
> > >    choice based on that use case.â**
> > >    Your points below about which would be easier, modifying
> > python code or
> > >    yaml files, really apply either way. The argument seems to be
> > "let's use
> > >    Mistral because it's backed by yaml which is easier for
> > operators to
> > >    modify".
> > 
> > How is wanting to make a single HTTP request to some non-openstack
> > system
> > the same as "completely set on forking TripleO"?  Sorry, but I
> > can't
> > reconcile my simple (and IME realistic, from working with actual
> > customers)
> > example with your response at all.
> 
> I'm not disagreeing that one way would be easier to change than the
> other. I'm totally with you that your yaml example is simpler, and if
> that is an easier on ramp for operators to submit patches back to
> TripleO, then that is definitely an advantage of one choice over the
> other.
> 
> More importantly though, I'm talking about how the system is intended
> to be used: What, we as community, advertise as the supported way to
> interact and customize TripleO for individual needs.
> 
> Just for clarity's sake, obviously we welcome all contributions if
> they are improvements and are inline with the project's goals.
> 
> But, that doesn't mean that we advertise that the way to use TripleO
> is to directly modify the yaml workflows that we would deliver and
> release. And that we say we support that as a community, will deliver
> automatic updates for you in spite of your customizations, and will
> promise API compatibility in spite of your changes.
> 
> If someone finds that they must do modifications to files we ship,
> and they still choose to use TripleO, then it is a fork -- not saying
> that it's a big "evil" fork -- just that they have diverged from
> upstream TripleO, and are on their own for maintaining that until
> such time it's merged upstream.


I think this is an overstatement. There are parts of t-h-t which were
designed with an interface (nested stack parameters, etc.) such that
customers could create their own versions. The network templates which
drive os-net-config come to mind here.

Yes, we have some reference architecture templates that live in
tripleo-heat-templates, but those will never cover all the cases for
how low level physical networks can be configured. A smart
operator/user would go as far as writing a tool to generate these
templates automatically, and I would welcome this approach as an
improvement to upstream if we could make such a tool generic.

I agree with your statement below that all of this is probably
unrelated to the thread too BTW. Just wanted to point out that there
are some pieces of our deployment YAML templates that were actually
designed for customization. And that there may also be aspects of
workflows that might benefit from customization in a similar manner.

Dan



> 
> Now, whether they've modified yaml or python, I think it makes little
> difference. Obviously, yaml is likely easier to initially modify. But
> I'm not convinced those modifications are then any easier for that
> operator to have to maintain, update, and make compatible with future
> TripleO releases.
> 
> I think what I'm seeking in this discussion is getting off on the
> right foot in terms of defining the interface if we choose Mistral.
> In hopes to avoid the situation we have now with tripleo-heat-
> templates, where we have hacked up and customized templates being
> used in production that are very difficult to support as a community
> (if even possible at all). The ExtraConfig resources address just one
> piece of the problem. Things like required parameter changes, nic
> config changes requiring people to modify their nic templates
> continually, etc.
> 
> Anyway, my whole point here is not Mistral vs TripleO API...so I'm
> guilty of threadjacking a bit. I just get itchy when I see it held up
> as an advantage of Mistral of how easy it will be for people to
> customize yaml files that TripleO will release...without expressly
> stating that such situations should be the exception for operators,
> not the expectation.
> 
>  
> >  
> > Lets use another example - you have a proprietary revision control
> > system,
> > and you want to pull your golden templates from there, instead of
> > from
> > swift or the local filesystem.  Same problem!
> > 
> > The point is, it's *far* easier to make and maintain a simple
> > change to a
> > relatively constrained but general purpose workflow interface than
> > it is to
> > fork and maintain a bunch of python code indefinitely (unless we
> > reinvent a
> > plugin interface exactly like Mistral already has).
> Again, the point here is that whether you fork and maintain yaml or
> python...it's the same. You're maintaining somthing indefinitely
> until such time your change is accepted upstream.
>  
> >  Even if we were to decide the workflows were strictly internal
> > the same argument holds for developers - lets say the requirement I
> > outline
> > appears on your backlog tomorrow - how many days of python
> > development will 
> >  it take, vs the two-line change I outline?
>  
> 
> 
>  
> >  It's only an example, but I'm trying to illustrate there is
> > potentially
> > value in not reinventing every.single.wheel every time :)
> > 
> > Cheers,
> > 
> > Steve
> > 
> > ___________________________________________________________________
> > _______
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsu
> > bscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> 
> _____________________________________________________________________
> _____
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list