Open Stack

On Wed, Jan 27, 2016 at 10:22:54AM -0500, Dan Prince wrote:
> On Wed, 2016-01-27 at 09:36 -0500, Dan Prince wrote:
> > On Wed, 2016-01-27 at 14:32 +0100, Jiri Tomasek wrote:
> > > On 01/26/2016 09:05 PM, Ben Nemec wrote:
> > > > On 01/25/2016 04:36 PM, Dan Prince wrote:
> > > > > On Mon, 2016-01-25 at 15:31 -0600, Ben Nemec wrote:
> > > > > > On 01/22/2016 06:19 PM, Dan Prince wrote:
> > > > > > > On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > > > > > > > So I haven't weighed in on this yet, in part because I
> > > > > > > > was
> > > > > > > > on
> > > > > > > > vacation
> > > > > > > > when it was first proposed and missed a lot of the
> > > > > > > > initial
> > > > > > > > discussion,
> > > > > > > > and also because I wanted to take some time to order my
> > > > > > > > thoughts
> > > > > > > > on
> > > > > > > > it.
> > > > > > > >   Also because my initial reaction...was not conducive to
> > > > > > > > calm and
> > > > > > > > rational discussion. ;-)
> > > > > > > > 
> > > > > > > > The tldr is that I don't like it.  To explain why, I'm
> > > > > > > > going to
> > > > > > > > make
> > > > > > > > a
> > > > > > > > list (everyone loves lists, right? Top $NUMBER reasons we
> > > > > > > > should
> > > > > > > > stop
> > > > > > > > expecting other people to write our API for us):
> > > > > > > > 
> > > > > > > > 1) We've been down this road before.  Except last time it
> > > > > > > > was
> > > > > > > > with
> > > > > > > > Heat.
> > > > > > > >   I'm being somewhat tongue-in-cheek here, but expecting
> > > > > > > > a
> > > > > > > > general
> > > > > > > > service to provide us a user-friendly API for our
> > > > > > > > specific
> > > > > > > > use
> > > > > > > > case
> > > > > > > > just
> > > > > > > > doesn't make sense to me.
> > > > > > > We've been down this road with Heat yes. But we are
> > > > > > > currently
> > > > > > > using
> > > > > > > Heat for some things that we arguable should be (a
> > > > > > > workflows
> > > > > > > tool
> > > > > > > might
> > > > > > > help offload some stuff out of Heat). Also we haven't
> > > > > > > implemented
> > > > > > > custom Heat resources for TripleO either. There are mixed
> > > > > > > opinions
> > > > > > > on
> > > > > > > this but plugging in your code to a generic API is quite
> > > > > > > nice
> > > > > > > sometimes.
> > > > > > > 
> > > > > > > That is the beauty of Mistral I think. Unlike Heat it
> > > > > > > actually
> > > > > > > encourages you to customize it with custom Python actions.
> > > > > > > Anything
> > > > > > > we
> > > > > > > want in tripleo-common can become our own Mistral action
> > > > > > > (these get
> > > > > > > registered with stevedore entry points so we'd own the
> > > > > > > code)
> > > > > > > and
> > > > > > > the
> > > > > > > YAML workflows just tie them together via tasks.
> > > > > > > 
> > > > > > > We don't have to go off and build our own proxy deployment
> > > > > > > workflow
> > > > > > > API. The structure to do just about anything we need
> > > > > > > already
> > > > > > > exists
> > > > > > > so
> > > > > > > why not go and use it?
> > > > > > > 
> > > > > > > > 2) The TripleO API is not a workflow API.  I also largely
> > > > > > > > missed
> > > > > > > > this
> > > > > > > > discussion, but the TripleO API is a _Deployment_
> > > > > > > > API.  In
> > > > > > > > some
> > > > > > > > cases
> > > > > > > > there also happens to be a workflow going on behind the
> > > > > > > > scenes,
> > > > > > > > but
> > > > > > > > honestly that's not something I want our users to have to
> > > > > > > > care
> > > > > > > > about.
> > > > > > > Agree that users don't have to care about this.
> > > > > > > 
> > > > > > > Users can get as involved as they want here. Most users I
> > > > > > > think
> > > > > > > will
> > > > > > > use python-tripleoclient to drive the deployment or the new
> > > > > > > UI.
> > > > > > > They
> > > > > > > don't have to interact with Mistral directly unless they
> > > > > > > really
> > > > > > > want
> > > > > > > to. So whether we choose to build our own API or use a
> > > > > > > generic one
> > > > > > > I
> > > > > > > think this point is mute.
> > > > > > Okay, I think this is a very fundamental point, and I believe
> > > > > > it gets
> > > > > > right to the heart of my objection to the proposed change.
> > > > > > 
> > > > > > When I hear you say that users will use tripleoclient to talk
> > > > > > to
> > > > > > Mistral, it raises a big flag.  Then I look at something like
> > > > > > https://github.com/dprince/python-tripleoclient/commit/77ffd2
> > > > > > fa
> > > > > > 7b1642
> > > > > > b9f05713ca30b8a27ec4b322b7
> > > > > > and the flag gets bigger.
> > > > > > 
> > > > > > The thing is that there's a whole bunch of business logic
> > > > > > currently
> > > > > > sitting in the client that shouldn't/can't be there.  There
> > > > > > are
> > > > > > historical reasons for it, but the important thing is that
> > > > > > the
> > > > > > current
> > > > > > client architecture is terribly flawed.  Business logic
> > > > > > should
> > > > > > never
> > > > > > live in the client like it does today.
> > > > > Totally agree here. In fact I have removed business logic from
> > > > > python-
> > > > > tripleoclient in this patch and moved it into a Mistral action.
> > > > > Which
> > > > > can then be used via a stable API from anywhere.
> > > > > 
> > > > > > Looking at that change, I see a bunch of business logic
> > > > > > around
> > > > > > taking
> > > > > > our configuration and passing it to Mistral.  In order for us
> > > > > > to do
> > > > > > something like that and have a sustainable GUI, that code
> > > > > > _has_
> > > > > > to
> > > > > > live
> > > > > > behind an API that the GUI and CLI alike can call.  If we ask
> > > > > > the GUI
> > > > > > to
> > > > > > re-implement that code, then we're doomed to divergence
> > > > > > between
> > > > > > the
> > > > > > CLI
> > > > > > and GUI code and we'll most likely end up back where we are
> > > > > > with a
> > > > > > GUI
> > > > > > that can't deploy half of our features because they were
> > > > > > implemented
> > > > > > solely with the CLI in mind and made assumptions the GUI
> > > > > > can't
> > > > > > meet.
> > > > > The latest feedback I've gotten from working with the UI
> > > > > developers on
> > > > > this was that we should have a workflow to create the
> > > > > environment. That
> > > > > would get called via the Mistral API via python-tripleoclient
> > > > > and
> > > > > any
> > > > > sort of UI you could imagine and would essentially give us a
> > > > > stable
> > > > > environment interface.
> > > > Anything that requires tripleoclient means !GUI though.  I know
> > > > the
> > > > current GUI still has a bunch of dependencies on the CLI, but
> > > > that
> > > > seems
> > > > like something we need to fix, not a pattern to repeat.  I still
> > > > think
> > > > any sentence containing "call Mistral via tripleoclient" is
> > > > indicative
> > > > of a problem in the design.
> > > 
> > > I am not sure I understand the argument here.
> > > 
> > > Regardless of which API we use (Mistral API or TripleO API) GUI is
> > > going 
> > > to call the API and tripleoclient (CLI) is going to call the API 
> > > (through mistralclient - impl. detail).
> > > 
> > > GUI can't and does not call API through tripleoclient. This is why
> > > the 
> > > work on extracting the common business logic to tripleo-common
> > > happened. 
> > > So tripleo-common is the place which holds the business logic.
> > > 
> > > The proposed API (in the spec) is supposed only to work as a thin
> > > layer 
> > > that provides the tripleo-common functionality.
> > > 
> > > 
> > > Most of the operations we do in TripleO are workflows that consists
> > > of 
> > > several requests to various OpenStack services, e.g. Nodes 
> > > Introspection, deployment progress tracking, etc. Mistral fits as a
> > > best 
> > > tool to use to manage this and provides an API that can be used
> > > directly 
> > > by both CLI and GUI. The work involved is just about turning 
> > > tripleo-common into Mistral Actions.
> > > Alternatives are to create TripleO API and implement our own
> > > custom 
> > > workflows (reinvent the Mistral wheel) as part of  tripleo-common.
> > > Or 
> > > Shield the Mistral API behind a very thin TripleO API. Which in my 
> > > opinion is not worth it and just adds new layer to the project
> > > (GUI/CLI 
> > >  > TripleO API > mistralclient > Mistral API > tripleo-common
> > > actions)
> > > 
> > > But there are also some operations that do not require a workflow.
> > > Such 
> > > as listing available environments, posting the parameter values
> > > and 
> > > storing them in temporary environment... In those cases the Mistral
> > > API 
> > > does not work as good as TripleO API because Mistral API is able
> > > to 
> > > start a workflow but is not able to take a request and return an 
> > > immediate response to that request.
> > 
> > What I did in my demo/prototype was to just use Mistral's generic API
> > to update the environment directly. The things we want to store in
> > the
> > environment are actually quite simple:
> > 
> >  -container name (for heat templates)
> >  -set of selected heat environments
> >  -extra parameters
> >  -cached things to help UI select these things like a capabilities
> > map
> > 
> > Last time we talked on there was interest in creating an "interface"
> > around managing creating/updating/deleting these things... and I
> > suggested we could use a workflow to do it just the same.
> > 
> > That isn't my preference, I'm fine with defining these as a loose
> > data
> > structure (a set of properties really). If we do it that we you can
> > simply use the Mistral REST API to update an environment quite
> > nicely.
> > 
> > If we prefer a more rigidly defined data structure we could wrap the
> > same logic with workflows to manage it. The UI/CLI could still read
> > the
> > information directly via the Mistral environments API, and just use a
> > workflow to update it. Or we could use workflows for everything. Any
> > combination would be possible here.
> > 
> > Mistral environments are a useful abstraction for workflows. At the
> > end
> > of the day nothing would prevent an end user from using mistralclient
> > to manage the same thing via a JSON file... But if we really want to
> > wrap it with a workflow we really could do that. I think for the
> > simple
> > data construct we are talking about here it really is fine to manage
> > as
> > just a JSON POST request to Mistral's environments API:
> > 
> > http://paste.openstack.org/show/485140/
> > 
> > Dan
> > 
> > 
> > >  The Mistral workflow needs to notify 
> > > (e.g. via Zaqar) or get polled for a result of the Workflow
> > > execution.
> 
> I think I misread your request here. For now polling is required. I'd
> just create a common Python and/or Javascript function to do this for
> python-tripleoclient and the UI. That way you can call a workflow and
> wait on the result in a synchronous manner if you want a convenient way
> to do it.
> 
> Lets do proceed with looking at Zaqar integration and see if that
> helps. That seems like the best long term solution. I don't think it
> would be very hard to integrate with Zaqar within our workflows...

As a first step, I should point out that heat already supports Zaqar for
event notification, via a simple addition to the heat environment:

http://docs.openstack.org/developer/heat/template_guide/environment.html#retrieving-events

So, it might make sense as an interim step to not poll for events from the
heat stack while the deployment workflow is executing - we could just get
events via Zaqar instead (which will probably be a lot more efficient).

For example, here's some test code showing waiting for a completion event
via websocket and Zaqar (thanks therve!):

https://gist.github.com/therve/9c7c762d3eb81b076be4

Sorry, this is a bit OT from this thread, but I wanted to point out we do
have some Zaqar integration already (just not yet directly in Mistral AIUI)

Steve