Open Stack

On Wed, 2016-01-13 at 04:41 -0500, Tzu-Mainn Chen wrote:
> Hey all,
> 
> I realize now from the title of the other TripleO/Mistral thread [1]
> that
> the discussion there may have gotten confused.  I think using Mistral
> for
> TripleO processes that are obviously workflows - stack deployment,
> node
> registration - makes perfect sense.  That thread is exploring
> practicalities
> for doing that, and I think that's great work.
> 
> What I inappropriately started to address in that thread was a
> somewhat
> orthogonal point that Dan asked in his original email, namely:
> 
> "what it might look like if we were to use Mistral as a replacement
> for the
> TripleO API entirely"
> 
> I'd like to create this thread to talk about that; more of a 'should
> we'
> than 'can we'.  And to do that, I want to indulge in a thought
> exercise
> stemming from an IRC discussion with Dan and others.  All, please
> correct me
> if I've misstated anything.
> 
> The IRC discussion revolved around one use case: deploying a Heat
> stack
> directly from a Swift container.  With an updated patch, the Heat CLI
> can
> support this functionality natively.  Then we don't need a TripleO
> API; we
> can use Mistral to access that functionality, and we're done, with no
> need
> for additional code within TripleO.  And, as I understand it, that's
> the
> true motivation for using Mistral instead of a TripleO API: avoiding
> custom
> code within TripleO.

The True motivation for investigating Mistral was to counter the
assertion that we needed to build our own REST API for workflows in the
TripleO API spec. This:

 "We need a REST API that supports the overcloud deployment workflow."

https://review.openstack.org/#/c/230432/13/specs/mitaka/tripleo-overclo
ud-deployment-api.rst

In doing that I was trying to wrap some of the existing code in
tripleo-common in Mistral actions and it occurred to me that some
things (like the ability to deploy heat templates from Swift) might
benefit others more if we put them in the respective client libraries
(like heatclient) instead of carrying them in tripleo-common where they
only benefit us. Especially since in the heatclient case it already had
a --template-object option to begin with (https://bugs.launchpad.net/py
thon-heatclient/+bug/1532326)

This goes towards a similar pattern we follow with other components
like Puppet which is that instead of trying to create our own
functionality in puppet-tripleo we are trying to as much as possible
add the features we need to the individual modules for each project
(puppet-nova, puppet-swift, etc).

In other words avoiding custom code within TripleO is just good
practice in general, not something that is specific to the Mistral vs
TripleO API discussion. When someone looks at TripleO as a project I
would very much like them to admire our architecture... not what we've
had to build to support it. As an example I'm very glad to see TripleO
be a bit more friendly to config management tooling... rather than
trying to build our own version (os-apply-config, etc.). And adding
code to the right place usually works out better for everyone and can
help build up the larger OpenStack community too.

> 
> That's definitely a worthy goal... except from my perspective, the
> story
> doesn't quite end there.  A GUI needs additional functionality, which
> boils
> down to: understanding the Heat deployment templates in order to
> provide
> options for a user; and persisting those options within a Heat
> environment
> file.

TripleO API was previously described as what would be our "workflows
API" for TripleO. A places for common workflows that are used for the
CLI, and UI together (one code path for all!).

What you describe here sounds like a different sort of idea which is a
GUI helper sort of API. FWIW I think it is totally fine for a GUI to
maintain its own API for caching reasons etc if the members of that
team find it to be a requirement. Please don't feel that any of the
workflow API comparisons block requirements for things to build a GUI
though.


> 
> Right away I think we hit a problem.  Where does the code for
> 'understanding
> options' go?  Much of that understanding comes from the capabilities
> map
> in tripleo-heat-templates [2]; it would make sense to me that
> responsibility
> for that would fall to a TripleO library.
> 
> Still, perhaps we can limit the amount of TripleO code.  So to give
> API
> access to 'getDeploymentOptions', we can create a Mistral workflow.
> 
>   Retrieve Heat templates from Swift -> Parse capabilities map

I have no issues with this workflow if it helps the UI team to create
say a cached copy of the capabilities map.

Similar to the above heatclient discussion I would ask why put this
into a TripleO library if we don't have to. Could this code perhaps be
made into a more generic Heat feature? If not immediately, then perhaps
eventually? What I mean is... it would be totally fine to write this
code in say TripleO common with an eye towards moving it eventually to
someplace better like Heat API (proper) or heatclient perhaps.

Regardless of where the code for parsing the capabilities map lives
though I think it would be a fine use of a Mistral workflow.

> 
> Which is fine-ish, except from an architectural perspective
> 'getDeploymentOptions' violates the abstraction layer between storage
> and
> business logic, a problem that is compounded because
> 'getDeploymentOptions'
> is not the only functionality that accesses the Heat templates and
> needs
> exposure through an API.  And, as has been discussed on a separate
> TripleO
> thread, we're not even sure Swift is sufficient for our needs; one
> possible
> consideration right now is allowing deployment from templates stored
> in
> multiple places, such as the file system or git.
> 
> Are we going to have duplicate 'getDeploymentOptions' workflows for
> each
> storage mechanism?  If we consolidate the storage code within a
> TripleO
> library, do we really need a *workflow* to call a single
> function?  Is a
> thin TripleO API that contains no additional business logic really so
> bad
> at that point?

Do we really need our TripleO libaries to support multiple storage
backends at all right now. As a cloud deployer I can store my heat
templates wherever I like... git, svn, cvs... whatever. When I'm ready
to deploy I would then use python-tripleoclient or the UI to "upload"
my local copy of the templates into Swift.  At this point I could
either use the UI or python-tripleoclient to drive the rest of the
deployment to completion. The fact that the TripleO workflows use Swift
for storage is actually abstracted from me such that I don't really
care much.

> 
> My gut reaction is to say that proposing Mistral in place of a
> TripleO API
> is to look at the engineering concerns from the wrong direction.  The
> Mistral alternative comes from a desire to limit custom TripleO code
> at all
> costs.  I think that is an extremely dangerous attitude that leads to
> compromises and workarounds that will quickly lead to a shaky code
> base
> full of design flaws that make it difficult to implement or extend
> any
> functionality cleanly.

Similar to what I said above limiting code in TripleO isn't necessarily
the primary desire here I think. It is more of a question of can we use
a generic workflow API like Mistral, or should we go and build our own.

Whether more or less of the code lives in a TripleO common library is
something we can debate and take a step at a time I think.

> 
> I think the correct attitude is to simply look at the problem we're
> trying to solve and find the correct architecture.  For these get/set
> methods that the API needs, it's pretty simple: storage -> some logic
> ->
> a REST API.  Adding a workflow engine on top of that is unneeded, and
> I
> believe that means it's an incorrect solution.

Totally agree on finding the correct architecture. What you suggest as
a "dangerous attitude" is actually just trying to ask the question that
to me have not yet been asked. Trying out tools and solutions we may
not have tried yet. (FWIW I found the dangerous attitude bit to be a
bit un-constructive to this conversation and would prefer to leave that
verbiage elsewhere)

If you feel strongly that this particular case of parsing the
capabilities map and managing config settings is not suitably
implemented with a generic workflow tool then perhaps we implement both
and see how it works out. Continue work on creating TripleO API...

I would however like to perhaps see us scale back selling TripleO API
as our generic workflow API. Perhaps there still is a niche where it
helps with some things and I've got no issue with pursuing this idea. I
will likely continue to ask the questions about putting code into the
right place (like could some of this be implemented as a Heat API
feature instead). Perhaps this just comes in time after we've refined
our deployment workflows a bit to accomidate but UI and CLI.

Dan

> 
> 
> Thanks,
> Tzu-Mainn Chen
> 
> 
> 
> [1] http://lists.openstack.org/pipermail/openstack-dev/2016-January/0
> 83757.html
> [2] https://github.com/openstack/tripleo-heat-templates/blob/master/c
> apabilities_map.yaml
> 
> _____________________________________________________________________
> _____
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev