Open Stack

On 01/14/2016 12:54 PM, Steven Hardy wrote:
> On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:
>> Hey all,
>>
>> I realize now from the title of the other TripleO/Mistral thread [1] that
>> the discussion there may have gotten confused.  I think using Mistral for
>> TripleO processes that are obviously workflows - stack deployment, node
>> registration - makes perfect sense.  That thread is exploring practicalities
>> for doing that, and I think that's great work.
>>
>> What I inappropriately started to address in that thread was a somewhat
>> orthogonal point that Dan asked in his original email, namely:
>>
>> "what it might look like if we were to use Mistral as a replacement for the
>> TripleO API entirely"
>>
>> I'd like to create this thread to talk about that; more of a 'should we'
>> than 'can we'.  And to do that, I want to indulge in a thought exercise
>> stemming from an IRC discussion with Dan and others.  All, please correct me
>> if I've misstated anything.
>>
>> The IRC discussion revolved around one use case: deploying a Heat stack
>> directly from a Swift container.  With an updated patch, the Heat CLI can
>> support this functionality natively.  Then we don't need a TripleO API; we
>> can use Mistral to access that functionality, and we're done, with no need
>> for additional code within TripleO.  And, as I understand it, that's the
>> true motivation for using Mistral instead of a TripleO API: avoiding custom
>> code within TripleO.
>>
>> That's definitely a worthy goal... except from my perspective, the story
>> doesn't quite end there.  A GUI needs additional functionality, which boils
>> down to: understanding the Heat deployment templates in order to provide
>> options for a user; and persisting those options within a Heat environment
>> file.
>>
>> Right away I think we hit a problem.  Where does the code for 'understanding
>> options' go?  Much of that understanding comes from the capabilities map
>> in tripleo-heat-templates [2]; it would make sense to me that responsibility
>> for that would fall to a TripleO library.
>>
>> Still, perhaps we can limit the amount of TripleO code.  So to give API
>> access to 'getDeploymentOptions', we can create a Mistral workflow.
>>
>>    Retrieve Heat templates from Swift -> Parse capabilities map
>>
>> Which is fine-ish, except from an architectural perspective
>> 'getDeploymentOptions' violates the abstraction layer between storage and
>> business logic, a problem that is compounded because 'getDeploymentOptions'
>> is not the only functionality that accesses the Heat templates and needs
>> exposure through an API.  And, as has been discussed on a separate TripleO
>> thread, we're not even sure Swift is sufficient for our needs; one possible
>> consideration right now is allowing deployment from templates stored in
>> multiple places, such as the file system or git.
> Actually, that whole capabilities map thing is a workaround for a missing
> feature in Heat, which I have proposed, but am having a hard time reaching
> consensus on within the Heat community:
>
> https://review.openstack.org/#/c/196656/
>
> Given that is a large part of what's anticipated to be provided by the
> proposed TripleO API, I'd welcome feedback and collaboration so we can move
> that forward, vs solving only for TripleO.

Yes, the original intent was to provide user with a means to safely 
construct the deployment template tree which iiuc is what this proposed 
feature provides. Then in the process of figuring out how to provide 
user with a reasonable choices for designing the deployment, I realized 
that more natural way is to provide the choices on the environments 
level (which is what we already do in THT - provide the list of 
alternative/complementary environments) The capabilities map provides a 
list of environments which user is able to choose from when designing 
the deployment supporting the choice with environment description and 
information about whether the environment is required to use, if it 
depends on the use of other environment or if the environment is one of 
the mutually exclusive choices.

So in addition to what the mentioned spec defines, to implement a 
replacement for capabilities_map in Heat, we'd need a way to specify 
environment description, other environments dependency, mutual 
exclusivity to other environments etc from within environment itself. 
Heat then could return a response which resembles to what 
capabilities_map provides now.

Maybe the current spec is sufficient and based on the information that 
this spec brings Heat would already be able to identify the 
inter-environment behavior based on parsing the resource_registry in the 
environments? Only thing we'd need to add is a 'description' to 
environment file.

-- Jirka

>
>> Are we going to have duplicate 'getDeploymentOptions' workflows for each
>> storage mechanism?  If we consolidate the storage code within a TripleO
>> library, do we really need a *workflow* to call a single function?  Is a
>> thin TripleO API that contains no additional business logic really so bad
>> at that point?
> Actually, this is an argument for making the validation part of the
> deployment a workflow - then the interface with the storage mechanism
> becomes more easily pluggable vs baked into an opaque-to-operators API.
>
> E.g, in the long term, imagine the capabilities feature exists in Heat, you
> then have a pre-deployment workflow that looks something like:
>
> 1. Retrieve golden templates from a template store
> 2. Pass templates to Heat, get capabilities map which defines features user
> must/may select.
> 3. Prompt user for input to select required capabilites
> 4. Pass user input to Heat, validate the configuration, get a mapping of
> required options for the selected capabilities (nested validation)
> 5. Push the validated pieces ("plan" in TripleO API terminology) to a
> template store
>
> This is a pre-deployment validation workflow, and it's a superset of the
> getDeploymentOptions feature you refer to.
>
> Historically, TripleO has had a major gap wrt workflow, meaning that we've
> always implemented it either via shell scripts (tripleo-incubator) or
> python code (tripleo-common/tripleo-client, potentially TripleO API).
>
> So I think what Dan is exploring is, how do we avoid reimplementing a
> workflow engine, when a project exists which already does that.
>
>> My gut reaction is to say that proposing Mistral in place of a TripleO API
>> is to look at the engineering concerns from the wrong direction.  The
>> Mistral alternative comes from a desire to limit custom TripleO code at all
>> costs.  I think that is an extremely dangerous attitude that leads to
>> compromises and workarounds that will quickly lead to a shaky code base
>> full of design flaws that make it difficult to implement or extend any
>> functionality cleanly.
> I think it's not about limiting TripleO code at all costs, it's about
> learning from past mistakes, where long-term TripleO specific workarounds
> for gaps in other projects have become serious technical debt.
>
> For example, the old merge.py approach to template composition was a
> workaround for missing heat features, then Tuskar was another workaround
> (arguably) for missing heat features, and now we're again proposing a
> long-term workaround for some missing heat features, some of which are
> already proposed (referring to the API for capabilities resolution).
>
>> I think the correct attitude is to simply look at the problem we're
>> trying to solve and find the correct architecture.  For these get/set
>> methods that the API needs, it's pretty simple: storage -> some logic ->
>> a REST API.  Adding a workflow engine on top of that is unneeded, and I
>> believe that means it's an incorrect solution.
> What may help is if we can work through the proposed API spec, and
> identify which calls can reasonably be considered workflows vs those where
> it's really just proxying an API call with some logic?
>
> When we have a defined list of "not workflow" API requirements, it'll
> probably be much easier to rationalize over the value of a bespoke API vs
> mistral?
>
> Steve
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev