Open Stack

Steven,

Agree with your opinion on HOT expansion. I see that inclusion of
imperative workflows and ALM would require major Heat redesign and probably
would be impossible without loosing compatibility with previous HOT syntax.
It would blur Heat mission, confuse current users and rise a lot of
questions what should and what should not be in Heat. Thats why we chose to
built a system on top of Heat rather then expending HOT.

Now I would like to clarify why have we chosen imperative approach with DSL.

You see a DSL as an alternative to HOT but it is not. DSL is alternative to
Python-encoded resources in Heat (heat/engine/resources/*.py). Imagine how
Heat would look like if you let untrusted users to upload Python plugins to
Heat engine and load them on the fly. Heat resources are written in Python
which is imperative language. So that MuranoPL for the same reason.

We want application authors to be able to express application deployment
and maintenance logic of any complexity. This may involve communication
with 3rd party REST services (APIs of applications being deployed, external
services like DNS server API, application licensing server API, billing
systems, some hardware component APIs etc) and internal OpenStack services
like Trove, Sahara, Marconi and others including those that are not
incubated yet and those to come in the future. You cannot have such things
in HOT and when you required to you need to develop custom resource in
Python. Independence  on custom plugins is not good for Murano because they
cannot be uploaded by end users and thus he cannot write application
definition that can be imported to/run on any cloud and need to convince
cloud administrator to install his Python plugin (something that is
unimaginable in real life).

Because DSL is a way to write custom resources (in Heats terminology) it
has to be Turing-complete and have all the characteristics of
general-purpose language. It also has to have domain-specific features
because we cannot expect that DSL users would be as skilled as Heat
developers and could write such resources without knowledge on hosting
engine architecture and internals.

HOT DSL is declarative because all the imperative stuff is hardcoded into
Heat engine. Thus all is left for HOT is to define "state of the world" -
desired outcome. That is analogous to Object Model in Murano (see [1]). It
is Object Model that can be compared to HOT, not DSL. As you can see it not
more complex than HOT. Object Model is what end-user produces in Murano.
And he event don't need to write it cause it can be composed in UI.

Now because DSL provides not only a way to write sandboxed isolated code
but also a lot of declarations (classes, properties, parameters,
inheritance and contracts) that are mostly not present in Python we don't
need Parameters or Output sections in Object Model because all of this can
be inferred from resource (classes) DSL declaration. Another consequence is
that most of the things that can be written wrong in HOT can be verified on
client side by validating classes' contracts without trying to deploy the
stack and then go through error log debugging. Because all resources'
attributes types their constraints are known in advance (note that resource
attribute may be a reference to another resource with constraints on that
reference like "I want any (regular, Galera etc) MySQL implementation") UI
knows how to correctly compose the environment and can point out your
mistakes at design time. This is similar to how statically typed languages
like C++/Java can do a lot of validation at compile time rather then in
runtime as in Python.

Personally I would love to see many of this features in HOT. What is your
vision on this? What of the mentioned above can be contributed to Heat? We
definitely would like to integrate more with HOT and eliminate all
duplications between projects. I think that Murano and Heat are
complimentary products that can effectively coexist. Murano provides access
to all HOT features and relies on Heat for most of its activities. I
believe that we need to find an optimal way to integrate Heat, Murano,
Mistral, Solum, Heater, TOSCA, do some integration between ex-Thermal and
Murano Dashboard, be united regarding Glance usage for metadata and so on.
We are okay with throwing MuranoPL out if the issues it solves would be
addressed by HOT.

If you have a vision on how HOT can address the same domain MuranoPL does
or any plans for such features in upcoming Heat releases I would ask you to
share it.

[1] https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Object_model



On Wed, Mar 19, 2014 at 8:06 AM, Steven Dake <sdake at redhat.com> wrote:

> Ruslan,
>
> Some of my thoughts on the evolution of the HOT DSL to date.
>
>
> On 03/18/2014 05:32 PM, Ruslan Kamaldinov wrote:
>
>> Here is my 2 cents:
>>
>> I personally think that evolving Heat/HOT to what Murano needs for it's
>> use
>> cases is the best way to make PaaS layer of OpenStack to look and feel as
>> a
>> complete and fully integrated solution.
>>
>> Standardising these things in a project like TOSCA is another direction
>> we all
>> should follow. I think that TOSCA is the place where developers (like us),
>> application developers and enterprises can collaborate to produce a common
>> standard for application lifecycle management in the clouds.
>>
>>
>> But before Murano contributors jump into direction of extending HOT to
>> the goal
>> of application (or system) lifecycle management, we need an agreement
>> that this
>> is the right direction for Heat/HOT/DSL and the Orchestration program.
>> There are
>> a lot of use cases that current HOT doesn't seem to be the right tool to
>> solve.
>> As it was said before, it's not a problem to collaborate on extending it
>> those
>> use cases. I'm just unsure if Heat team would like these use cases to be
>> solved
>> with Heat/HOT/DSL. For instance:
>> - definition of an application which is already exposed via REST API.
>> Think of
>>    something like Sahara (ex. Savanna) or Trove developed in-house for
>> internal
>>    company needs. app publishers wouldn't be happy if they'll be forced to
>>    develop a new resource for Heat
>> - definition of billing rules for an application
>>
>>
>> If everyone agrees that this is the direction we all should follow, that
>> we
>> should expand HOT/DSL to that scope, that HOT should be the answer on
>> "can you
>> express it?", then awesome - we can start speaking about implementation
>> details.
>>
>> If it's not the direction these projects should follow then at least
>> finding
>> where Heat ends and Murano starts to avoid any functionality duplication
>> would
>> be great.
>>
>
> The HOT DSL for the most part, either by design or subconscious
> development choices, enables the application of Miller's Law[1] in a
> positive way.  HOT as a DSL takes less then a few hours to learn and use
> effectively.  Its relative simplicity is its *key* advantage as a DSL.
>  DSL's by their very nature declare a desired state.  It is the
> responsibility of the DSL processor to convert that desired state into
> reality.  On a fundamental level, this is precisely what Heat does.
>
> A DSL by its very definition is meant to express a desired outcome without
> specifying the intermediate steps.  To express the intermediate steps would
> require recording state in variables and offering conditional operations on
> those variables.  This implies individual steps in the processing of the
> input to the language.  If HOT were to add these sorts of features, it
> would no longer be a DSL, but a general purpose language (perhaps less
> general purpose then python or C).  A DSL is by definition a declarative
> language. I don't like the idea of expanding the scope of HOT to add an
> imperative model of operation.
>
> Learning imperative languages takes inordinately more time and brainpower
> then learning declarative languages, especially those which generally
> follow the advantages provided by languages operating inside the
> constraints of Miller's Law.  We want Heat to be dead simple to explain and
> learn.  Realistically I'd like folks to be able to write a template in
> under an hour with 15 minutes of explanation, and I think we have hit that
> mark.
>
> The idea of expanding the scope of the Heat APIs and engine to include ALM
> and Workflow don't make sense to me from an engineering perspective.  It
> over-complicates the code base.  I know we have already covered those
> thoughts in detail on the mailing list previously and the Murano folks
> agree that is a bad idea.
>
> I see a parallel between expanding the scope of HOT to support ALM and
> Workflow and expanding the scope of the heat-engine in the same fashion
> that is not appealing.  What would make more sense is to follow the general
> laws of Unix (do one thing, do it well) and layer these other possibly
> imperative languages on top of Heat using HOT and the Heat APIs to
> implement such imperative programming models. Then if someone really wanted
> to invest in the complexity of ALM or Workflow, they may be more willing to
> invest in learning the complexity of a new imperative programming language.
>
> My personal opinion is expanding the scope of HOT to include imperative
> programming models is not desirable for Heat in isolation.  I understand
> such an outcome may be appealing as a holistic approach to handling the
> entire orchestration space, but feel the costs of learning an imperative
> model for HOT do not pay for the advantages of having only one language to
> program all the things.
>
> I see no issue with HOT remaining simple and tidy focused entirely on
> orchestration (taking a desired state and converting that into reality)
> with some other imperative language layered on top to handle workflow and
> ALM.  I believe this separation of concerns is best for OpenStack and
> should be the preferred development path.
>
> Regards,
> -steve
>
> [1] http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_
> Plus_or_Minus_Two
>
>
>> Thanks,
>> Ruslan
>>
>> On Wed, Mar 19, 2014 at 2:07 AM, Keith Bray <keith.bray at rackspace.com>
>> wrote:
>>
>>> Georgy,
>>>
>>> In consideration of the "can you express it" instead of the "who will
>>> generate it," I see Heat's HOT evolving to support the expression of
>>> complex
>>> multi-tier architectures and applications (I would argue you can already
>>> do
>>> this today, perhaps with some additional features desired, e.g. Ability
>>> to
>>> define cloud workflows and workflow execution rules which could come
>>> when we
>>> have a workflow service like Mistral).  Therefore, I would encourage
>>> Murano
>>> contributors to consider whether they can help make Heat sufficiently
>>> cover
>>> desired use cases.  I have never viewed Heat templates as isolated
>>> components of a multi-tier architecture.  Instead, a single template or a
>>> combination of master/subordinate templates together (using references,
>>> nesting, or inclusion) could express the complete architecture, both
>>> infrastructure and applications.
>>>
>>> If I've read your previous comments and threads correctly, you desire a
>>> way
>>> to express System Lifecycle Management across multiple related
>>> applications
>>> or components, whereby you view the System as a grouping of independently
>>> developed and/or deployed (but systematically related) "components,"
>>> whereby
>>> you view Components as individual disconnected Heat templates that
>>> independently describe different application stacks of the System.  Did I
>>> get that correct?   If so, perhaps the discussion here is one of "scope"
>>> of
>>> what can or should be expressed in a Heat template. Is it correct to
>>> state
>>> that your argument is that a separate system (such as Murano) should be
>>> used
>>> to express System Lifecycle Management as I've defined it here?  If so,
>>> why
>>> could we not use the Heat DSL to also define the System?  The System
>>> definition could be logically separated out into its own text file...
>>> But,
>>> we'd have a common DSL syntax and semantics for both lower level and
>>> higher
>>> level component interaction (a building block effect of sorts).
>>>
>>> As for "who will generate it," ( with "it" being the Heat multi-tier
>>> application/infrastructure definition) I think that question will go
>>> through
>>> a lot more evolution and could be any number of sources: e.g. Solum,
>>> Murano,
>>> Horizon, Template Author with a text editor, etc.
>>>
>>> Basically, I'm a +1 for as few DSLs as possible. I support the position
>>> that
>>> we should evolve HOT if needed vs. having two separate DSLs that are both
>>> related to expressing application and infrastructure semantics.
>>>
>>> Workflow is quite interesting ... Should we be able to express imperative
>>> workflow semantics in HOT?  Or, should we only be able to declare
>>> workflow
>>> configurations that get configured in a service like Mistral whereby
>>> Mistral's execution of a workflow may need to invoke Heat hooks or Stack
>>> Updates?  Or, some other solution?
>>>
>>> I look forward to a design discussion on all this at the summit... This
>>> is
>>> fun stuff to think about!
>>>
>>> -Keith
>>>
>>> From: Georgy Okrokvertskhov <gokrokvertskhov at mirantis.com>
>>>
>>> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
>>> <openstack-dev at lists.openstack.org>
>>> Date: Tuesday, March 18, 2014 1:49 PM
>>>
>>> To: "OpenStack Development Mailing List (not for usage questions)"
>>> <openstack-dev at lists.openstack.org>
>>> Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
>>>
>>> I see this in the following way - who will generate HOT template for my
>>> complex multi-tier applications when I have only templates for
>>> components?
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>  _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Sincerely yours
Stanislav (Stan) Lagun
Senior Developer
Mirantis
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
www.mirantis.com
slagun at mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140319/4ab8ac7c/attachment.html>