[openstack-dev] [Murano][Heat] MuranoPL questions?

Stan Lagun slagun at mirantis.com
Fri Mar 21 22:58:35 UTC 2014


I appreciate your explanations on Heat/HOT. This really makes sense.
I didn't mean to say that MuranoPL is better for Heat. Actually HOT is good
for Heat's mission. I completely acknowledge it.
I've tried to avoid comparison between languages and I'm sorry if it felt
that way. This is not productive as I don't offer you to replace HOT with
MuranoPL (although I believe that certain elements of MuranoPL syntax can
be contributed to HOT and be valuable addition there). Also people tend to
protect what they have developed and invested into and to be fair this is
what we did in this thread to great extent.

What I'm trying to achieve is that you and the rest of Heat team understand
why it was designed the way it is. I don't feel that Murano can become
full-fledged member of OpenStack ecosystem without a bless from Heat team.
And it would be even better if we agree on certain design, join our efforts
and contribute to each other for sake of Orchestration program.

I'm sorry for long mail texts written in not-so-good English and appreciate
you patience reading and answering them.

Having said that let me step backward and explain our design decisions.

Cloud administrators are usually technical guys that are capable of
learning HOT and writing YAML templates. They know exact configuration of
their cloud (what services are available, what is the version of OpenStack
cloud is running) and generally understands how OpenStack works. They also
know about software they intent to install. If such guy wants to install
Drupal he knows exactly that he needs HOT template describing Fedora VM
with Apache + PHP + MySQL + Drupal itself. It is not a problem for him to
write such HOT template.

Note that such template would be designed for very particular
configuration. There are hundreds of combinations that may be used to
install that Drupal - use RHEL/Windows/etc instead of Fedora, use
ngnix/IIS/etc instead of Apache, use FastCGI instead of mod_php, PostgreSQL
instead of MySQL. You may choose to have all software on single VM or have
one VM for database and another for Drupal. There are also constraints to
those combinations. For example you cannot have Fedora + IIS on the same
VM. You cannot have Apache and Drupal on different VMs.

So the HOT template represent fixed combination of those software
components. HOT may have input parameters like "username" or "dbImageName"
but the overall structure of template is fixed. You cannot have template
that choses whether to use Windows or Linux based on parameter value. You
cannot write HOT that accepts number of instances it allowed to create and
then decide what would be installed on each of them. This is just not
needed for Heat users.

With Murano the picture looks the opposite. Typical Murano user is a guy
who bought account from cloud hosting vendor (cloud operator) and want to
run some software in the cloud. He may not even be aware that it is
OpenStack. He knows nothing about programming in general and Heat in
particular. He doesn't want to write YAMLs. He may not know how exactly
Drupal is installed and what components it consists of.

So what he does is he goes to his cloud (Murano) dashboard, browses through
application catalog, finds Drupal and drags it onto his environment board
(think like Visio-style designer). He can stop at this point, click
"deploy" button and the system will deploy Drupal. In another words the
system (or maybe better to say cloud operator or application developer)
decides what set of components is going to be installed (like 1 Fedora VM
for MySQL and 1 CentOS VM for Apache-PHP-Drupal). But user may decide he
wants to customize his environment. He digs down and sees that Drupal
requires database instance and the default is MySQL. He clicks on a button
to see what are other options available for that role.

In Heat HOT developer is the user. But in Murano those are completely
different roles. There are developers that write application definitions
(that is DSL code) and there are end users who compose environments from
those applications (components). Application developers may have nothing to
do with particular cloud their application deployed on. As for Drupal
application the developer knows that Drupal can be run with MySQL or
PostgreSQL. But there may be many compatible implementations of those
DBMSes - Galera MySQL, TroveMySQL, MMM MySQL etc. So to get a list of what
components can be placed in a database role Murano needs to look at all
applications in Application Catalog and find which of them are compatible
with MySQL and PostgreSQL so that user could choose what implementation is
better suits his needs (trade performance for high-availability etc.).

User can go deeper and to decide that he wants that MySQL instance (this
can be 1 or more VMs depending on implementation) to be shared between
Drupal and another application in that environment (say WordPress). He can
go even deeper to VM level and decide that he wants to have WordPress,
Drupal, Apcache and slave MySQL node on one VM and MySQL master node on

The ultimate goal is to create sort of construction kit for applications.
Cloud LEGO. And let users build environments of any complexity from small
components while providing reasonable default so that one-click deployment
would also be possible. And such system to be useful the design process
need to be guided and driven by UI. System must know what combination of
components are possible and what are not and not let user create Microsoft
IIS hosted on Fedora.

This leads us to following design decisions:

a. We need a DSL that would allow describing those small components
independently in such a way that the environment could be composed from
components written by different people that are not aware of each other

b.  Components need to have properties (parameters in Heat terms).
Properties may be of different types (strings, numbers, booleans etc.).
There can be constraints on property values ("not null", "length > 5").

c. Components may reference (require, make use of) other components (Drupal
uses MySQL, hosted on Apache etc). There can be constraints for those

d. Constraints for both properties and references can be *very* complex and
indirect. Like "IIS can be deployed on any VM who's image is known to be
Windows with version >= WinServer2003". Or "my app requires 10 to 20
Windows instances that are all member of the same ActiveDirectory domain".
Or "I have 2 properties: minInstanceCount and maxInstanceCount and
constraint maxInstaceCount >= minInstanceCount". So we need a language to
express conditions of any complexity.

e. Both properties and references must be declared in a way that would make
possible for UI dashboard to discover their existence, ask user appropriate
input and validate the constraints on the fly.

f. Properties and especially references can have complex structure (lists,
maps, combination of them) rather than plain scalars. For example
ActiveDirectory consists of (e.g. references) *list* of domain controllers.
Constraints must respect that and be capable of validating things like
len(list) >= 2, all elements in list have some predicate true, there is at
least one element with desired set of properties etc.

g. Language need to have some sort mechanism to express polymorphism. E,g,
GaleraMySQL *is* MySQL (and thus can be used anywhere where "just MySQL"
can be). This also implies ability to have declare interfaces (class
contracts) and ability to implement those interface to that it would be
possible that two components are conform to the same interface.

h. If we want components to be reusable and easy to write we need to have
ability to one component to inherit/extend another. Inheritance implies
ability to override and customize some methods

i. We don't want to do deployment ourself. We don't intend to replace Heat,
Puppet etc. So we need a way to construct Heat templates that will do the
job. We need to map component's properties to
HOT parameters. This requires a language to express those bindings

j. Because there can be lists of references (or properties) it should be
possible to express something like
  for(domain_controller in self.controllers):


  if self.deploy_on_windows:


 for i in range(self.instance_count):

k. Need to provide most simple method to compose stack from pre-made HOT
snippets and avoid dynamic generation when its not really needed

l. Need to have some standard format to represent view of the world that is
what was designed in dashboard. In Murano it is JSON that describes object
graph with object property values and references to another objects. This
is somewhat similar to simple HOT template but with hierarchical structure

I think at this point it becomes obvious that above describes
object-oriented Turing-complete language. And the most important property
of that language is that *everything* is declared in a way useful to be
accessed from dashboard/app catalog. You not often see
properties/references/constraints declared in dynamic language. I've
answered several times already why this language cannot be JavaScript or
Lua. And this is just on the surface. There are many things under the hood
that makes MuranoPL stand out from all embeddable languages I know and
thats for a good reason.

And I didn't even mentioned ALM aspects, orchestration, and many many other
things that go far beyond what can be expressed by HOT-style DSL


Now few words ( :-))) on Mistral. In theory Mistral could be capable to
handle MuranoPL workflows and do what MuranoPL does on his own. But here
are the obstacles on this path:

1. Mistral follows Amazon SWF model and not BPEL model. 90% of SWF's power
comes from the fact the you don't work with it directly. With SWF you use
Java bindings/SDK and not consume it directly. So you still write code in
Turing-complete sophisticated language Java

2. Our initial idea was to translate MuranoPL into Mistral workflows
(similar to compiling C++ to assembly) but it turns to be too difficult to

3. The DSL Mistral currently has is very very limited and academic. You
have to be real genius with a twisted mind to write real-life workflows
with Mistral DSL.

4. Mistral is at PoC stage and it would take him years to grow to something
that is as powerful as MuranoPL (or maybe BPEL)

5. Mistral does not need all that OOP stuff, inheritance, constraints etc.
And in practice it is very hard to separate declarations from the workflow
code as they are very coupled in their nature

6. DSL is a hart and brain of Murano. It is the most vital part of it. You
cannot develop system of such level as Murano without having control over
it. Any decision taken by Mistral team can become fatal for Murano.

Anyway I hope that some day we will be able to somehow merge our

On Fri, Mar 21, 2014 at 10:20 PM, Zane Bitter <zbitter at redhat.com> wrote:

> I completely agree with Georgy, but you raised some questions about Heat
> that I want to answer in the interests of spreading knowledge about how
> Heat works. A heavily-snipped response follows...
> On 21/03/14 05:11, Stan Lagun wrote:
>> 3. Despite HOT being more secure on the surface it is not necessary so
>> in reality. There is a Python class behind each entry in resources
>> section of HOT template. That Python code is run with root privileges
>> and not guaranteed to be safe. People make mistakes, forget to validate
>> parameters, make incorrect assumptions etc. Even if the code is proven
>> to be secure every single commit can introduce security breach. And no
>> testing system can detect this.
> Quite right, I should acknowledge that it would be crazy to assume that
> HOT is secure just because it is not a programming language, and I do not
> make that assumption. (Indeed, YAML itself has been the subject of many
> security problems, though afaik not in the safe mode that we use in Heat.)
> Thanks for pointing out that I was not clear.
>      The operator can install whatever plugins they want.
>> They do but that is a bad solution. The reason is that plugins can
>> introduce additional resource types but they cannot modify existing
>> code. Most of the time cloud operators need to customize existing
>> resources' logic for their needs rather then rewriting it from scratch.
>> And they want their changes to be opaque to end-users. Imagine that
>> cloud operator need thats to get permission from his proprietary quota
>> management system for each VM spawned. If he would create custom
>> MyInstance resource type end-users could bypass it by using standard
>> Instance resource rather than custom one. Patching existing Python code
>> is not good in that then operator need to maintain his private fork of
>> the Heat and have troubles with CD, upgrades to newer versions etc.
> It's not as bad as you think. All of the things you mentioned were
> explicit design goals of the plugin system. If you install a plug-in
> resource with the same type as a built-in resource then it replaces the
> built-in one. And of course you can inherit from the existing plugin to
> customise it.
> So in this example, the operator would create a plugin like this:
>   from heat.engine.resources import server
>   from my.package import do_my_proprietary_quota_thing
>   class MyServer(server.Server):
>       def handle_create(self):
>           do_my_proprietary_quota_thing()
>           return super(MyServer, self).handle_create()
>   def resource_mapping():
>       return {'OS::Nova::Server': MyServer}
> and drop it in /usr/lib/heat. As you can see, this is a simple
> customisation (10 lines of code), completely opaque to end users
> (OS::Nova::Server is replaced), and highly unlikely to be broken by any
> changes in Heat (we consider the public APIs of
> heat.engine.resource.Resource as a contract with existing plugins that we
> can't break, at least without a lot of notice).
> (I'm ignoring here that if this is needed for _every_ server, it makes no
> sense to do it in Heat, unless you don't expose the Nova API to users at
> all.)
>  Besides plugin system is not secure because plugins run with the
>> privileges of Heat engine and while I may trust Heat developers
>> (community) but not necessary trust 3rd party proprietary plugin.
> I'm not sure who 'I' means in this context? As an end-user, you have no
> way of auditing what code your cloud provider is running in general.
>          What
>>         if he wants that auto-scaling would be based on input from his
>>         existing
>>         Nagios infrastructure rather then Ceilometer?
>>     This is supported already in autoscaling. Ceilometer just hits a URL
>>     for an alarm, but you don't have to configure it this way. Anything
>>     can hit the URL.
>>     And this is a good example for our general approach - we provide a
>>     way that works using built-in OpenStack services and a hook that
>>     allows you to customise it with your own service, running on your
>>     own machine (whether that be an actual machine or an OpenStack
>>     Compute server). What we *don't* do is provide a way to upload your
>>     own code that we then execute for you as some sort of secondary
>>     Compute service.
>> 1. Anything can hit the URL but it is auto-scaling resource that creates
>> Ceilometer alarms. And what should I do to make it create Nagios alarms
>> for example?
> That's incorrect, autoscaling doesn't create any alarms. You create an
> alarm explicitly using the Ceilometer API, or using an
> OS::Ceilometer::Alarm resource in Heat. Or not, if you want to use some
> other source for alarms. You connect them together by getting the alarm_url
> attribute from the autoscaling policy resource and passing it to the
> Ceilometer alarm, but you could also allow any alarm source you care to use
> to hit that URL.
>       [Ceilometer]                      [Heat]
>   Metrics ---> Alarm - - - - -> Policy ---> Scaling Group
>                          ^
>                       (webhook)
> A second option is that you can also feed metrics to the Ceilometer API
> yourself. In many cases this may be what you want; the hook exists more so
> that you can implement more complex policies than the ones that autoscaling
> supports natively. (Note that we *could* have defined a new language for
> implementing arbitrarily-complex policies and executing them in the
> autoscaling service, but instead we just added this hook.)
>       [Ceilometer]                                [Heat]
>   Metrics ---> Alarm - - ->|  Ext.  | - - -> Policy |
>                            | Policy |               |---> Scaling Group
>                            | Engine | - - -> Policy |
>   Metrics ---> Alarm ----->|        |
>         [Nagios]
>  2. Your approach has its cons and pros. I do acknowledge and respect
>> strong sides of such decision. But it has its limitations.
> Yes, I accept that it has limitations. And I even acknowledge that some
> people will see that as a Bad Thing ;)
>      Everything is a combination of existing resources, because the set
>>     of existing resources is the set of things which the operator
>>     provides as-a-Service. The set of things that the operator provides
>>     as a service plus the set of things that you can implement yourself
>>     on your own server (virtual or not) covers the entire universe of
>>     things. What you appear to be suggesting is that OpenStack must
>>     provide *Everything*-as-a-Service by allowing users to write their
>>     own services and have the operator execute them as-a-Service. This
>>     would be a breathtakingly ambitious undertaking, and I don't mean
>>     that in a good way.
>> 1. By existing resources I mean resource types that are available in
>> Heat. If I need to talk to Marconi during deployment but there is no
>> Marconi plugin yet available in my Heat installation or use the latest
>> feature introduced by yesterdays commit to Nova I'm in trouble.
> This is the cloud provider's responsibility to deal with. If your cloud
> provider provides a Message Queue service but doesn't provide a Heat plugin
> for it, you vote with your feet and find one that does. OpenStack is Open
> Source, and providers will be subject to these kinds of competitive
> pressures by design.
> (It has actually been suggested to require incubated projects to come up
> with Heat plugins, although not for this reason, but sadly that has been
> torpedoed for now by politicking on an unrelated topic.)
>  2. If you can implement something on user-land resources you can do the
>> same with Murano. It is not that Murano enforces to do it on server side.
>> 3. Not everything can be done from VMs. There are use cases when you
>> need to access cloud operator's proprietary services and even hardware
>> components that just cannot be done from user-land or they need to be
>> done prior to VM spawn.
> If a cloud provider provides proprietary services, it's up to the cloud
> provider to provide you with a way to access them. They have every
> incentive to do so.
>  1. As for autoscaling last time I checked (it may be fixed since then)
>> Heat's LoadBalancer spawned HAProxy on Fedora VM with hardcoded image
>> name and hardcoded nested stack template. This is not what I would call
>> highly-customizable solution. It is hard to imagine a generic enough
> Yah, it's terrible :D
> The current implementation is a kind of template/plugin hybrid, simply
> because it existed long before the provider feature we've been discussing.
> It needs to be reimplemented in a single template file where operators can
> easily modify it according to their needs. That said, there are no
> remaining known technical barriers to doing this.
> The good news is that with the provider templates feature, you can use
> your own definition (defined in a Heat template) for this, or indeed any
> other, resource type.
>  autoscaling implementation that could work with all possible
>> health-monitoring systems, load-balancers and would be as useful for
>> scaling RabbitMQ, MongoDB and MS SQL Server clusters as it is for web
> This is absolutely the goal for autoscaling in Heat. Notifying the load
> balancer is the hardest part, but some ideas have been proposed (including
> the same sort of notification system I'm suggesting for the workflow hooks).
>  farms. Surely you can have your own implementation on user-land
>> resources but why have you chose it to be Heat resource and not sample
>> HOT template in extras repository?
> Historical reasons. The implementation pre-dates provider templates by
> about a year, and exists primarily as a way of demonstrating our ability to
> launch existing CloudFormation templates. The recommended way, of course,
> is to use the Neutron load balancer resource, now that that exists. If you
> want to test on a cloud that doesn't support it, then that's a good time to
> use a provider template to replace it.
>  Besides we both want Heat and Murano
>> to be really useful and not just chef/puppet bootstappers :)
> FWIW I personally am completely fine with Heat being 'just' a Puppet/Chef
> bootstrapper. Heat's goal is to orchestrate *OpenStack* resources (i.e.
> infrastructure), not to replace configuration management. If replacing
> configuration management were the goal, I don't think it would belong in
> OpenStack, since that is neither something that should be tied to OpenStack
> (configuration management is useful everywhere), nor something OpenStack
> should be tied to (you should be able to use *any* configuration management
> system with OpenStack).
>  You know that there is a de facto standard DSL exists for workflow
>> definitions. It is called BPEL and it is Turing-complete and as
>> expressive as MuranoPL. There are alternative standards like BPMN, YAWL
>> and AFAIKS they are all Turing-complete.
> My feelings about those are not fit for publication ;)
>  So what makes you feel like
>> Mistral DSL doesn't need to be so?
> Amazon SWF.
> cheers,
> Zane.
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Sincerely yours
Stanislav (Stan) Lagun
Senior Developer
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
slagun at mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140322/590e5b3b/attachment.html>

More information about the OpenStack-dev mailing list