[openstack-dev] [Murano][Heat] MuranoPL questions?

Georgy Okrokvertskhov gokrokvertskhov at mirantis.com
Fri Mar 21 02:21:25 UTC 2014

Hi Zane,

Thank you for the deep analysis. I agree that doing these deep technical
debates around languages hides the real problem we should focus on. Thank
you for bringing back to the light.

You are right, TC highlighted that Murano did a huge step forward up to
stack probably passing some important areas which should be covered in
OpenStack before that. Workflows area is one of them. I think we need bring
Mistral team here to find out how to align this workflows stuff with all
three project Heat, Murano and Mistral.  As Heat also wants to have a hooks
mechanism to work with external workflow system and Murano uses Heat and
workflows I think it will a lot of sense to design this properly together.
It makes sense to bring TOSCA guys too as they have similar concepts in
standard and Heat eventually will support TOSCA format. Having this aligned
I see a Murano package as an archive with all necessary definitions and
resources and Murano service will just properly pass them to related
services like Heat and Mistral. I think then Murano DSL will be much more
simple and probably will be closer to declarative format with some specific
data operations.

What will be the best way to do this? I know that Heat team is pretty busy
now with Icehouse release which will happen pretty soon.

I would like to have some draft before summit in Atlanta. We can polish
them on the summit sessions but before some home work it will be hard.


On Thu, Mar 20, 2014 at 5:28 PM, Zane Bitter <zbitter at redhat.com> wrote:

> On 19/03/14 17:13, Stan Lagun wrote:
>> On Wed, Mar 19, 2014 at 9:18 PM, Zane Bitter <zbitter at redhat.com
>> <mailto:zbitter at redhat.com>> wrote:
>>     On 19/03/14 05:00, Stan Lagun wrote:
>>         Steven,
>>         Agree with your opinion on HOT expansion. I see that inclusion of
>>         imperative workflows and ALM would require major Heat redesign and
>>         probably would be impossible without loosing compatibility with
>>         previous
>>         HOT syntax. It would blur Heat mission, confuse current users
>>         and rise a
>>         lot of questions what should and what should not be in Heat.
>>         Thats why
>>         we chose to built a system on top of Heat rather then expending
>> HOT.
>>     +1, I agree (as we have discussed before) that it would be a mistake
>>     to shoehorn workflow stuff into Heat. I do think we should implement
>>     the hooks I mentioned at the start of this thread to allow tighter
>>     integration between Heat and a workflow engine (i.e. Mistral).
>>     So building a system on top of Heat is good. Building it on top of
>>     Mistral as well would also be good, and that was part of the
>>     feedback from the TC.
>>     To me, building on top means building on top of the languages (which
>>     users will have to invest a lot of work in learning) as well, rather
>>     than having a completely different language and only using the
>>     underlying implementation(s).
>> 1. Murano application developers (publishers) are using Heat directly.
>> It is not the case that Murano executes some HOT templates under the
>> hood but HOT templates are essential part of application definition.
>> Developers still write HOT templates in HOT syntax
>> 2. Please get me right here. It is not that we wanted to develop another
>> language just for fun or to stand out from community. It is not that we
>> wrote DSL and then started to think how we gonna use it and prove to
>> others that is needed. We completely understand all concerns regarding
>> new language. The only reason is that we had very concrete list of
>> problems and use cases that we wanted to address in Murano. We did
>> investigate on using HOT, Mistral, JavaScript, Lua and BPEL and we found
>> overwhelming obstacles with each of those approaches. It don't pretend
>> that having own DSL is good. Just that not having it is much worse. I'm
>> also not a HOT expert as you are and thus can be (partially) wrong about
>> HOT and not aware of some of its power features. If so as a technical
>> guys we would quickly come to consensus.
>>         Now I would like to clarify why have we chosen imperative
>>         approach with DSL.
>>         You see a DSL as an alternative to HOT but it is not. DSL is
>>         alternative
>>         to Python-encoded resources in Heat (heat/engine/resources/*.py).
>>         Imagine how Heat would look like if you let untrusted users to
>>         upload
>>         Python plugins to Heat engine and load them on the fly. Heat
>>         resources
>>         are written in Python which is imperative language. So that
>>         MuranoPL for
>>         the same reason.
>>     We had this exact problem in Heat, and we decided to solve it
>>     with... HOT:
>>     http://lists.openstack.org/__pipermail/openstack-dev/2013-_
>> _April/007989.html
>>     <http://lists.openstack.org/pipermail/openstack-dev/2013-
>> April/007989.html>
>>     If I may be so bold as to quote myself:
>>     "...no cloud operator in the world - not even your friendly local IT
>>     department - is going to let users upload Python code to run
>>     in-memory in their orchestration engine along with all of the other
>>     users' code.
>>     "If only there were some sort of language for defining OpenStack
>>     services that could be safely executed by users...
>>     "Of course that's exactly what we're all about on this project :).
>>     So my proposal is to allow users to define their own resource types
>>     using a Heat template."
>> Excellent quotes! I can sign under each of them. No one would ever allow
>> to upload Python code so we haven't used Python (and it is not the only
>> reason not to use it)
> There's a fine line between a domain-specific programming language (as
> opposed to a domain-specific modelling language like HOT) and a
> general-purpose programming language. The reason no-one would run untrusted
> Python code is that Python is a programming language, not that it is
> general-purpose.
>  "If only there were some sort of language for defining OpenStack
>> services that could be safely executed by users..."  - we created such
>> language.
> And we found we already had one, that we successfully re-used for the
> purpose :)
>  It is exactly what you wanted to have then and now when it
>> exists you say it is useless. How this ironic.
> I'm not saying it's useless; I'm saying that it's *too* powerful for the
> job. It's clear that you guys have done amazing work and I don't want to
> criticise it, especially after having become aware of it only very late,
> but I also have serious reservations as to whether this is the right
> solution for OpenStack.
>  It may be conceitedly to
>> assume that MuranoPL is 100% is safe for all kind of attacks exists. It
>> is not. But I know how to make it, I know it is doable because it was
>> designed to be such. MuranoPL code is completely isolated from
>> underlying operating system and its resources (sockets, files etc.). It
>> is not translated to Python or any other language and thus cannot be a
>> subject to code injection attacks. It is 100% interpreted and absolutely
>> everything about the interpreter can be customized and controlled. You
>> can meter everything, have ACL for every single aspect, you can limit
>> MuranoPL execution time, number of executed instructions. Almost
>> anything is possible. With Python even if you make it somehow 100%
>> secure it would be hard to convince people it is. But there is no such
>> problem with YAML DSL :)
> This is all good news. I still worry that you have a very large footprint
> to secure.
>  The only thing I cannot agree with (but I hope you prove me wrong) is
>> that HOT is a solution for the problem. Surely you can do a lot of
>> things in HOT. But there is even a bigger list of things that you can't.
>> Otherwise why not Heat resources are written in Python and not in HOT?
> Because there have to be some base types that you can use as building
> blocks. In the case of Heat, those base types are the set of things that
> you can create by talking to OpenStack APIs authenticated with the user's
> token. In the case of Mistral, I would expect it to be the set of actions
> that you can take by talking to OpenStack APIs authenticated with the
> user's token. And in the case of Murano, I would expect it to be the union
> of those two.
>  Can you implement auto-scaling or load-balancer using HOT alone without
>> Python code?
> We can and do implement the load-balancer using a Heat template :)
> You could certainly implement autoscaling yourself, but you'd have to spin
> up a Compute server to run it on as part of the template. Which is sort of
> my point.
>  And what if application author (the guy that has absolutely
>> no control over the cloud where it going to be deployed. Even not aware
>> of its existence) needs his app to be autoscaled using Microsoft stack
>> technologies?
> I'm not sure what this means, and I feel like we're getting off-topic.
>  What if cloud operator wants  to have additional control
>> on scaling strategy so that it can be bound to his billing system?
> The operator can install whatever plugins they want.
>  What
>> if he wants that auto-scaling would be based on input from his existing
>> Nagios infrastructure rather then Ceilometer?
> This is supported already in autoscaling. Ceilometer just hits a URL for
> an alarm, but you don't have to configure it this way. Anything can hit the
> URL.
> And this is a good example for our general approach - we provide a way
> that works using built-in OpenStack services and a hook that allows you to
> customise it with your own service, running on your own machine (whether
> that be an actual machine or an OpenStack Compute server). What we *don't*
> do is provide a way to upload your own code that we then execute for you as
> some sort of secondary Compute service.
>  One can imagine any number
>> of use cases that just cannot be addressed by HOT without HOT becoming
>> another MuranoPL. So the only option you left with for such cases is
>> writing custom Python plugin. Now I don't want to reimplement in Python
>> the whole auto-scaling logic just because I need some customization. But
>> if I just patch existing Heat code then I would have to maintain my
>> custom Heat fork and have all sorts of problems when next version of
>> OpenStack arrives.There is a risk of introducing security breaches with
>> my new code that would be executed with root account permissions.
> I certainly hope *nobody* is running as root.
>  The
>> changes you would make to Heat would affect all tenants and may break
>> existing resources. And making application depend on presence of some
>> custom plugin makes it not portable across different clouds.
>> My point here is that HOT is not sufficient for coding new resource
>> types that are not superposition of existing resources. How can you do
>> that with a language that cannot even do the simplest arithmetics
>> operations?
> Everything is a combination of existing resources, because the set of
> existing resources is the set of things which the operator provides
> as-a-Service. The set of things that the operator provides as a service
> plus the set of things that you can implement yourself on your own server
> (virtual or not) covers the entire universe of things. What you appear to
> be suggesting is that OpenStack must provide *Everything*-as-a-Service by
> allowing users to write their own services and have the operator execute
> them as-a-Service. This would be a breathtakingly ambitious undertaking,
> and I don't mean that in a good way.
>      Which we did:
>>     https://blueprints.launchpad.__net/heat/+spec/provider-__resource
>>     <https://blueprints.launchpad.net/heat/+spec/provider-resource>
>>         We want application authors to be able to express application
>>         deployment
>>         and maintenance logic of any complexity. This may involve
>>         communication
>>         with 3rd party REST services (APIs of applications being deployed,
>>         external services like DNS server API, application licensing
>>         server API,
>>         billing systems, some hardware component APIs etc) and internal
>>         OpenStack services like Trove, Sahara, Marconi and others
>> including
>>         those that are not incubated yet and those to come in the
>>         future. You
>>         cannot have such things in HOT and when you required to you need
>> to
>>         develop custom resource in Python. Independence  on custom
>>         plugins is
>>         not good for Murano because they cannot be uploaded by end users
>> and
>>         thus he cannot write application definition that can be imported
>>         to/run
>>         on any cloud and need to convince cloud administrator to install
>> his
>>         Python plugin (something that is unimaginable in real life).
>>     Shouldn't Mistral be able to do all of those same things too?
>> Yes it should. In ideal world. Assembly language can do everything
>> Python can do but you do not expect people to use assembly unless they
>> have to. The same is true for Mistral - although it can possibly
>> improved to do all above but you need to be real genius to write
>> something like auto-scaling in Mistral DSL.
> I'm still confused by this autoscaling example. We can all agree that
> clearly you would be insane to write your own autoscaling implementation
> (whatever this means) in any language when your cloud provider already has
> a highly-customisable one like Heat. But if you were to write your own the
> way to do it would be in a programming language of your choice running on a
> server you control. I still don't see how this has anything to do with
> Mistral.
> You listed above a list of features that, I submit, are required for both
> Mistral and Murano. You are proposing to implement them twice, rather than
> once.
>  We are working closely with Mistral team and maybe once we would be able
>> to lay all MuranoPL tasks on Mistral shoulders and consume Mistral DSL
>> instead of our own, but this is not going to happen soon.
> When the TC said "Murano is slightly too far up the stack at this point to
> meet the "measured progression of openstack as a whole" requirement", IMO
> one of the major things they meant was that you're inventing your own
> workflow thing, leading to duplication of effort between this and Workflow
> as a Service. (And Mistral folks are in turn doing the same thing by not
> using the same workflow library, taskflow, as the rest of OpenStack.)
> Candidly, I'm surprised that this is not the #1 thing on your priority list
> because IMO it's the #1 thing that will delay getting the project incubated.
>  Mistral DSL is
>> good for another use cases and its developers may not like the idea of
>> having Turing-complete language in Mistral.
> They sound very wise ;)
>      Talking to existing OpenStack services doesn't seem hard - you could
>>     write plugins for those, or save time (and save the user learning
>>     another language) by using python-openstackclient and its syntax.
>>     For everything else you have a small number of generic operations -
>>     e.g. post to a Marconi queue (ReST API calls to untrusted services
>>     are problematic from a security perspective) - and allow the user to
>>     handle them in their own code in a language of their choice running
>>     either on their own machine or on a sandboxed, metered Compute
>>     server, rather than in a custom Turing-complete DSL running
>>     unmetered on the Murano server.
>> This is usually true. Usually, not alway. I can easily see how workflow
>> may need to talk to some cloud operator proprietary services that are
>> not accessible from VMs for security reasons. Workflow may need to
> I said for everything *else*. For things the operator specifically wants
> you to have access to (e.g. some proprietary service they run), they would
> have the option of installing a native command plugin to handle it. For
> everything *else* you drop a message in a Marconi queue and let the user
> handle it.
>  acquire some licenses before instantiating VMs. It may require to talk
> Where is the license server running? Can you stick a process on it to
> listen for Marconi messages and generate license requests to the actual
> server?
>  to cloud operator's billing system or quotas management system or
>> identity management system. There are many things that cannot be done
>> from VMs or need to be done before VM is spawned.
>> Calling external REST APIs is secure as long as you control the process
>> (can limit amount of traffic, request rate, timeouts, ACLs etc). This is
>> possible with MuranoPL.
>> And once again - everything that is done on server-side can be metered
>> and constrained.
> It's only secure in a very narrow sense. You're effectively allowing
> people to use your servers to make any request they like, and that's going
> to be highly prone to confused-deputy-type problems. The only completely
> safe URLs to fetch are ones that are in the Keystone catalog, passing the
> token obtained from the user. (We break this rule in Heat BTW, and we
> should stop. It's hard once you have users though, and even harder when we
> can't depend on Marconi yet.)
> cheers,
> Zane.
>          Because DSL is a way to write custom resources (in Heats
>>         terminology) it
>>         has to be Turing-complete and have all the characteristics of
>>         general-purpose language. It also has to have domain-specific
>>         features
>>         because we cannot expect that DSL users would be as skilled as
>> Heat
>>         developers and could write such resources without knowledge on
>>         hosting
>>         engine architecture and internals.
>>         HOT DSL is declarative because all the imperative stuff is
>> hardcoded
>>         into Heat engine. Thus all is left for HOT is to define "state
>>         of the
>>         world" - desired outcome. That is analogous to Object Model in
>>         Murano
>>         (see [1]). It is Object Model that can be compared to HOT, not
>>         DSL. As
>>         you can see it not more complex than HOT. Object Model is what
>>         end-user
>>         produces in Murano. And he event don't need to write it cause it
>>         can be
>>         composed in UI.
>>         Now because DSL provides not only a way to write sandboxed
>>         isolated code
>>         but also a lot of declarations (classes, properties, parameters,
>>         inheritance and contracts) that are mostly not present in Python
>> we
>>         don't need Parameters or Output sections in Object Model because
>>         all of
>>         this can be inferred from resource (classes) DSL declaration.
>>         Another
>>         consequence is that most of the things that can be written wrong
>>         in HOT
>>         can be verified on client side by validating classes' contracts
>>         without
>>         trying to deploy the stack and then go through error log
>> debugging.
>>     This is being worked on:
>>     https://blueprints.launchpad.__net/heat/+spec/param-__constraints
>>     <https://blueprints.launchpad.net/heat/+spec/param-constraints>
>> I agree that contracts can become part of HOT. Note that MuranoPL
>> contracts are orders of magnitude more powerful than parameter
>> constraints. This don't necessary mean that HOT needs that powerful
>> capabilities.  I'll be happy to collaborate on this in another
>> thread/place
>>         Because all resources' attributes types their constraints are
>>         known in
>>         advance (note that resource attribute may be a reference to
>> another
>>         resource with constraints on that reference like "I want any
>>         (regular,
>>         Galera etc) MySQL implementation") UI knows how to correctly
>>         compose the
>>         environment and can point out your mistakes at design time. This
>> is
>>         similar to how statically typed languages like C++/Java can do a
>>         lot of
>>         validation at compile time rather then in runtime as in Python.
>>         Personally I would love to see many of this features in HOT. What
>> is
>>         your vision on this? What of the mentioned above can be
>>         contributed to
>>         Heat? We definitely would like to integrate more with HOT and
>>         eliminate
>>         all duplications between projects. I think that Murano and Heat
>> are
>>         complimentary products that can effectively coexist. Murano
>> provides
>>         access to all HOT features and relies on Heat for most of its
>>         activities. I believe that we need to find an optimal way to
>>         integrate
>>         Heat, Murano, Mistral, Solum, Heater, TOSCA, do some integration
>>         between
>>         ex-Thermal and Murano Dashboard, be united regarding Glance
>>         usage for
>>         metadata and so on.
>>     +1
>>     To me that implies that Murano should be a relatively thin wrapper
>>     that ties together HOT and Mistral's DSL.
>>         We are okay with throwing MuranoPL out if the issues
>>         it solves would be addressed by HOT.
>>         If you have a vision on how HOT can address the same domain
>> MuranoPL
>>         does or any plans for such features in upcoming Heat releases I
>>         would
>>         ask you to share it.
>>         [1]
>>         https://wiki.openstack.org/__wiki/Murano/DSL/Blueprint#__
>> Object_model
>>         <https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#
>> Object_model>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Georgy Okrokvertskhov
OpenStack Platform Products,
Tel. +1 650 963 9828
Mob. +1 650 996 3284
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140320/a3254065/attachment.html>

More information about the OpenStack-dev mailing list