[openstack-dev] [Murano][Heat] MuranoPL questions?
zbitter at redhat.com
Fri Mar 21 00:28:48 UTC 2014
On 19/03/14 17:13, Stan Lagun wrote:
> On Wed, Mar 19, 2014 at 9:18 PM, Zane Bitter <zbitter at redhat.com
> <mailto:zbitter at redhat.com>> wrote:
> On 19/03/14 05:00, Stan Lagun wrote:
> Agree with your opinion on HOT expansion. I see that inclusion of
> imperative workflows and ALM would require major Heat redesign and
> probably would be impossible without loosing compatibility with
> HOT syntax. It would blur Heat mission, confuse current users
> and rise a
> lot of questions what should and what should not be in Heat.
> Thats why
> we chose to built a system on top of Heat rather then expending HOT.
> +1, I agree (as we have discussed before) that it would be a mistake
> to shoehorn workflow stuff into Heat. I do think we should implement
> the hooks I mentioned at the start of this thread to allow tighter
> integration between Heat and a workflow engine (i.e. Mistral).
> So building a system on top of Heat is good. Building it on top of
> Mistral as well would also be good, and that was part of the
> feedback from the TC.
> To me, building on top means building on top of the languages (which
> users will have to invest a lot of work in learning) as well, rather
> than having a completely different language and only using the
> underlying implementation(s).
> 1. Murano application developers (publishers) are using Heat directly.
> It is not the case that Murano executes some HOT templates under the
> hood but HOT templates are essential part of application definition.
> Developers still write HOT templates in HOT syntax
> 2. Please get me right here. It is not that we wanted to develop another
> language just for fun or to stand out from community. It is not that we
> wrote DSL and then started to think how we gonna use it and prove to
> others that is needed. We completely understand all concerns regarding
> new language. The only reason is that we had very concrete list of
> problems and use cases that we wanted to address in Murano. We did
> overwhelming obstacles with each of those approaches. It don't pretend
> that having own DSL is good. Just that not having it is much worse. I'm
> also not a HOT expert as you are and thus can be (partially) wrong about
> HOT and not aware of some of its power features. If so as a technical
> guys we would quickly come to consensus.
> Now I would like to clarify why have we chosen imperative
> approach with DSL.
> You see a DSL as an alternative to HOT but it is not. DSL is
> to Python-encoded resources in Heat (heat/engine/resources/*.py).
> Imagine how Heat would look like if you let untrusted users to
> Python plugins to Heat engine and load them on the fly. Heat
> are written in Python which is imperative language. So that
> MuranoPL for
> the same reason.
> We had this exact problem in Heat, and we decided to solve it
> with... HOT:
> If I may be so bold as to quote myself:
> "...no cloud operator in the world - not even your friendly local IT
> department - is going to let users upload Python code to run
> in-memory in their orchestration engine along with all of the other
> users' code.
> "If only there were some sort of language for defining OpenStack
> services that could be safely executed by users...
> "Of course that's exactly what we're all about on this project :).
> So my proposal is to allow users to define their own resource types
> using a Heat template."
> Excellent quotes! I can sign under each of them. No one would ever allow
> to upload Python code so we haven't used Python (and it is not the only
> reason not to use it)
There's a fine line between a domain-specific programming language (as
opposed to a domain-specific modelling language like HOT) and a
general-purpose programming language. The reason no-one would run
untrusted Python code is that Python is a programming language, not that
it is general-purpose.
> "If only there were some sort of language for defining OpenStack
> services that could be safely executed by users..." - we created such
And we found we already had one, that we successfully re-used for the
> It is exactly what you wanted to have then and now when it
> exists you say it is useless. How this ironic.
I'm not saying it's useless; I'm saying that it's *too* powerful for the
job. It's clear that you guys have done amazing work and I don't want to
criticise it, especially after having become aware of it only very late,
but I also have serious reservations as to whether this is the right
solution for OpenStack.
> It may be conceitedly to
> assume that MuranoPL is 100% is safe for all kind of attacks exists. It
> is not. But I know how to make it, I know it is doable because it was
> designed to be such. MuranoPL code is completely isolated from
> underlying operating system and its resources (sockets, files etc.). It
> is not translated to Python or any other language and thus cannot be a
> subject to code injection attacks. It is 100% interpreted and absolutely
> everything about the interpreter can be customized and controlled. You
> can meter everything, have ACL for every single aspect, you can limit
> MuranoPL execution time, number of executed instructions. Almost
> anything is possible. With Python even if you make it somehow 100%
> secure it would be hard to convince people it is. But there is no such
> problem with YAML DSL :)
This is all good news. I still worry that you have a very large
footprint to secure.
> The only thing I cannot agree with (but I hope you prove me wrong) is
> that HOT is a solution for the problem. Surely you can do a lot of
> things in HOT. But there is even a bigger list of things that you can't.
> Otherwise why not Heat resources are written in Python and not in HOT?
Because there have to be some base types that you can use as building
blocks. In the case of Heat, those base types are the set of things that
you can create by talking to OpenStack APIs authenticated with the
user's token. In the case of Mistral, I would expect it to be the set of
actions that you can take by talking to OpenStack APIs authenticated
with the user's token. And in the case of Murano, I would expect it to
be the union of those two.
> Can you implement auto-scaling or load-balancer using HOT alone without
> Python code?
We can and do implement the load-balancer using a Heat template :)
You could certainly implement autoscaling yourself, but you'd have to
spin up a Compute server to run it on as part of the template. Which is
sort of my point.
> And what if application author (the guy that has absolutely
> no control over the cloud where it going to be deployed. Even not aware
> of its existence) needs his app to be autoscaled using Microsoft stack
I'm not sure what this means, and I feel like we're getting off-topic.
> What if cloud operator wants to have additional control
> on scaling strategy so that it can be bound to his billing system?
The operator can install whatever plugins they want.
> if he wants that auto-scaling would be based on input from his existing
> Nagios infrastructure rather then Ceilometer?
This is supported already in autoscaling. Ceilometer just hits a URL for
an alarm, but you don't have to configure it this way. Anything can hit
And this is a good example for our general approach - we provide a way
that works using built-in OpenStack services and a hook that allows you
to customise it with your own service, running on your own machine
(whether that be an actual machine or an OpenStack Compute server). What
we *don't* do is provide a way to upload your own code that we then
execute for you as some sort of secondary Compute service.
> One can imagine any number
> of use cases that just cannot be addressed by HOT without HOT becoming
> another MuranoPL. So the only option you left with for such cases is
> writing custom Python plugin. Now I don't want to reimplement in Python
> the whole auto-scaling logic just because I need some customization. But
> if I just patch existing Heat code then I would have to maintain my
> custom Heat fork and have all sorts of problems when next version of
> OpenStack arrives.There is a risk of introducing security breaches with
> my new code that would be executed with root account permissions.
I certainly hope *nobody* is running as root.
> changes you would make to Heat would affect all tenants and may break
> existing resources. And making application depend on presence of some
> custom plugin makes it not portable across different clouds.
> My point here is that HOT is not sufficient for coding new resource
> types that are not superposition of existing resources. How can you do
> that with a language that cannot even do the simplest arithmetics
Everything is a combination of existing resources, because the set of
existing resources is the set of things which the operator provides
as-a-Service. The set of things that the operator provides as a service
plus the set of things that you can implement yourself on your own
server (virtual or not) covers the entire universe of things. What you
appear to be suggesting is that OpenStack must provide
*Everything*-as-a-Service by allowing users to write their own services
and have the operator execute them as-a-Service. This would be a
breathtakingly ambitious undertaking, and I don't mean that in a good way.
> Which we did:
> We want application authors to be able to express application
> and maintenance logic of any complexity. This may involve
> with 3rd party REST services (APIs of applications being deployed,
> external services like DNS server API, application licensing
> server API,
> billing systems, some hardware component APIs etc) and internal
> OpenStack services like Trove, Sahara, Marconi and others including
> those that are not incubated yet and those to come in the
> future. You
> cannot have such things in HOT and when you required to you need to
> develop custom resource in Python. Independence on custom
> plugins is
> not good for Murano because they cannot be uploaded by end users and
> thus he cannot write application definition that can be imported
> on any cloud and need to convince cloud administrator to install his
> Python plugin (something that is unimaginable in real life).
> Shouldn't Mistral be able to do all of those same things too?
> Yes it should. In ideal world. Assembly language can do everything
> Python can do but you do not expect people to use assembly unless they
> have to. The same is true for Mistral - although it can possibly
> improved to do all above but you need to be real genius to write
> something like auto-scaling in Mistral DSL.
I'm still confused by this autoscaling example. We can all agree that
clearly you would be insane to write your own autoscaling implementation
(whatever this means) in any language when your cloud provider already
has a highly-customisable one like Heat. But if you were to write your
own the way to do it would be in a programming language of your choice
running on a server you control. I still don't see how this has anything
to do with Mistral.
You listed above a list of features that, I submit, are required for
both Mistral and Murano. You are proposing to implement them twice,
rather than once.
> We are working closely with Mistral team and maybe once we would be able
> to lay all MuranoPL tasks on Mistral shoulders and consume Mistral DSL
> instead of our own, but this is not going to happen soon.
When the TC said "Murano is slightly too far up the stack at this point
to meet the "measured progression of openstack as a whole" requirement",
IMO one of the major things they meant was that you're inventing your
own workflow thing, leading to duplication of effort between this and
Workflow as a Service. (And Mistral folks are in turn doing the same
thing by not using the same workflow library, taskflow, as the rest of
OpenStack.) Candidly, I'm surprised that this is not the #1 thing on
your priority list because IMO it's the #1 thing that will delay getting
the project incubated.
> Mistral DSL is
> good for another use cases and its developers may not like the idea of
> having Turing-complete language in Mistral.
They sound very wise ;)
> Talking to existing OpenStack services doesn't seem hard - you could
> write plugins for those, or save time (and save the user learning
> another language) by using python-openstackclient and its syntax.
> For everything else you have a small number of generic operations -
> e.g. post to a Marconi queue (ReST API calls to untrusted services
> are problematic from a security perspective) - and allow the user to
> handle them in their own code in a language of their choice running
> either on their own machine or on a sandboxed, metered Compute
> server, rather than in a custom Turing-complete DSL running
> unmetered on the Murano server.
> This is usually true. Usually, not alway. I can easily see how workflow
> may need to talk to some cloud operator proprietary services that are
> not accessible from VMs for security reasons. Workflow may need to
I said for everything *else*. For things the operator specifically wants
you to have access to (e.g. some proprietary service they run), they
would have the option of installing a native command plugin to handle
it. For everything *else* you drop a message in a Marconi queue and let
the user handle it.
> acquire some licenses before instantiating VMs. It may require to talk
Where is the license server running? Can you stick a process on it to
listen for Marconi messages and generate license requests to the actual
> to cloud operator's billing system or quotas management system or
> identity management system. There are many things that cannot be done
> from VMs or need to be done before VM is spawned.
> Calling external REST APIs is secure as long as you control the process
> (can limit amount of traffic, request rate, timeouts, ACLs etc). This is
> possible with MuranoPL.
> And once again - everything that is done on server-side can be metered
> and constrained.
It's only secure in a very narrow sense. You're effectively allowing
people to use your servers to make any request they like, and that's
going to be highly prone to confused-deputy-type problems. The only
completely safe URLs to fetch are ones that are in the Keystone catalog,
passing the token obtained from the user. (We break this rule in Heat
BTW, and we should stop. It's hard once you have users though, and even
harder when we can't depend on Marconi yet.)
> Because DSL is a way to write custom resources (in Heats
> terminology) it
> has to be Turing-complete and have all the characteristics of
> general-purpose language. It also has to have domain-specific
> because we cannot expect that DSL users would be as skilled as Heat
> developers and could write such resources without knowledge on
> engine architecture and internals.
> HOT DSL is declarative because all the imperative stuff is hardcoded
> into Heat engine. Thus all is left for HOT is to define "state
> of the
> world" - desired outcome. That is analogous to Object Model in
> (see ). It is Object Model that can be compared to HOT, not
> DSL. As
> you can see it not more complex than HOT. Object Model is what
> produces in Murano. And he event don't need to write it cause it
> can be
> composed in UI.
> Now because DSL provides not only a way to write sandboxed
> isolated code
> but also a lot of declarations (classes, properties, parameters,
> inheritance and contracts) that are mostly not present in Python we
> don't need Parameters or Output sections in Object Model because
> all of
> this can be inferred from resource (classes) DSL declaration.
> consequence is that most of the things that can be written wrong
> in HOT
> can be verified on client side by validating classes' contracts
> trying to deploy the stack and then go through error log debugging.
> This is being worked on:
> I agree that contracts can become part of HOT. Note that MuranoPL
> contracts are orders of magnitude more powerful than parameter
> constraints. This don't necessary mean that HOT needs that powerful
> capabilities. I'll be happy to collaborate on this in another thread/place
> Because all resources' attributes types their constraints are
> known in
> advance (note that resource attribute may be a reference to another
> resource with constraints on that reference like "I want any
> Galera etc) MySQL implementation") UI knows how to correctly
> compose the
> environment and can point out your mistakes at design time. This is
> similar to how statically typed languages like C++/Java can do a
> lot of
> validation at compile time rather then in runtime as in Python.
> Personally I would love to see many of this features in HOT. What is
> your vision on this? What of the mentioned above can be
> contributed to
> Heat? We definitely would like to integrate more with HOT and
> all duplications between projects. I think that Murano and Heat are
> complimentary products that can effectively coexist. Murano provides
> access to all HOT features and relies on Heat for most of its
> activities. I believe that we need to find an optimal way to
> Heat, Murano, Mistral, Solum, Heater, TOSCA, do some integration
> ex-Thermal and Murano Dashboard, be united regarding Glance
> usage for
> metadata and so on.
> To me that implies that Murano should be a relatively thin wrapper
> that ties together HOT and Mistral's DSL.
> We are okay with throwing MuranoPL out if the issues
> it solves would be addressed by HOT.
> If you have a vision on how HOT can address the same domain MuranoPL
> does or any plans for such features in upcoming Heat releases I
> ask you to share it.
More information about the OpenStack-dev