Open Stack

Hi Stan,

Comments inline.

> Zane,
> 
> I appreciate your explanations on Heat/HOT. This really makes sense.
> I didn't mean to say that MuranoPL is better for Heat. Actually HOT is good
> for Heat's mission. I completely acknowledge it.
> I've tried to avoid comparison between languages and I'm sorry if it felt
> that way. This is not productive as I don't offer you to replace HOT with
> MuranoPL (although I believe that certain elements of MuranoPL syntax can be
> contributed to HOT and be valuable addition there). Also people tend to
> protect what they have developed and invested into and to be fair this is
> what we did in this thread to great extent.
> 
> What I'm trying to achieve is that you and the rest of Heat team understand
> why it was designed the way it is. I don't feel that Murano can become
> full-fledged member of OpenStack ecosystem without a bless from Heat team.
> And it would be even better if we agree on certain design, join our efforts
> and contribute to each other for sake of Orchestration program.

Note that I feel that most people outside of the Murano project are against the idea of using a DSL. You should feel that it could block the integration in OpenStack.

> I'm sorry for long mail texts written in not-so-good English and appreciate
> you patience reading and answering them.
> 
> Having said that let me step backward and explain our design decisions.
> 
> Cloud administrators are usually technical guys that are capable of learning
> HOT and writing YAML templates. They know exact configuration of their cloud
> (what services are available, what is the version of OpenStack cloud is
> running) and generally understands how OpenStack works. They also know about
> software they intent to install. If such guy wants to install Drupal he
> knows exactly that he needs HOT template describing Fedora VM with Apache +
> PHP + MySQL + Drupal itself. It is not a problem for him to write such HOT
> template.
> 
> Note that such template would be designed for very particular configuration.
> There are hundreds of combinations that may be used to install that Drupal -
> use RHEL/Windows/etc instead of Fedora, use ngnix/IIS/etc instead of Apache,
> use FastCGI instead of mod_php, PostgreSQL instead of MySQL. You may choose
> to have all software on single VM or have one VM for database and another
> for Drupal. There are also constraints to those combinations. For example
> you cannot have Fedora + IIS on the same VM. You cannot have Apache and
> Drupal on different VMs.
> 
> So the HOT template represent fixed combination of those software components.
> HOT may have input parameters like "username" or "dbImageName" but the
> overall structure of template is fixed. You cannot have template that choses
> whether to use Windows or Linux based on parameter value. You cannot write
> HOT that accepts number of instances it allowed to create and then decide
> what would be installed on each of them. This is just not needed for Heat
> users.
> 
> With Murano the picture looks the opposite. Typical Murano user is a guy who
> bought account from cloud hosting vendor (cloud operator) and want to run
> some software in the cloud. He may not even be aware that it is OpenStack.
> He knows nothing about programming in general and Heat in particular. He
> doesn't want to write YAMLs. He may not know how exactly Drupal is installed
> and what components it consists of.

So that's where I want to make a first stop. If your primary user is not a developer, there is no reason to introduce a DSL for security reasons. The provider can trust the code he writes, and there is no need to create a dedicated language.

> So what he does is he goes to his cloud (Murano) dashboard, browses through
> application catalog, finds Drupal and drags it onto his environment board
> (think like Visio-style designer). He can stop at this point, click "deploy"
> button and the system will deploy Drupal. In another words the system (or
> maybe better to say cloud operator or application developer) decides what
> set of components is going to be installed (like 1 Fedora VM for MySQL and 1
> CentOS VM for Apache-PHP-Drupal). But user may decide he wants to customize
> his environment. He digs down and sees that Drupal requires database
> instance and the default is MySQL. He clicks on a button to see what are
> other options available for that role.
> 
> In Heat HOT developer is the user. But in Murano those are completely
> different roles. There are developers that write application definitions
> (that is DSL code) and there are end users who compose environments from
> those applications (components). Application developers may have nothing to
> do with particular cloud their application deployed on. As for Drupal
> application the developer knows that Drupal can be run with MySQL or
> PostgreSQL. But there may be many compatible implementations of those DBMSes
> - Galera MySQL, TroveMySQL, MMM MySQL etc. So to get a list of what
> components can be placed in a database role Murano needs to look at all
> applications in Application Catalog and find which of them are compatible
> with MySQL and PostgreSQL so that user could choose what implementation is
> better suits his needs (trade performance for high-availability etc.).
> 
> User can go deeper and to decide that he wants that MySQL instance (this can
> be 1 or more VMs depending on implementation) to be shared between Drupal
> and another application in that environment (say WordPress). He can go even
> deeper to VM level and decide that he wants to have WordPress, Drupal,
> Apcache and slave MySQL node on one VM and MySQL master node on another.
> 
> The ultimate goal is to create sort of construction kit for applications.
> Cloud LEGO. And let users build environments of any complexity from small
> components while providing reasonable default so that one-click deployment
> would also be possible. And such system to be useful the design process need
> to be guided and driven by UI. System must know what combination of
> components are possible and what are not and not let user create Microsoft
> IIS hosted on Fedora.

Those goals are really nice and I'd like to have something like that in OpenStack.

> This leads us to following design decisions:
> 
> a. We need a DSL that would allow describing those small components
> independently in such a way that the environment could be composed from
> components written by different people that are not aware of each other

You somewhat start by the end here. You haven't convinced us that you need a DSL.

> b. Components need to have properties (parameters in Heat terms). Properties
> may be of different types (strings, numbers, booleans etc.). There can be
> constraints on property values ("not null", "length > 5").

We have those constraints in Heat, they don't require imperative constructs.

> c. Components may reference (require, make use of) other components (Drupal
> uses MySQL, hosted on Apache etc). There can be constraints for those
> references.

Present in HOTs.

> d. Constraints for both properties and references can be *very* complex and
> indirect. Like "IIS can be deployed on any VM who's image is known to be
> Windows with version >= WinServer2003". Or "my app requires 10 to 20 Windows
> instances that are all member of the same ActiveDirectory domain". Or "I
> have 2 properties: minInstanceCount and maxInstanceCount and constraint
> maxInstaceCount >= minInstanceCount". So we need a language to express
> conditions of any complexity.

We support pluggable constraints in Heat, that you can write in Python.

> e. Both properties and references must be declared in a way that would make
> possible for UI dashboard to discover their existence, ask user appropriate
> input and validate the constraints on the fly.

Possible as well.

> f. Properties and especially references can have complex structure (lists,
> maps, combination of them) rather than plain scalars. For example
> ActiveDirectory consists of (e.g. references) *list* of domain controllers.
> Constraints must respect that and be capable of validating things like
> len(list) >= 2, all elements in list have some predicate true, there is at
> least one element with desired set of properties etc.

If not possible, HOT should be extended to support this.

> g. Language need to have some sort mechanism to express polymorphism. E,g,
> GaleraMySQL *is* MySQL (and thus can be used anywhere where "just MySQL" can
> be). This also implies ability to have declare interfaces (class contracts)
> and ability to implement those interface to that it would be possible that
> two components are conform to the same interface.

You don't need to have polymorphism for this. I'm mostly familiar with the mechanism used by Juju, where you *declare* an interface. It's basically metadata, and it's much simpler.

> h. If we want components to be reusable and easy to write we need to have
> ability to one component to inherit/extend another. Inheritance implies
> ability to override and customize some methods
> 
> i. We don't want to do deployment ourself. We don't intend to replace Heat,
> Puppet etc. So we need a way to construct Heat templates that will do the
> job. We need to map component's properties to
> HOT parameters. This requires a language to express those bindings
> 
> j. Because there can be lists of references (or properties) it should be
> possible to express something like
> for(domain_controller in self.controllers):
> hot_template.insert(resources_for_particular_controller)
> 
> or
> 
> if self.deploy_on_windows:
> generate_windows_hot_template()
> else:
> generate_linux_hot_template()
> 
> or
> 
> for i in range(self.instance_count):
> hot_template.insert(generate_instance())

CloudFormation recently added support for conditionals in templates. I'm not a super fan of it, but we could presumably have those in HOT as well.

> k. Need to provide most simple method to compose stack from pre-made HOT
> snippets and avoid dynamic generation when its not really needed
> 
> l. Need to have some standard format to represent view of the world that is
> what was designed in dashboard. In Murano it is JSON that describes object
> graph with object property values and references to another objects. This is
> somewhat similar to simple HOT template but with hierarchical structure
> 
> I think at this point it becomes obvious that above describes object-oriented
> Turing-complete language. And the most important property of that language
> is that *everything* is declared in a way useful to be accessed from
> dashboard/app catalog. You not often see properties/references/constraints
> declared in dynamic language. I've answered several times already why this
> language cannot be JavaScript or Lua. And this is just on the surface. There
> are many things under the hood that makes MuranoPL stand out from all
> embeddable languages I know and thats for a good reason.
> 
> And I didn't even mentioned ALM aspects, orchestration, and many many other
> things that go far beyond what can be expressed by HOT-style DSL

Well orchestration should be Heat's job.

What I can say is that I'm not convinced. The only use-case for a DSL would be if you have to upload user-written code, but what you mentioned is a Web interface, where the user doesn't use the DSL, and the cloud provider is the developer. There is no reason in this case to have a secure environment for the code.

Cheers,

-- 
Thomas