Open Stack

Hi Thomas,

I think we went to the second loop of the discussion about generic language
concepts.  Murano does not use a new language for the sole purpose of
having parameters, constraints and polymorphism. These are generic concepts
which are common for different languages, so keeping arguing about these
generic concepts is just like a holy war  like Python vs. C. Keeping these
arguments is just like to say that we don't need Python as functions and
parameters already exists in C which is used under the hood in Python.

Yes Murano DSL have some generic concepts similar to HOT. I think this is a
benefit as user will see the familiar syntax constructions and it will be a
lower threshold for him to start using Murano DSL.

In a simplified view Murano uses DSL for application definition to solve
several particular problems:
a) control UI rendering of Application Catalog
b) control HOT template generation

These aspects are not covered in HOT and probably should not be covered. I
don't like an idea of expressing HOT template generation in HOT as it
sounds like a creation another Lisp like language :-)

I don't think that your statement that most of the people in the community
are against new DSL is a right summary. There are some disagreements how it
should look like and what are the goals. You will be probably surprised but
we are not the first who use DSL for HOT templates generation. Here is an
e-mail thread right about Ruby based DSL used in IBM for the same purpose:
http://lists.openstack.org/pipermail/openstack-dev/2014-February/026606.html

The term "Orchestration" is quite generic. Saying that orchestration should
be Heat job sounds like a well know Henry Ford's phrase "You can have any
colour as long as it's black.".
I think this is again a lack of understanding of the difference between
Orchestration program and Heat project. There are many aspects of
Orchestration and OpenStack has the Orchestration program for the projects
which are focused on some aspects of orchestration. Heat is one of the
project inside Orchestration program but it does not mean that Heat should
cover everything. That is why we discussed in this thread how workflows
aspects should be aligned and how they should be placed into this
Orchestration program.

Thanks
Georgy


On Mon, Mar 24, 2014 at 8:28 AM, Dmitry <meytin at gmail.com> wrote:

> MuranoPL supposed to provide a solution for the real needs to manage
> services in the centralized manner and to allow cloud provider customers to
> create their own services.
> The application catalog similar to AppDirect (www.appdirect.com) natively
> supported by OpenStack is a huge step forward.
> Think about Amazon which provides different services for the different
> needs: Amazon Cloud Formation, Amazon OpsWorks and Amazon Beanstalk.
> Following the similar logic (which is fully makes sense for me), OpenStack
> should provide resource reservation and orchestration (Heat and Climate),
> Application Catalog (Murano) and PaaS (Solum).
> Every project can live in harmony with other and contribute for the cloud
> service provider service completeness.
> This is my opinion and i would happy to use Murano in our internal
> solution.
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Georgy Okrokvertskhov
Architect,
OpenStack Platform Products,
Mirantis
http://www.mirantis.com
Tel. +1 650 963 9828
Mob. +1 650 996 3284


On Mon, Mar 24, 2014 at 5:13 AM, Thomas Herve <thomas.herve at enovance.com>wrote:

> Hi Stan,
>
> Comments inline.
>
> > Zane,
> >
> > I appreciate your explanations on Heat/HOT. This really makes sense.
> > I didn't mean to say that MuranoPL is better for Heat. Actually HOT is
> good
> > for Heat's mission. I completely acknowledge it.
> > I've tried to avoid comparison between languages and I'm sorry if it felt
> > that way. This is not productive as I don't offer you to replace HOT with
> > MuranoPL (although I believe that certain elements of MuranoPL syntax
> can be
> > contributed to HOT and be valuable addition there). Also people tend to
> > protect what they have developed and invested into and to be fair this is
> > what we did in this thread to great extent.
> >
> > What I'm trying to achieve is that you and the rest of Heat team
> understand
> > why it was designed the way it is. I don't feel that Murano can become
> > full-fledged member of OpenStack ecosystem without a bless from Heat
> team.
> > And it would be even better if we agree on certain design, join our
> efforts
> > and contribute to each other for sake of Orchestration program.
>
> Note that I feel that most people outside of the Murano project are
> against the idea of using a DSL. You should feel that it could block the
> integration in OpenStack.
>
> > I'm sorry for long mail texts written in not-so-good English and
> appreciate
> > you patience reading and answering them.
> >
> > Having said that let me step backward and explain our design decisions.
> >
> > Cloud administrators are usually technical guys that are capable of
> learning
> > HOT and writing YAML templates. They know exact configuration of their
> cloud
> > (what services are available, what is the version of OpenStack cloud is
> > running) and generally understands how OpenStack works. They also know
> about
> > software they intent to install. If such guy wants to install Drupal he
> > knows exactly that he needs HOT template describing Fedora VM with
> Apache +
> > PHP + MySQL + Drupal itself. It is not a problem for him to write such
> HOT
> > template.
> >
> > Note that such template would be designed for very particular
> configuration.
> > There are hundreds of combinations that may be used to install that
> Drupal -
> > use RHEL/Windows/etc instead of Fedora, use ngnix/IIS/etc instead of
> Apache,
> > use FastCGI instead of mod_php, PostgreSQL instead of MySQL. You may
> choose
> > to have all software on single VM or have one VM for database and another
> > for Drupal. There are also constraints to those combinations. For example
> > you cannot have Fedora + IIS on the same VM. You cannot have Apache and
> > Drupal on different VMs.
> >
> > So the HOT template represent fixed combination of those software
> components.
> > HOT may have input parameters like "username" or "dbImageName" but the
> > overall structure of template is fixed. You cannot have template that
> choses
> > whether to use Windows or Linux based on parameter value. You cannot
> write
> > HOT that accepts number of instances it allowed to create and then decide
> > what would be installed on each of them. This is just not needed for Heat
> > users.
> >
> > With Murano the picture looks the opposite. Typical Murano user is a guy
> who
> > bought account from cloud hosting vendor (cloud operator) and want to run
> > some software in the cloud. He may not even be aware that it is
> OpenStack.
> > He knows nothing about programming in general and Heat in particular. He
> > doesn't want to write YAMLs. He may not know how exactly Drupal is
> installed
> > and what components it consists of.
>
> So that's where I want to make a first stop. If your primary user is not a
> developer, there is no reason to introduce a DSL for security reasons. The
> provider can trust the code he writes, and there is no need to create a
> dedicated language.
>
> > So what he does is he goes to his cloud (Murano) dashboard, browses
> through
> > application catalog, finds Drupal and drags it onto his environment board
> > (think like Visio-style designer). He can stop at this point, click
> "deploy"
> > button and the system will deploy Drupal. In another words the system (or
> > maybe better to say cloud operator or application developer) decides what
> > set of components is going to be installed (like 1 Fedora VM for MySQL
> and 1
> > CentOS VM for Apache-PHP-Drupal). But user may decide he wants to
> customize
> > his environment. He digs down and sees that Drupal requires database
> > instance and the default is MySQL. He clicks on a button to see what are
> > other options available for that role.
> >
> > In Heat HOT developer is the user. But in Murano those are completely
> > different roles. There are developers that write application definitions
> > (that is DSL code) and there are end users who compose environments from
> > those applications (components). Application developers may have nothing
> to
> > do with particular cloud their application deployed on. As for Drupal
> > application the developer knows that Drupal can be run with MySQL or
> > PostgreSQL. But there may be many compatible implementations of those
> DBMSes
> > - Galera MySQL, TroveMySQL, MMM MySQL etc. So to get a list of what
> > components can be placed in a database role Murano needs to look at all
> > applications in Application Catalog and find which of them are compatible
> > with MySQL and PostgreSQL so that user could choose what implementation
> is
> > better suits his needs (trade performance for high-availability etc.).
> >
> > User can go deeper and to decide that he wants that MySQL instance (this
> can
> > be 1 or more VMs depending on implementation) to be shared between Drupal
> > and another application in that environment (say WordPress). He can go
> even
> > deeper to VM level and decide that he wants to have WordPress, Drupal,
> > Apcache and slave MySQL node on one VM and MySQL master node on another.
> >
> > The ultimate goal is to create sort of construction kit for applications.
> > Cloud LEGO. And let users build environments of any complexity from small
> > components while providing reasonable default so that one-click
> deployment
> > would also be possible. And such system to be useful the design process
> need
> > to be guided and driven by UI. System must know what combination of
> > components are possible and what are not and not let user create
> Microsoft
> > IIS hosted on Fedora.
>
> Those goals are really nice and I'd like to have something like that in
> OpenStack.
>
> > This leads us to following design decisions:
> >
> > a. We need a DSL that would allow describing those small components
> > independently in such a way that the environment could be composed from
> > components written by different people that are not aware of each other
>
> You somewhat start by the end here. You haven't convinced us that you need
> a DSL.
>
> > b. Components need to have properties (parameters in Heat terms).
> Properties
> > may be of different types (strings, numbers, booleans etc.). There can be
> > constraints on property values ("not null", "length > 5").
>
> We have those constraints in Heat, they don't require imperative
> constructs.
>
> > c. Components may reference (require, make use of) other components
> (Drupal
> > uses MySQL, hosted on Apache etc). There can be constraints for those
> > references.
>
> Present in HOTs.
>
> > d. Constraints for both properties and references can be *very* complex
> and
> > indirect. Like "IIS can be deployed on any VM who's image is known to be
> > Windows with version >= WinServer2003". Or "my app requires 10 to 20
> Windows
> > instances that are all member of the same ActiveDirectory domain". Or "I
> > have 2 properties: minInstanceCount and maxInstanceCount and constraint
> > maxInstaceCount >= minInstanceCount". So we need a language to express
> > conditions of any complexity.
>
> We support pluggable constraints in Heat, that you can write in Python.
>
> > e. Both properties and references must be declared in a way that would
> make
> > possible for UI dashboard to discover their existence, ask user
> appropriate
> > input and validate the constraints on the fly.
>
> Possible as well.
>
> > f. Properties and especially references can have complex structure
> (lists,
> > maps, combination of them) rather than plain scalars. For example
> > ActiveDirectory consists of (e.g. references) *list* of domain
> controllers.
> > Constraints must respect that and be capable of validating things like
> > len(list) >= 2, all elements in list have some predicate true, there is
> at
> > least one element with desired set of properties etc.
>
> If not possible, HOT should be extended to support this.
>
> > g. Language need to have some sort mechanism to express polymorphism.
> E,g,
> > GaleraMySQL *is* MySQL (and thus can be used anywhere where "just MySQL"
> can
> > be). This also implies ability to have declare interfaces (class
> contracts)
> > and ability to implement those interface to that it would be possible
> that
> > two components are conform to the same interface.
>
> You don't need to have polymorphism for this. I'm mostly familiar with the
> mechanism used by Juju, where you *declare* an interface. It's basically
> metadata, and it's much simpler.
>
> > h. If we want components to be reusable and easy to write we need to have
> > ability to one component to inherit/extend another. Inheritance implies
> > ability to override and customize some methods
> >
> > i. We don't want to do deployment ourself. We don't intend to replace
> Heat,
> > Puppet etc. So we need a way to construct Heat templates that will do the
> > job. We need to map component's properties to
> > HOT parameters. This requires a language to express those bindings
> >
> > j. Because there can be lists of references (or properties) it should be
> > possible to express something like
> > for(domain_controller in self.controllers):
> > hot_template.insert(resources_for_particular_controller)
> >
> > or
> >
> > if self.deploy_on_windows:
> > generate_windows_hot_template()
> > else:
> > generate_linux_hot_template()
> >
> > or
> >
> > for i in range(self.instance_count):
> > hot_template.insert(generate_instance())
>
> CloudFormation recently added support for conditionals in templates. I'm
> not a super fan of it, but we could presumably have those in HOT as well.
>
> > k. Need to provide most simple method to compose stack from pre-made HOT
> > snippets and avoid dynamic generation when its not really needed
> >
> > l. Need to have some standard format to represent view of the world that
> is
> > what was designed in dashboard. In Murano it is JSON that describes
> object
> > graph with object property values and references to another objects.
> This is
> > somewhat similar to simple HOT template but with hierarchical structure
> >
> > I think at this point it becomes obvious that above describes
> object-oriented
> > Turing-complete language. And the most important property of that
> language
> > is that *everything* is declared in a way useful to be accessed from
> > dashboard/app catalog. You not often see
> properties/references/constraints
> > declared in dynamic language. I've answered several times already why
> this
> > language cannot be JavaScript or Lua. And this is just on the surface.
> There
> > are many things under the hood that makes MuranoPL stand out from all
> > embeddable languages I know and thats for a good reason.
> >
> > And I didn't even mentioned ALM aspects, orchestration, and many many
> other
> > things that go far beyond what can be expressed by HOT-style DSL
>
> Well orchestration should be Heat's job.
>
> What I can say is that I'm not convinced. The only use-case for a DSL
> would be if you have to upload user-written code, but what you mentioned is
> a Web interface, where the user doesn't use the DSL, and the cloud provider
> is the developer. There is no reason in this case to have a secure
> environment for the code.
>
> Cheers,
>
> --
> Thomas
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Georgy Okrokvertskhov
Architect,
OpenStack Platform Products,
Mirantis
http://www.mirantis.com
Tel. +1 650 963 9828
Mob. +1 650 996 3284
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140325/7acf18eb/attachment.html>