Open Stack

Hello all,

This has been a great discussion that I would like to chime in on. I am an architect at HP where I just spent two years on a continuous delivery enterprise software product before moving full time to an OpenStack based project 4 months ago.  In the last few months, amongst other things, I have spent a portion of my time working with TOSCA and participating on an interoperability committee. I've also been keeping an eye on CAMP and its concepts.

We are currently trying to better understand how we should be using and viewing Heat's role in the overall ecosystem of application + infrastructure orchestration in a managed enterprise environment. Based on these threads and even Zane's comment, there isn't a real agreement on the level and role that Heat plays, and it seems that this is an ideal time to have this discussion, so +1 for initiating the discussion!

Zane Bitner wrote:
> FWIW it appears to me that CAMP is targeted at the PaaS level,
> and while I'm supportive of that effort it doesn't seem to me to 
> be in scope for Heat, which I see as limited to the IaaS layer. 
> I'm willing to be re-educated though ;)

Perhaps Heat was intended to be at the IaaS level today (?), but there are already platform and application examples in the test templates today (e.g. wordpress). I know from experience that infrastructure has complexity and that things become even more complex when we start considering how applications use the infrastructure. 

>From my perspective, there are a few fundamental categories of management to consider in the orchestration of applications and infrastructure.

+ Physical Resource Management  (Machine, Network, Storage, Load Balancing, etc.)
+ Configuration Management (Setup users, install software, configure software, deploy code)
+ Performance Management (Availability, Scalability, Placement)
+ Security Management (Data, Connectivity, Placement)

To accomplish all of the above, we have a point of view that cloud orchestration at the enterprise-level cannot be solved with a single engine or tool and that different enterprises will want to leverage their existing investments in their tools. This is why Chef may be used for configuration management (and even infrastructure resource management), but it doesn't make sense to use Chef for monitoring or making policy decisions about when and where to instantiate resources.

If I were to categorize Heat *today* from an external perspective, I might call it an API orchestrator expressed in a mixed declarative / imperative template script language which provides a way to create, update, and delete multiple resources in a coordinated fashion by calling their APIs through resource plugins (including core resources).  The engine doesn't care about what the resource is, it doesn't care about why it is being called, all it knows is that it has been asked to orchestrate a physical resource stack based on an input template. 

If I am correct in my categorization of Heat (open for comments!), what this implies is that Heat provides a tightly scoped service for physical resource API orchestration. It doesn't attempt to intermingle knowledge of the resources with the engine, which would distract from its core capability. I like that in theory, it can treat an infrastructure resource (VM) the same as a configuration resource (Chef cookbook) - a set of inputs that will be passed to a plugin at the appropriate time based upon dependency graphing and let the plugin handle it. It doesn't even care if a resource plugin recursively invokes other Heat resources. It is this intended ignorance of what it is doing that makes it work as an API orchestrator.

All of which leads me back to the core questions surfacing that I am very interested in:
 + Does Heat support alternate ways to express the resource orchestration?
 + Does Heat play a greater role in orchestration other than as pre-scripted resource orchestration?

And the implied questions are:
+ Should Heat support a logical way to model stack layers independently and without physical resource binding?
+ Does this start breaking down into sub services like Nova did (some of which have now been separated)?

I believe that as software projects go from small to large, that the modeling and scripting needs change and often become more complex.  Some people will care about cloud portability and some won't. People in different roles will desire fine grained control and others will not. In other words, there is a need for both logical / declarative modeling as well as physical / imperative modeling and they are complementary. I also believe that there can be course grained declarative modeling as well as fine grained declarative modeling.

So, ultimately, the question for Heat is how much, how far, and how soon is it going to go in all of these areas?

Thanks!
-Travis

-----Original Message-----
From: Zane Bitter [mailto:zbitter at redhat.com] 
Sent: Wednesday, April 10, 2013 3:45 AM
To: openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [Heat] TOSCA, CAMP, CloudFormation, ???

On 10/04/13 10:18, Thomas Spatzier wrote:
> Hi Clint,
>
> you are raising a very valid point here. I agree that just adding 
> support for all of the format brought forward could be problematic, 
> and I have a few thoughts on that.
>
> My experience is that there are many pattern engines (let me just use 
> that term here) out there - some done as open source, some in various 
> products of different vendors. I have seen very similar concepts in 
> all of those engines, but each has its own proprietary format, e.g. we 
> have an internal format in our product, partners in the OASIS TOSCA TC 
> have their internal format in their engines and so on. So what we did 
> when adopting TOSCA was to keep our internal format and add TOSCA as 
> kind of front-end format on-top. This works, if the internal format 
> and TOSCA (or any other
> standard) are well enough aligned.
>
> So the discussion should maybe not be the adoption of one or the other 
> format as the "native Heat metamodel", but the evolution of the 
> internal meta model to a state where it (1) fits the use cases Heat 
> wants to address, and (2) is aligned with the desired other formats as 
> closely as possible. Then we could have very thin pluggable layers 
> that add support for alternative "front-end formats".

+1

> Maybe we can have exactly this discussion next week. I have a session 
> on the TOSCA proposal at 11:50am on Monday, and I see your session is 
> scheduled for after lunch.
> My plan was to give an overview of TOSCA in my session since maybe not 
> all are familiar with it, but not spend too much time on it, but use 
> the session for discussing the points above.

Both the TOSCA overview and this discussion sound very valuable to me. 
Thanks to a last-minute change of plans I will be at Summit, so I look forward to hearing them.

FWIW it appears to me that CAMP is targeted at the PaaS level, and while I'm supportive of that effort it doesn't seem to me to be in scope for Heat, which I see as limited to the IaaS layer. I'm willing to be re-educated though ;)

cheers,
Zane.

>
> Does that make sense to you?
>
> Regards,
> Thomas
> ---------------------------------------------------------------------
> Tivoli CTO Office - Cloud Orchestration, Cloud Standards
>
>
>
> From:	Clint Byrum <clint at fewbar.com>
> To:	openstack-dev <openstack-dev at lists.openstack.org>,
> Date:	10.04.2013 08:25
> Subject:	[openstack-dev] [Heat] TOSCA, CAMP, CloudFormation, ???
>
>
>
> As we near the next OpenStack Summit, it has become clear that the way 
> forward for Heat users is.. well, not very clear.
>
> Heat currently only supports one way to define applications. That is 
> the hybrid Heat-specific plus CloudFormation-compatible templating language.
>
> We also have a proposal to add TOSCA support to Heat:
>
> https://blueprints.launchpad.net/heat/+spec/tosca-support
> http://lists.openstack.org/pipermail/openstack-dev/2013-April/007252.h
> tml
>
> Adrian from RackSpace has stated that they are working on an "Open API 
> & DSL" and a "Declarative Model":
>
> http://lists.openstack.org/pipermail/openstack-dev/2013-April/007126.h
> tml
>
> I've also heard OASIS/CAMP thrown around as something that might be 
> implemented in Heat.
>
> I count no less than 4 possible ways for users to express application 
> deployments. Succinctly:
>
> * CloudFormation-ish
> * Open DSL w/ Declarative model(undefined yet)
> * TOSCA
> * CAMP
>
> http://xkcd.com/927/
>
> I am concerned about the fragmentation that users may be presented 
> with if we do not take a step back and talk about the ramifications of 
> supporting all of them. Its great to be as broad and accepting as 
> possible when it makes sense, but not at the expense of user confusion 
> and/or incompatibility.
>
> So, I am calling on those interested in the various formats to at 
> least raise your hands and let us, the heat users and developers, know 
> what you expect from Heat as a project, and what makes any of these 
> standards worth implementing. I think this is worth having a break-out 
> session at the summit next week to discuss, and would also encourage 
> everyone to start (or continue) the discussion right here on the mailing list.
>
>
> Clint Byrum
> HP Cloud Services
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev