Open Stack

On 03/04/13 01:03, Duncan Mcgreggor wrote:
> On 4/2/13 3:47 PM, Adrian Otto wrote:
>> Hello,
>>
>> Rackspace has resourced two dedicated development teams for the sole
>> purpose of contributing new features and capabilities to OpenStack's
>> HEAT project. We are very excited, and would like to share with you
>> what we plan to design and contribute together with you:

Welcome! :)

BTW Heat is not an acronym. Just putting that out there ;)

>> *1) Open API & DSL *- This allows templates to be agnostic to the
>> underlying cloud and encourages community contribution for the
>> betterment of all users across all cloud providers. We want a solution
>> that does not depend on semantics from any single service provider. We
>> think there is a way for HEAT to work equally well with CloudFormation
>> templates, and a completely open template format as well.

First off, I would note that Heat already has an OpenStack-native ReST 
API[1] that is not derived in any way from the CloudFormation API and 
that I hope meets your standards of Openness.

As for the DSL, I'm not sure what y'all are proposing here. I believe 
everybody is agreed that we can and should go to town on implementing 
OpenStack-native resource types (and, indeed, a few exist already - 
especially for Quantum[2]).

Adding another template _format_, however, seems like a very involved 
task (though I would be interested to hear your ideas for implementing 
that). If that's what you're suggesting, then I'd invite you, as I have 
invited everybody else who has suggested it, to start a discussion on 
this list about how a different format can add value for users (for 
anyone keeping score at home, total discussions started to date: 0).

As examples, we've already added support for YAML-format templates, 
which I think are much more readable and easier to diff than 
CloudFormation's JSON ones, and we have some blueprint proposals for new 
built-in functions. If anybody has other ideas to contribute, it would 
be great to hear them.

>>
>> *2) Declarative Model* - Although CloudFormation Templates were
>> designed to be declarative, in practice the templates are very
>> imperative artifacts (for example, those that embed shell scripts).
>> Templates that are expressed using a declarative approach are compact,
>> simple, and portable between clouds that have different services
>> available. We want the cloud implementation specific details to be
>> handled by modules, not wired into the templates. Declarative modeling
>> encourages broad contribution from the user base to improve the
>> overall community library of available solutions. While modeling may
>> be easy to implement, they are more difficult to expand to support
>> generic cloud portable use cases.

So, to clarify for readers who are less familiar with Heat, the main 
point of Heat templates is that they can model *all* of the resource 
types that exist in OpenStack, and the templates are completely 
declarative. What is imperative, however, is the UserData passed to Nova 
instances, and in practice this tends to be a large part of the template 
because obviously Nova instances and what you run on them are very 
important to many of the things you might want to do in a cloud. I 
wanted to clear that up, though, because one of the most common 
misconceptions about Heat is that it's _only_ about launching Nova 
instances.

That said, I would love to see some sort of native integration with e.g. 
Puppet/Chef/Juju for configuring instances instead of shell scripts, and 
that sounds like the kind of thing you are proposing? I'll look forward 
to seeing the blueprints :)

>>
>> *3) Modular Implementation* - We want HEAT to be modular in a way
>> that's consistent with the level of modularity offered in Nova,
>> Quantum, Cinder and others where a common, extendable API is offered
>> and a variety of extensions may be added for various back-end services
>> and features. We want to keep the architecture as simple as possible
>> while allowing individual cloud operators to add features and
>> capabilities in a way that keeps templates crisp and portable.

Heat already has a pluggable resource model[3], so the cloud operator 
can select which resource implementations to deploy (e.g. you could 
easily plug in a custom resource to control your native loadbalancer 
instead of using the one built in to Heat, which spins up a Nova 
instance to do it). There's a bit more work to do in making it easier 
for operators to configure which of the built-in resources they want 
enabled[4], and also for exposing information about the resources 
available to the user[5]. Contributions in this area would be welcome.

There is already an interface for plugging in different cloud backends 
(e.g. use Apache Deltacloud APIs instead of OpenStack APIs) although, as 
I understand it, it relies on backends emulating the OpenStack 
python-*client APIs.

There is a blueprint for adding pluggable functions, so operators can 
supply their own definitions for template functions.[6]

If you can think of other interfaces that need to be pluggable, please 
go ahead and add blueprints for them.

>>
>> *4) Auto-Scale Implementation* - The solution will allow deployments
>> to scale up and down dynamically based on demand. We want to design
>> and implement this with you. We have considerable experience and
>> resources to bring with us. We have a dedicated team to contribute
>> solutions here.
>>
> Just to clarify the autoscale bit: we are well aware that there is
> currently autoscale support in Heat right now, and there's no intent
> (nor desire!) to reimplement any of that, nor throw anything over the
> wall ;-)
>
> We had some great chats at PyCon with some folks about Heat and are
> really looking forward to ODS to dive in more deeply and get to know the
> current status, project priorities, etc. We've been lurking on IRC and
> started attending the weekly meetings recently.
>
> There does seem to be some missing integration for monitoring and LBaaS
> (no surprises there for anyone, as that is all currently under active
> development), and this is where we want to focus our initial efforts.
> Well, here as well as advocating for consensus around an autoscale API
> suitable for consumption by integrators/devops/application devs/etc.
> We've created a blueprint in LP and proposed a session for discussing
> some of these things (focused on defining where folks think we are with
> regard to an AS API and where we want to go with that).
>
> I've got a blog post pending with some more thoughts about this, and
> that should be up soon. I'll reply with a link when it has been published...

Sounds good, there's obvious value in having an API for autoscaling that 
can be used even by users that aren't using Heat templates. It seems to 
me that the logical course of action would be to start by splitting out 
the existing code in Heat into a separate service, making sure the 
interface between the two is clean, and turning that into an API.

>> We look forward to discussing more with you at the Summit in Portland. Leaders from each of our new teams will be there ready to work with you on implementation details.

That's great, but for the benefit of those of us who won't be at Summit 
(i.e. me ;) please also generate blueprints and/or mailing list threads.

cheers,
Zane.

[1] https://github.com/openstack/heat/blob/master/docs/api.md#readme
[2] 
https://github.com/openstack/heat/tree/master/heat/engine/resources/quantum
[3] https://wiki.openstack.org/wiki/Heat/Plugins
[4] https://blueprints.launchpad.net/heat/+spec/abstract-aws
[5] https://blueprints.launchpad.net/heat/+spec/resource-properties-schema
[6] https://blueprints.launchpad.net/heat/+spec/function-plugins