Open Stack

On 25/04/13 23:36, Clint Byrum wrote:
>
> Kudos for driving real use cases first. Let me just say that I think as
> it is written, without modification, it will be useful. I have a bunch
> of ideas that I'll add in-line below that, if nothing else, I hope can
> drive the ordering so that we can start to benefit from some of this
> before a new DSL is brought in.

Thanks Clint, I appreciate the feedback. I added some comments inline, 
with heavy snipping.

>> Of course that's exactly what we're all about on this project :). So
>> my proposal is to allow users to define their own resource types using
>> a Heat template. Heat would know to use this template instead of a
>> built-in type from a "Provider" member in the Resource definition that
>> contains a URL to the template. (I'm appropriating the name "Provider"
>> from Rackspace's DSL proposal for now because inventing new names for
>> things that already exist is a sucker's game.)
>>
>
> IMO nested stacks really are already this feature, they just need to be
> elevated to the level of, say, images, by letting heat users upload and
> name templates.

The main point here is that you don't have to change your template to 
run it on a different cloud, because one of these nested stacks could 
masquerade as another resource type and take the same parameters, expose 
the same attributes &c.

So in the example given, if you want to develop on a server sitting 
under your buddy's desk with a nested stack of your choice providing the 
database then you can, and when you go to deploy on a proper cloud with 
a DBaaS then the only change to your template is removing the provider 
URL (could easily be a parameter) - and if we're allowing people to 
upload templates and potentially set them as the default provider, you 
might not even have to change that much.

So, yes, it's just syntactic sugar. But I think it's syntactic sugar 
that makes workloads more portable across heterogeneous clouds.

(BTW there's probably an important missing detail in my writeup: the 
provider stack needs to supply a RefId output, so that other resources 
can refer to it as if it were the encapsulated resource.)

>> * Introduce the concept of _stack_ Metadata, and provide a way to
>> access it in a template (pseudo-parameter?).
>
> +1
>
> I know we're polishing off a new DSL, but
>
> {Ref: Heat::Stack::Metadata} for accessing your own, and Fn::GetAtt [
> StackResource, Metadata ] for accessing a nested stack's metadata would
> work. And really couldn't that just be an Output?

Yeah, so the idea was that Metadata isn't a Property, but the provider 
templates still need to refer to it. The pseudo-parameter idea is 
basically the first one you listed: {"Ref": "OS::Heat::StackMetadata"} 
or something, and that's probably the way to go (need to support JSON 
parameters first though). Exporting the Metadata is probably not even 
necessary at the template level (you can't refer to another resource's 
metadata in a template now).

>> A possible avenue for increasing flexibility:
>> * Add support for more template functions that manipulate the template
>> directly:
>>   - Array/dictionary (uh, object) lookup
>>   - more string functions?
>>   - maybe conditionals?
>>   - definitely NOT loops/map.
>>
>
> -1 for logical operations
> +1 for better data structures
>
> The goal is always, IMO, to build the graph based on the templates.
> Conditional logic belongs inside instances and inside Heat itself, not
> in templates. Perhaps you have a concrete example where having
> conditionals is warranted?

Not really (I think I was thinking about the Provider field in 
particular), and that's why I was very tentative on it. But I can see 
the need arising for richer processing of template data in the future. 
It needs to be driven by real use cases that folks like you are running 
into.

>> What might all of this give us?
>>  + Users are no longer dependent on the operator to provide the
>> resource type they need (perhaps for cross-cloud compatibility), but
>> can supply their own.
>
> Really users can do this now by using nested stacks, and changing the
> URL based on the cloud provider. This is rather cumbersome, but can be
> used today with the current CFN implementation to show how one template
> can deploy using different implementations of RDS or something else.

Yeah, see above. It's possible (but cumbersome) for basic stuff. But if 
e.g. you can change the default provider for the Instance type then you 
could have your autoscaling group autoscale your nested stack. There's 
no way to do that at the moment.

>>  + Users can effectively subclass built-in types. For example, you
>> could create a Puppet Instance template that configures an instance as
>> a Puppet slave, then selectively use that in place of a regular
>> instance and just pass the metadata to specialise it.
>
> Getting dangerously close to needing multiple inheritance and all of the
> evil complexity that brings. I'd rather focus on composition. Just give
> me a way to express all of the things I want to have in the instance:
>
> In CFN, what I really want is:
>
> PuppetAndUsefulThingServer:
>    Type: OS::Heat::Stack
>    Properties:
>      Url: url_to_implementation
>      Parameters:
>        Metadata:
>          - {Include: PuppetMetadata}
>          - {Include: UsefulThingsMetadata}
>
> Anyway, if we focus on sub-classing rather than composition, we are
> going to have to define all of the ways inheritance affects behavior.

Multiple inheritance is definitely out of the question, but chains of 
inheritance would be possible in this model.

I need to understand this example better because I think it's pretty 
core to what we're trying to achieve, so please correct me where I 
misunderstand, but here's kind of my picture of how it would work:

  - The UsefulThing is configured by Puppet
  - url_to_implementation could be a link to a Puppet provider template
  - PuppetMetadata would be the Puppet manifest (or whatever) to tell 
puppet to set up UsefulThing, and whatever else you want to set up.
  - UsefulThingsMetadata is the configuration data for your UsefulThing.
  - At some point you are using lots of UsefulThings and you want to 
make a provider just for that.
  - Now you only need to pass UsefulThingsMetadata, the PuppetMetadata 
having been abstracted into the UsefulThing provider template
  - The UsefulThing provider combines its incoming metadata with the 
puppet metadata and passes it to the Puppet provider template.

You could also invert this and have a Puppet provider that takes a 
parameter that tells it which provider to use for it's instance. I have 
a feeling that may turn out to be more flexible... but we won't really 
know until people start using it.

>> How else might we improve this? Well, having to load a template from
>> a URL is definitely a limitation - albeit not a new one (the
>> AWS::CloudFormation::Stack resource type already has this limitation).
>> Perhaps we should let the user post multiple, named, implementations
>> of a resource type and reference them by name (instead of URL) in the
>> "Provider" field.
>>
>
> Right, I think we could very easily add bits to the REST API to allow
> users to store templates in Heat and reference them by name or UUID.
> Just doing that for the nested stack would be a big win from my
> perspective.

OK, good to know.

> As a longer term goal, being able to import template
> repositories from outside one's own tenant (or even cloud...) would be a
> nice implementation of the "let puppet publish puppet templates". Sort
> of like the idea that has kicked around for a while to run a public
> glance server with all of the stock images in it.

Isn't this just getting back to grabbing them from a public URL again?

>>
>> For easier collation, please categorise your response as follows:
>>  (A) I'm appalled at the mere suggestion
>>  (B) This just prevents us solving the real problem (please specify)
>>  (C) Meh
>>  (D) This looks kind of interesting
>
> D+ for me. Seems like we can take some real steps toward a nice
> infrastructure that will support all of the incoming template formats.
>
>>  (E) OMG!! UNICORNS!!!

Thanks Clint, very helpful input.

cheers,
Zane.