Open Stack

On 06/05/13 22:55, Clint Byrum wrote:
> On 2013-05-06 07:38, Zane Bitter wrote:
>> On 02/05/13 19:39, Tripp, Travis S wrote:
>>> I agree with Clint.  This essentially boils down to being able to
>>> describe my application(s) and how to deploy them separately from the
>>> resources where I want to deploy them. Then based on the target
>>> environment, I have a way to deploy my app to different resources
>>> without having to modify / copy paste my app model everywhere.  In
>>> TOSCA terms, this is what the "relationship" concept provides. For
>>> example, I can design my application in one template and
>>> infrastructure(s) in another template. Then I essentially can have
>>> different deployments where I use the relationship to establish a
>>> source and a target for these relationship (App part A is associated
>>> to Infra part X). I just spoke with Thomas Spatzier and I think he is
>>> going to provide a simplified JSON or YAML representation of this
>>> concept.
>>
>> This use case makes complete sense to me.
>>
>> Here's the part I'm struggling with:
>>
>>> For example, I can design my application in one template and
>>> infrastructure(s) in another template.
>>
>> So... this exists, right?
>>
>> We call the first part "configuration management" (Puppet, Chef) and
>> the second part "cloud orchestration" (Heat, CloudFormation).
>>
>
> As Thomas said, this is not really the case I'm talking about.
>
> Notice here:
>
> https://github.com/openstack-ops/templates/blob/master/heat.yaml
>
> All the things to deploy the Heat engine on one machine, and the API's
> all together in a scaling group.
>
> https://github.com/openstack-ops/templates/blob/master/keystone.yaml
>
> All the things to deploy Keystone on an instance group.
>
> You'll notice a complete lack of configuration management there. We use
> Metadata to drive os-config-applier and os-refresh-config, which are the
> TripleO minimalistic "config management" equivalents.

Yeah, I probably misspoke there. The concept I was intending to convey 
includes solutions like yours, which are more accurately described as 
"configuration management" than Configuration Management.
>
> What I'd like to do is run keystone and heat-api's on a single instance
> group which encompass all of my stateless API services. To do that, I
> have to duplicate *most* of the templates I've already written, only
> with the metadata merged.

The provider templates proposal would help with that somewhat, wouldn't 
it? You could abstract all the stuff you would otherwise have to 
duplicate into a provider template and just pass the metadata that is 
different (which, with some appropriate functions available in the 
provider template for merging JSON, could be as small as a handful of 
lines).

> So, I submit that there need to be a way to specify an abstract resource
> which has the Metadata/Parameters/Relationships needed to deploy said
> resource, but does not have the machine placement details. Those would
> then be included in the instance groups, something like:
>
> IncludeMetadata:
>    - HeatAPIMetadata
>    - KeystoneMetadata

Yes, point taken.

The key issue that seems to keep emerging is how to manage stacks where 
the templates exist across multiple files - if we had a good way of 
handling that then something like loading the metadata from a couple of 
separate files like this would be trivial to implement.

AWS::CloudFormation::Stack resources are already a problem in this 
respect, and provider templates will make it much, much worse. I'm not a 
fan of the zip file concept, but if we had a client which was smart 
enough to find the referenced files in the filesystem and produce the 
zip archive itself, maybe that would be almost tolerable. Or even better 
- the fact that our API is all JSON gives us the flexibility (unlike 
XML) to include multiple files in the request without zipping.

What would folks think of that? Could it replace the previous proposal I 
made about uploading providers 
(https://blueprints.launchpad.net/heat/+spec/provider-upload)? There 
seem to be a couple of downsides:

1) You'd lose the ability to upload the exact same template to different 
cloud providers and have them do the Right Thing based on provider 
templates you had already set up. On the other hand, you could probably 
select them via parameters, so that wouldn't be a disaster.
2) To really work automatically, it would require the client to be able 
to parse the template sufficiently to work out which file to send. This 
could create serious, and maybe insoluble, dependency issues with the 
code. Maybe if we just create a "File" function that works like "Ref" 
but for files? That would be simple for the client to find without 
needing to understand the template.

The upside, though, is that I'm frightened by the prospect of creating a 
system where the user has basically zero chance of ever creating the 
same stack twice from the same template without implementing a lot of 
external process (which most people won't do), and this would eliminate 
one of the biggest issues there.

I still like the replace-the-API-with-a-git-repo idea too, but it would 
be a pig to implement :/

cheers,
Zane.