Open Stack

On 27/05/13 13:10 +0200, Zane Bitter wrote:
>Or, more specifically, _where_ to handle multiple template formats. 
>Two options have been proposed (with patches, even!):
>
>In the API:
>    https://review.openstack.org/#/c/30405/
>In the Engine:
>    https://review.openstack.org/#/c/30439/
>
>As Steve H correctly points out in the reviews, the question of which 
>is preferred comes down to whether we should translate to some 
>internal format immediately on reception of the template, or if we 
>should delay until it is received by the engine. Originally it was 
>thought by everyone that translation would need to happen 
>immediately, and this assumption is captured in 
>https://wiki.openstack.org/wiki/Heat/Vision.
>
>However, to my mind this comes with a major disadvantage: the plan, 
>as I understand it, is to initially translate HOT documents to the 
>currently-native CloudFormation format to allow experimentation with 
>HOT. However this has a very short horizon of usefulness - the entire 
>purpose of HOT is to express things that cannot be expressed in the 
>CloudFormation format (indeed, to the extent that it represents only 
>a less-documented version of the same thing with different names, it 
>would be worse than useless). So this would be followed quickly by an 
>inversion of the translation and a switch to HOT as the native format 
>of the engine - a switch that afaict would need to be pulled off in 
>one massive hit.
>
>By contrast, putting the code in the engine offers the prospect of a 
>more incremental way of changing over: by gradually replacing code 
>that directly manipulates the structure of the template data with 
>programmatic interfaces to to it (e.g. instead of doing 
>template['Parameters'] and getting a JSON tree back, call 
>template.parameters() and get a Parameters object back). No doubt 
>this will present difficulties with maintaining two parallel template 
>formats, although it's hard to say how much worse that would be than 
>maintaining the translation code.
>

I think we need to seperate out the short term and long term.

Long term:
convert cfn to hot in the api and have a "pure" hot based engine.

The problem is we currently have this the other way.


>I believe that one of the reasons for our original instinct that 
>translation would need to happen on input was the uncertainty around 
>how we will deal with templates that are composed of multiple files. 

I think there are multiple problems, this been one.

- migrating all the database data (state/instance names, data
   structure).
- translating cfn concepts to new ones in hot

I think we simply need to start pushing the new concepts into
the engine and at the same time putting a covertion from cfn to
the partially implmented hot into the heat-cfn-api.


>(This is already a pain point, with AWS::CloudFormation::Stack 
>resources having to obtain their templates from a URL.) TOSCA solves 
>this with zip files, and if we were to follow suit then the need for 
>some translation on the front end would be basically a given.
>
>One concern I have with the HOT discussion as I have understood it so 
>far is that it seems to propose the template format as a bit of a 
>"file soup" - you throw a bunch of files at Heat, which figures out 
>how they relate to each other and forms them into a stack. This is, 
>of course, fairly trivial to do in code, but for the person reading 
>or writing a template it is not so easy. I would prefer something 
>more akin to the current system (including the proposed Provider 
>blueprints), which is essentially hierarchical. You start with one 
>thing that is The Template and can dig down into subsystems (nested 
>stack templates) or implementations (provider templates) - and, in 
>HOT, other resources like scripts or configuration management data. 
>This gives the user a mental foothold[1] from which to begin 
>interpreting the template.
>
>I'd like to propose in more concrete form an idea that I started 
>kicking around on the mailing list a few weeks back[2], but which 
>didn't engender any discussion at the time.
>
>* Templates could reference external files using a function similar 
>to "Ref", but for files. I'm leaning toward it being called "Url" and 
>it accepting both relative and absolute URLs. (Other options might be 
>"File" or "Include".)
>* Clients (i.e. python-heatclient) could recursively parse templates 
>for {"Url": "..."} references, and ensure all of the necessary files, 
>including e.g. relative file paths, are included in the request. 
>(This is the reason for the function - the client need not know 
>anything else about the template format.)
>* We would take advantage of our APIs being JSON/YAML-only to simply 
>include all of the files inline in a request (either as JSON or as 
>strings). This gives us the advantages of zip files, without the 
>disadvantages (opaqueness to diffs being the major one).
>* Files other than the top-level template would be passed as a 
>dictionary/map keyed by the URL in the "Url" function. The engine, 
>when interpreting a "Url" function would first look in the files map, 
>before attempting to fetch from the URL if necessary and appropriate.
>
>A trivial example of what a ReST API POST request might look like:
>
>{
>    "stack_name": "{stack_name}",
>    "parameters": {
>        "param_name-1": "param_value-1",
>        "param_name-2": "param_value-2"
>    },
>    "timeout_mins": {timeout_mins},
>    "template": {
>        "parameters": {
>            <elided> ...
>        },
>        "resources": {
>            "nested-stack-1": {
>                type: "os::orchestration::stack",
>                template: {"Url": "./substack-1.template"},
>                parameters: {"param_name": {"Ref": "param_name-1"}}
>            },
>            "nested-stack-2": {
>                type: "os::orchestration::stack",
>                template: {"Url": "./substack-2.template"},
>                parameters: {"param_name": {"Ref": "param_name-2"}}
>            }
>        }
>    },
>    "files": {
>         "./substack-1.template": {
>             <elided> ...
>         },
>         "./substack-2.template": {
>             <elided> ...
>         }
>    }
>}
>
>(only the "files" section is new from the API's point of view).
>
>I would really like this to be able to replace the blueprint that I 
>proposed earlier for uploading provider templates and storing them in 
>the engine[3]. I have always feared that that proposal will result in 
>the sort of statefulness that reduces the chances of anybody ever 
>being able to launch the same stack twice to near zero. Whether this 

Yea, I worry about that too, but at the same time declartive templates
are a bit like libraries in that bugs can be fixed and if you
re-create the stack later it won't have the same bugs (maybe some new
ones;).

>can replace that, however, depends very much on what the 
>"Environments" feature ends up looking like, and figuring that out is 
>another discussion[4]. (One thing that would help: an option to 
>specify a different URL Base in the environment for providers.)
>
>Getting back to the topic at hand (remember that?), I believe that 
>handling multiple input files in roughly this fashion, or something 
>equivalent, would obviate the need to do translation of templates at 
>the API level and tilt the calculus in favour of doing it in the 
>engine.
>
>cheers,
>Zane.
>
>
>[1] Sorry.
>[2] http://lists.openstack.org/pipermail/openstack-dev/2013-May/008703.html
>[3] https://blueprints.launchpad.net/heat/+spec/provider-upload
>[4] Angus has suggested an interesting starting point here: 
>https://wiki.openstack.org/wiki/Heat/Environments
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev