Open Stack

Angus Salkeld <asalkeld at redhat.com> wrote on 27.05.2013 14:35:11:

> From: Angus Salkeld <asalkeld at redhat.com>
> To: openstack-dev at lists.openstack.org,
> Date: 27.05.2013 14:41
> Subject: Re: [openstack-dev] [Heat] On how to handle multiple template
formats
>
> On 27/05/13 13:10 +0200, Zane Bitter wrote:
> >Or, more specifically, _where_ to handle multiple template formats.
> >Two options have been proposed (with patches, even!):
> >
> >In the API:
> >    https://review.openstack.org/#/c/30405/
> >In the Engine:
> >    https://review.openstack.org/#/c/30439/
> >
> >As Steve H correctly points out in the reviews, the question of which
> >is preferred comes down to whether we should translate to some
> >internal format immediately on reception of the template, or if we
> >should delay until it is received by the engine. Originally it was
> >thought by everyone that translation would need to happen
> >immediately, and this assumption is captured in
> >https://wiki.openstack.org/wiki/Heat/Vision.
> >
> >However, to my mind this comes with a major disadvantage: the plan,
> >as I understand it, is to initially translate HOT documents to the
> >currently-native CloudFormation format to allow experimentation with
> >HOT. However this has a very short horizon of usefulness - the entire
> >purpose of HOT is to express things that cannot be expressed in the
> >CloudFormation format (indeed, to the extent that it represents only
> >a less-documented version of the same thing with different names, it
> >would be worse than useless). So this would be followed quickly by an
> >inversion of the translation and a switch to HOT as the native format
> >of the engine - a switch that afaict would need to be pulled off in
> >one massive hit.
> >
> >By contrast, putting the code in the engine offers the prospect of a
> >more incremental way of changing over: by gradually replacing code
> >that directly manipulates the structure of the template data with
> >programmatic interfaces to to it (e.g. instead of doing
> >template['Parameters'] and getting a JSON tree back, call
> >template.parameters() and get a Parameters object back). No doubt
> >this will present difficulties with maintaining two parallel template
> >formats, although it's hard to say how much worse that would be than
> >maintaining the translation code.
> >
>
> I think we need to seperate out the short term and long term.
>
> Long term:
> convert cfn to hot in the api and have a "pure" hot based engine.
>
> The problem is we currently have this the other way.
>
>
> >I believe that one of the reasons for our original instinct that
> >translation would need to happen on input was the uncertainty around
> >how we will deal with templates that are composed of multiple files.
>
> I think there are multiple problems, this been one.
>
> - migrating all the database data (state/instance names, data
>    structure).
> - translating cfn concepts to new ones in hot
>
> I think we simply need to start pushing the new concepts into
> the engine and at the same time putting a covertion from cfn to
> the partially implmented hot into the heat-cfn-api.

I think that's a very good point, i.e. in parallel to working on HOT
support in the engine based on the 2nd template class patch that came in
write cfn to HOT conversion code at the API level for the set of features
defined and implemented in HOT at any given point in time. Theoretically,
that would provide us with a complete translation layer for CFN to HOT once
HOT is mature enough to cover CFN functionality. I can try to pick that
idea up in the hot-hello-world BP related work.

>
>
> >(This is already a pain point, with AWS::CloudFormation::Stack
> >resources having to obtain their templates from a URL.) TOSCA solves
> >this with zip files, and if we were to follow suit then the need for
> >some translation on the front end would be basically a given.
> >
> >One concern I have with the HOT discussion as I have understood it so
> >far is that it seems to propose the template format as a bit of a
> >"file soup" - you throw a bunch of files at Heat, which figures out
> >how they relate to each other and forms them into a stack. This is,
> >of course, fairly trivial to do in code, but for the person reading
> >or writing a template it is not so easy. I would prefer something
> >more akin to the current system (including the proposed Provider
> >blueprints), which is essentially hierarchical. You start with one
> >thing that is The Template and can dig down into subsystems (nested
> >stack templates) or implementations (provider templates) - and, in
> >HOT, other resources like scripts or configuration management data.
> >This gives the user a mental foothold[1] from which to begin
> >interpreting the template.
> >
> >I'd like to propose in more concrete form an idea that I started
> >kicking around on the mailing list a few weeks back[2], but which
> >didn't engender any discussion at the time.
> >
> >* Templates could reference external files using a function similar
> >to "Ref", but for files. I'm leaning toward it being called "Url" and
> >it accepting both relative and absolute URLs. (Other options might be
> >"File" or "Include".)
> >* Clients (i.e. python-heatclient) could recursively parse templates
> >for {"Url": "..."} references, and ensure all of the necessary files,
> >including e.g. relative file paths, are included in the request.
> >(This is the reason for the function - the client need not know
> >anything else about the template format.)
> >* We would take advantage of our APIs being JSON/YAML-only to simply
> >include all of the files inline in a request (either as JSON or as
> >strings). This gives us the advantages of zip files, without the
> >disadvantages (opaqueness to diffs being the major one).
> >* Files other than the top-level template would be passed as a
> >dictionary/map keyed by the URL in the "Url" function. The engine,
> >when interpreting a "Url" function would first look in the files map,
> >before attempting to fetch from the URL if necessary and appropriate.
> >
> >A trivial example of what a ReST API POST request might look like:
> >
> >{
> >    "stack_name": "{stack_name}",
> >    "parameters": {
> >        "param_name-1": "param_value-1",
> >        "param_name-2": "param_value-2"
> >    },
> >    "timeout_mins": {timeout_mins},
> >    "template": {
> >        "parameters": {
> >            <elided> ...
> >        },
> >        "resources": {
> >            "nested-stack-1": {
> >                type: "os::orchestration::stack",
> >                template: {"Url": "./substack-1.template"},
> >                parameters: {"param_name": {"Ref": "param_name-1"}}
> >            },
> >            "nested-stack-2": {
> >                type: "os::orchestration::stack",
> >                template: {"Url": "./substack-2.template"},
> >                parameters: {"param_name": {"Ref": "param_name-2"}}
> >            }
> >        }
> >    },
> >    "files": {
> >         "./substack-1.template": {
> >             <elided> ...
> >         },
> >         "./substack-2.template": {
> >             <elided> ...
> >         }
> >    }
> >}
> >
> >(only the "files" section is new from the API's point of view).
> >
> >I would really like this to be able to replace the blueprint that I
> >proposed earlier for uploading provider templates and storing them in
> >the engine[3]. I have always feared that that proposal will result in
> >the sort of statefulness that reduces the chances of anybody ever
> >being able to launch the same stack twice to near zero. Whether this
>
> Yea, I worry about that too, but at the same time declartive templates
> are a bit like libraries in that bugs can be fixed and if you
> re-create the stack later it won't have the same bugs (maybe some new
> ones;).

You mean templates referenced by other templates (e.g. some middleware
platform used by an application level template) are like libraries?
If so, I strongly agree. And then being able to maintain them separately
and bind them at deployment time would be a nice feature. Also would fit
well with the environment concept, when you could have multiple
realizations of the same thing in different environments.

>
> >can replace that, however, depends very much on what the
> >"Environments" feature ends up looking like, and figuring that out is
> >another discussion[4]. (One thing that would help: an option to
> >specify a different URL Base in the environment for providers.)
> >
> >Getting back to the topic at hand (remember that?), I believe that
> >handling multiple input files in roughly this fashion, or something
> >equivalent, would obviate the need to do translation of templates at
> >the API level and tilt the calculus in favour of doing it in the
> >engine.
> >
> >cheers,
> >Zane.
> >
> >
> >[1] Sorry.
> >[2]
http://lists.openstack.org/pipermail/openstack-dev/2013-May/008703.html
> >[3] https://blueprints.launchpad.net/heat/+spec/provider-upload
> >[4] Angus has suggested an interesting starting point here:
> >https://wiki.openstack.org/wiki/Heat/Environments
> >
> >_______________________________________________
> >OpenStack-dev mailing list
> >OpenStack-dev at lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>