Open Stack

Hi Zane,

sorry for my late response. I was out on vacation last week and catching up
now with what happend in the meantime.
I have some inline comments below.

Regards,
Thomas

Zane Bitter <zbitter at redhat.com> wrote on 27.05.2013 13:10:20:

> From: Zane Bitter <zbitter at redhat.com>
> To: OpenStack Development Mailing List
<openstack-dev at lists.openstack.org>,
> Date: 27.05.2013 13:19
> Subject: [openstack-dev] [Heat] On how to handle multiple template
formats
>
> Or, more specifically, _where_ to handle multiple template formats. Two
> options have been proposed (with patches, even!):
>
> In the API:
>      https://review.openstack.org/#/c/30405/
> In the Engine:
>      https://review.openstack.org/#/c/30439/
>
> As Steve H correctly points out in the reviews, the question of which is
> preferred comes down to whether we should translate to some internal
> format immediately on reception of the template, or if we should delay
> until it is received by the engine. Originally it was thought by
> everyone that translation would need to happen immediately, and this
> assumption is captured in https://wiki.openstack.org/wiki/Heat/Vision.
>
> However, to my mind this comes with a major disadvantage: the plan, as I
> understand it, is to initially translate HOT documents to the
> currently-native CloudFormation format to allow experimentation with
> HOT. However this has a very short horizon of usefulness - the entire
> purpose of HOT is to express things that cannot be expressed in the
> CloudFormation format (indeed, to the extent that it represents only a
> less-documented version of the same thing with different names, it would
> be worse than useless). So this would be followed quickly by an
> inversion of the translation and a switch to HOT as the native format of
> the engine - a switch that afaict would need to be pulled off in one
> massive hit.
>
> By contrast, putting the code in the engine offers the prospect of a
> more incremental way of changing over: by gradually replacing code that
> directly manipulates the structure of the template data with
> programmatic interfaces to to it (e.g. instead of doing
> template['Parameters'] and getting a JSON tree back, call
> template.parameters() and get a Parameters object back). No doubt this
> will present difficulties with maintaining two parallel template
> formats, although it's hard to say how much worse that would be than
> maintaining the translation code.

My favorite option out of the two proposals would also have been the 2nd
template class, and looking at the updates in gerrit it seems like that one
got in :-) I fully agree that this is the right path for future iterations.
I will get started based on this for the work to be done under the
hot-hello-world BP.

>
> I believe that one of the reasons for our original instinct that
> translation would need to happen on input was the uncertainty around how
> we will deal with templates that are composed of multiple files. (This
> is already a pain point, with AWS::CloudFormation::Stack resources
> having to obtain their templates from a URL.) TOSCA solves this with zip
> files, and if we were to follow suit then the need for some translation
> on the front end would be basically a given.

I think establishing the support for HOT inside the core of the engine is
the right thing for HOT. For other formats we can still have something
outside the core. Also dealing multiple files could be something to be
added on top. I don't see a real conflict. Making the whole thing
consumable if of course something to keep in mind.

>
> One concern I have with the HOT discussion as I have understood it so
> far is that it seems to propose the template format as a bit of a "file
> soup" - you throw a bunch of files at Heat, which figures out how they
> relate to each other and forms them into a stack. This is, of course,
> fairly trivial to do in code, but for the person reading or writing a
> template it is not so easy. I would prefer something more akin to the
> current system (including the proposed Provider blueprints), which is
> essentially hierarchical. You start with one thing that is The Template
> and can dig down into subsystems (nested stack templates) or
> implementations (provider templates) - and, in HOT, other resources like
> scripts or configuration management data. This gives the user a mental
> foothold[1] from which to begin interpreting the template.

So for the provider topic part of what you said, would that include a model
like the one being discussed here?

https://wiki.openstack.org/wiki/Heat/Providers

I think that is something we should definitely have, since it formalizes
the de-composition of big templates into smaller, re-usable templates in a
very nice way. So it brings some order to the "file soup".

>
> I'd like to propose in more concrete form an idea that I started kicking
> around on the mailing list a few weeks back[2], but which didn't
> engender any discussion at the time.
>
> * Templates could reference external files using a function similar to
> "Ref", but for files. I'm leaning toward it being called "Url" and it
> accepting both relative and absolute URLs. (Other options might be
> "File" or "Include".)
> * Clients (i.e. python-heatclient) could recursively parse templates for
> {"Url": "..."} references, and ensure all of the necessary files,
> including e.g. relative file paths, are included in the request. (This
> is the reason for the function - the client need not know anything else
> about the template format.)
> * We would take advantage of our APIs being JSON/YAML-only to simply
> include all of the files inline in a request (either as JSON or as
> strings). This gives us the advantages of zip files, without the
> disadvantages (opaqueness to diffs being the major one).
> * Files other than the top-level template would be passed as a
> dictionary/map keyed by the URL in the "Url" function. The engine, when
> interpreting a "Url" function would first look in the files map, before
> attempting to fetch from the URL if necessary and appropriate.
>
> A trivial example of what a ReST API POST request might look like:
>
> {
>      "stack_name": "{stack_name}",
>      "parameters": {
>          "param_name-1": "param_value-1",
>          "param_name-2": "param_value-2"
>      },
>      "timeout_mins": {timeout_mins},
>      "template": {
>          "parameters": {
>              <elided> ...
>          },
>          "resources": {
>              "nested-stack-1": {
>                  type: "os::orchestration::stack",
>                  template: {"Url": "./substack-1.template"},
>                  parameters: {"param_name": {"Ref": "param_name-1"}}
>              },
>              "nested-stack-2": {
>                  type: "os::orchestration::stack",
>                  template: {"Url": "./substack-2.template"},
>                  parameters: {"param_name": {"Ref": "param_name-2"}}
>              }
>          }
>      },
>      "files": {
>           "./substack-1.template": {
>               <elided> ...
>           },
>           "./substack-2.template": {
>               <elided> ...
>           }
>      }
> }
>
> (only the "files" section is new from the API's point of view).

So is this really just the API view? I.e. different templates could still
be in separate files and then get composed into one POST?
For the multiple files being nested templates, I think I still find the
solution discussed on the Heat/Providers wiki page (see above) a cleaner
approach.

>
> I would really like this to be able to replace the blueprint that I
> proposed earlier for uploading provider templates and storing them in
> the engine[3]. I have always feared that that proposal will result in
> the sort of statefulness that reduces the chances of anybody ever being
> able to launch the same stack twice to near zero. Whether this can
> replace that, however, depends very much on what the "Environments"
> feature ends up looking like, and figuring that out is another
> discussion[4]. (One thing that would help: an option to specify a
> different URL Base in the environment for providers.)
>
> Getting back to the topic at hand (remember that?), I believe that
> handling multiple input files in roughly this fashion, or something
> equivalent, would obviate the need to do translation of templates at the
> API level and tilt the calculus in favour of doing it in the engine.

As I said earlier, at some point translation at the API level could make
sense. Maybe the current situation with HOT and CFN in the core is a
special case for the time being, but if in the future HOT is the only
format inside the core it could make sense to move as much as possible a
bit to the outside.

>
> cheers,
> Zane.
>
>
> [1] Sorry.
> [2]
http://lists.openstack.org/pipermail/openstack-dev/2013-May/008703.html
> [3] https://blueprints.launchpad.net/heat/+spec/provider-upload
> [4] Angus has suggested an interesting starting point here:
> https://wiki.openstack.org/wiki/Heat/Environments
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>