Open Stack

On 27/05/13 13:10 +0200, Zane Bitter wrote:
>Or, more specifically, _where_ to handle multiple template formats. 

[snip]

So I just want to show how this and the environment stuff I busy
with can work together:
https://review.openstack.org/#/q/status:open+project:openstack/heat+branch:master+topic:bp/environments,n,z
(after some rework, from the feedback)

I'll change the environment format to look like this (based on
Randall's idea):

#Use case 1: I want to use all the resource types from provider X
resource-registry:
   "OS::*": "Dreamhost::*"
   # could also use a url like this (assuming they could all be
   # expressed in nested stacks)
   "OS::*": http://dreamhost.com/bla/resources-types/*"

#Use case 2: I want to use mostly the default resources except my
custom one for a particular resource in the template.
resource-registry:
   resources:
     my_db_server:
       "OS::DBInstance": file://~/all_my_cool_templates/db.yaml

#Use case 3: I always want to always map resource type X to Y
resource-registry:
   "OS::Networking::FloatingIP": "OS::Nova::FloatingIP"
   "OS::Loadbalancer": file://~/all_my_cool_templates/lb.yaml

#Use case 4: I use custom resources a lot and want to shorten the
url/path
resource-registry:
   base_url: http://bla.foo/long/url/
   resources:
     my_db_server:
       "OS::DBInstance": dbaas.yaml

#Use case 5: I need to override a resource property in the template
properties:
   global:
   resources:
     my_db_server:
       ImageId: thisone_works_better
       InstanceType: x1.large 

#Use case 6: I need to override a resource property in the template
(for all resources)
properties:
   global:
       username: foo
       pass: thewiggles 
   resources:

Some notes on the above:
- there is no external concept of a provider, we only map resource types
   and templates to official resource types (OS::bla..). This way your
   template always uses "OS::.." and we map cloud specific or custom
   resource types to those.
- the heat client gets the file:// templates and adds them to the
   files section in Zane's proposal.
- the api or engine gets the url based templates
- there is no need to upload the templates ahead of time (but we
   _could_ still support that) - refering to:
https://wiki.openstack.org/wiki/Heat/Providers#How_might_it_work_in_practice.3F
- before creating a resource the engine asks the enviroment what
   is the requested implementation if a resource type.
   env.resource_realization_get('OS::whatever') that will give the
   engine what it actually needs.

-Angus

>
>I'd like to propose in more concrete form an idea that I started 
>kicking around on the mailing list a few weeks back[2], but which 
>didn't engender any discussion at the time.
>
>* Templates could reference external files using a function similar 
>to "Ref", but for files. I'm leaning toward it being called "Url" and 
>it accepting both relative and absolute URLs. (Other options might be 
>"File" or "Include".)
>* Clients (i.e. python-heatclient) could recursively parse templates 
>for {"Url": "..."} references, and ensure all of the necessary files, 
>including e.g. relative file paths, are included in the request. 
>(This is the reason for the function - the client need not know 
>anything else about the template format.)
>* We would take advantage of our APIs being JSON/YAML-only to simply 
>include all of the files inline in a request (either as JSON or as 
>strings). This gives us the advantages of zip files, without the 
>disadvantages (opaqueness to diffs being the major one).
>* Files other than the top-level template would be passed as a 
>dictionary/map keyed by the URL in the "Url" function. The engine, 
>when interpreting a "Url" function would first look in the files map, 
>before attempting to fetch from the URL if necessary and appropriate.
>
>A trivial example of what a ReST API POST request might look like:
>
>{
>    "stack_name": "{stack_name}",
>    "parameters": {
>        "param_name-1": "param_value-1",
>        "param_name-2": "param_value-2"
>    },
>    "timeout_mins": {timeout_mins},
>    "template": {
>        "parameters": {
>            <elided> ...
>        },
>        "resources": {
>            "nested-stack-1": {
>                type: "os::orchestration::stack",
>                template: {"Url": "./substack-1.template"},
>                parameters: {"param_name": {"Ref": "param_name-1"}}
>            },
>            "nested-stack-2": {
>                type: "os::orchestration::stack",
>                template: {"Url": "./substack-2.template"},
>                parameters: {"param_name": {"Ref": "param_name-2"}}
>            }
>        }
>    },
>    "files": {
>         "./substack-1.template": {
>             <elided> ...
>         },
>         "./substack-2.template": {
>             <elided> ...
>         }
>    }
>}
>
>(only the "files" section is new from the API's point of view).
>
>I would really like this to be able to replace the blueprint that I 
>proposed earlier for uploading provider templates and storing them in 
>the engine[3]. I have always feared that that proposal will result in 
>the sort of statefulness that reduces the chances of anybody ever 
>being able to launch the same stack twice to near zero. Whether this 
>can replace that, however, depends very much on what the 
>"Environments" feature ends up looking like, and figuring that out is 
>another discussion[4]. (One thing that would help: an option to 
>specify a different URL Base in the environment for providers.)
>
>Getting back to the topic at hand (remember that?), I believe that 
>handling multiple input files in roughly this fashion, or something 
>equivalent, would obviate the need to do translation of templates at 
>the API level and tilt the calculus in favour of doing it in the 
>engine.
>
>cheers,
>Zane.
>
>
>[1] Sorry.
>[2] http://lists.openstack.org/pipermail/openstack-dev/2013-May/008703.html
>[3] https://blueprints.launchpad.net/heat/+spec/provider-upload
>[4] Angus has suggested an interesting starting point here: 
>https://wiki.openstack.org/wiki/Heat/Environments
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev