Open Stack

Hi all,

I have read thru all the good discussion around software orchestration and
what to share my view. I see that there are two threads - this one and [1]
- but I am just replying to the one with the more generic topic line.

So on the recent twist to have software components implemented as
resources, I see some pros and cons. I am not actually decided yet what I
like most, so just shedding some thoughts here.
The good thing it would bring is definitely that it would give us a lot of
the things that Lakshmi has asked for in [1] (e.g. feed in properties,
get_attr, state) for free, and it would keep the HOT language smaller.

One of the big differences between the way resources are used today, and
what components are meant to be is that components had semantics of being
declared once and potentially used in many places in a template. Whereas a
resource would result in one instance at runtime. I think this is a key
issue that needs to be addressed.

One possibility would be the "SoftwareComponent - Deployment - Server"
resource triple that Zane brought up on the other thread. Or the original
component concept ...

With the latter one, the other open issue is how the binding gets defined,
and there are some debates about it. In summary, I think the following
option have been outlined:

(1) a components section with the server
(2) a hosted_on link with the component
(3) some explicit binding object

(1) seems doable, but has state dependency impacts that Steve Baker lined
out in a recent post. (2) would solve the state dependency problem, but has
the problem that the re-usable thing (the component) points explicitly to
its host. (3) sounds like Zane's "SoftwareComponent - Deployment - Server"
idea.

At the moment I am inclined to (3) because it actually makes the
distinction between a component declaration and its use ("Deployment") very
clear. And I think this would have to be done in any case - whether
software components are resources or a new concept.
Originally, I actually brought up hosted_on in an own proposal because I am
coming from the TOSCA world where we have a HostedOn relationship type. The
difference, though, is that in TOSCA we have relationships as own objects
so we avoid "hardcoding" the host in the software component object but
having association objects instead. Again, this seems to go into the
direction of option (3).

BTW: +10 on Angus' request for big whiteboards at the summit!

[1]
http://lists.openstack.org/pipermail/openstack-dev/2013-October/017540.html

Regards,
Thomas

Angus Salkeld <asalkeld at redhat.com> wrote on 29.10.2013 02:22:22:
> From: Angus Salkeld <asalkeld at redhat.com>
> To: openstack-dev at lists.openstack.org,
> Date: 29.10.2013 02:25
> Subject: Re: [openstack-dev] [Heat] HOT Software configuration proposal
>
> On 28/10/13 16:26 -0700, Clint Byrum wrote:
>
> [snip]
>
> >> >
> >> My components proposals had no hosted_on, but I've been thinking about
> >> the implications of implementing software configuration as resources,
> >> and one of the natural consequences might be that hosted_on is the
best
> >> way of establishing the relationship with compute and its
> >> configurations. Let me elaborate.
> >>
> >> Lets say that Heat has resource types for software configuration, with
> >> the following behaviours:
> >> * like other resources, a config resource goes into CREATE IN_PROGRESS
> >> as soon as its dependencies are satisfied (these dependencies may be
> >> values signalled from other config resources)
> >> * a config resource goes to state CREATE COMPLETE when it receives a
> >> signal that configuration on the compute resource is complete (by some
> >> mechanism; wait condition, marconi message, whatevs)
> >> * the relationship between config resources and compute resources are
> >> achieved with existing intrinsic functions (get_resource, get_attr)
> >>
> >> This lifecycle behaviour means that a configuration resource can only
> >> run on a single compute resource, and that relationship needs to be
> >> established somehow. Config resources will have a quirk in that they
> >> need to provide 2 sources of configuration data at different times:
> >> 1) cloud-init config-data (or other boot-only mechanism), which must
be
> >> available when the compute resource goes into CREATE IN_PROGRESS
> >> 2) the actual configuration data (oac metadata, puppet manifest, chef
> >> recipe) which the compute resource needs to be able to fetch and
execute
> >> whenever it becomes available.
> >>
> >> The data in 1) implies that the compute needs to depend on the config,
> >> but then all concurrency is lost (this won't matter for a
> >> cloud-init-only config resource).  Either way, the data for 1) will
need
> >> to be available when the config resource is still in state INIT
> >> COMPLETE, which may impose limitations on how that is defined (ie
> >> get_resource, get_attr not allowed).
> >>
> >> So, 2 concrete examples for handling config/compute dependencies:
> >>
> >> hosted_on
> >> ---------
> >> resources:
> >>   configA
> >>     type: Heat::ScriptConfig
> >>     properties:
> >>        hosted_on:
> >>          get_resource: the_server
> >>        script: |
> >>          #!/bin/bash
> >>          logger "1. hello config A"
> >>   configB
> >>     type: Heat::ScriptConfig
> >>     properties:
> >>        hosted_on:
> >>          get_resource: the_server
> >>        script: |
> >>          #!/bin/bash
> >>          logger "1. hello config B"
> >>   the_server:
> >>     type: OS::Nova::Server
> >>
> >> Here, configA and configB go into CREATE IN_PROGRESS as soon as
> >> the_server is CREATE COMPLETE. configA and configB go into CREATE
> >> COMPLETE when heat-engine receives a signal that they are complete.
This
> >> signal may include values that other resources depend on to start
their
> >> creation.
> >>
> >
> >My biggest objection to hosted_on is that if I want to reuse configA.. I
> >can't without an additional level of abstraction. This feels awkward,
> >and will be an awkward thing for users. In addition, it is very unclear
> >what what will actually be on "the_server" when looking at "the_server".
> >
> >Nobody has answered my question where else this sort of pattern is used.
> >
> >> OS::Nova::Server config_resources
> >> ---------------------------------
> >> resources:
> >>   configA
> >>     type: Heat::ScriptConfig
> >>     properties:
> >>        script: |
> >>          #!/bin/bash
> >>          logger "1. hello config A"
> >>   configB
> >>     type: Heat::ScriptConfig
> >>     properties:
> >>        script: |
> >>          #!/bin/bash
> >>          logger "1. hello config B"
> >>   the_server:
> >>     type: OS::Nova::Server
> >>     properties:
> >>       config_resources:
> >>         - {get_resource: configA}
> >>         - {get_resource: configB}
> >>
> >> Here there would need to be some bypassing of dependency calculation
to
> >> allow configA and configB to be created after the_server, maybe by:
> >> * special treatment of config_resources to prevent dependencies being
> >> created
> >> * a get_resource variant which doesn't create a hard dependency
> >> (get_resource_deferred?)
> >>
> >
> >IMO no, these are not best expressed as resources in the same way that
> >python classes are not expressed as objects.
> >
> >The template above reads more like code. If used as a component, configA
> >remains reusable throughout the rest of the template as it grows and
> >changes, and it is very obvious what is expected on "the_server". Each
> >server using configA gets its very own completion waitcondition for that
> >instantiation of the component.
> >
> >This is why I liked the components concept, rather than have them as
> >resources. It seems clear to me that components are parse-time objects,
> >so if I refer to the same component 4 times, I really want 4 copies of
> >it. Resources, on the other hand, are real things that are instantiated.
> >
> >> Neither the hosted_on nor the config_resources behaviours are ideal,
but
> >> I'm leaning towards hosted_on at the moment since it doesn't require
any
> >> new soft-dependency mechanism.
> >>
> >
> >IMO hosted_on is a new dependency mechanism, so avoiding the more clear
> >paradigm for the reason of not introducing dependency mechanism seems a
> >bit backwards too.
> >
> >> As for composability, what *actually* needs to be composable is the
> >> contents of the script (manifest, recipe) property. Everything else in
a
> >> config resource is just stack-specific wiring.  There are a couple of
> >> ways this composability could be achieved:
> >> 1) resource provider in the user environment which specifies the
script
> >> property
> >> 2) __include__ or some equivalent client-side inclusion mechanism
> >>
> >
> >At first it seems like all of it is just per-stack wiring, but the point
> >is that even just inside a stack, you will want to reuse whole
components,
> >not just the script/cloud-config/etc.
> >
> >I've done a client side __include__ method for TripleO. It works fine
> >actually. But since we're developing a language, we need to think
> >longer and harder than "works for me", otherwise we will make the PHP
> >of orchestration languages.
>
> Zane had an idea for a "FileRef" of some kind that I nealy had time
> to implement. This would work is a simerlar way to the way template
> resources are uploaded into the "files" structure.
>
> >
> >I'd like to see components as a first class facility in HOT so that
users
> >can compose templates in a logical straight forward manner. The
proposals
> >that you have laid out provide workable but less discoverable
mechanisms.
>
> At the moment we have something that alters a server in a somewhat
> simlar way to configuring - volume attaching.
>
> resources:
>    server:
>      bla...
>    volume:
>      bla...
>    volume_attachment:
>      volume_id: {Ref: volume}
>      server_id: {Ref: server}
>
>
> We could model config install in a simerlar way
>
> resources:
>    server:
>      bla...
>    swift_file:
>      bla...
>    config_installer:
>      config_url: {Ref: swift_file}
>      server_id: {Ref: server}
>
> This could be shortened to a url:
>
> resources:
>    server:
>      bla...
>    config_installer:
>      server_id: {Ref: server}
>      config_content:
>        str_replace:
>          template: {get_file: config_url}
>          params:
>            foo: whatever
>
> - you could reuse the config in multiple places
> - the installer could pass variables into config
>    (and potentially get outputs out for dynamic attributes)
>
>
> [I hope we get big whiteboards at summit :-) ]
>
> -Angus
>
> >_______________________________________________
> >OpenStack-dev mailing list
> >OpenStack-dev at lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>