Open Stack

On 19/02/14 02:48, Clint Byrum wrote:
> Since picking up Heat and trying to think about how to express clusters
> of things, I've been troubled by how poorly the CFN language supports
> using lists. There has always been the Fn::Select function for
> dereferencing arrays and maps, and recently we added a nice enhancement
> to HOT to allow referencing these directly in get_attr and get_param.
>
> However, this does not help us when we want to do something with all of
> the members of a list.
>
> In many applications I suspect the template authors will want to do what
> we want to do now in TripleO. We have a list of identical servers and
> we'd like to fetch the same attribute from them all, join it with other
> attributes, and return that as a string.
>
> The specific case is that we need to have all of the hosts in a cluster
> of machines addressable in /etc/hosts (please, Designate, save us,
> eventually. ;). The way to do this if we had just explicit resources
> named NovaCompute0, NovaCompute1, would be:
>
>    str_join:
>      - "\n"
>      - - str_join:
>          - ' '
>          - get_attr:
>            - NovaCompute0
>            - networks.ctlplane.0
>          - get_attr:
>            - NovaCompute0
>            - name
>        - str_join:
>          - ' '
>          - get_attr:
>            - NovaCompute1
>            - networks.ctplane.0
>          - get_attr:
>            - NovaCompute1
>            - name
>
> Now, what I'd really like to do is this:
>
> map:
>    - str_join:
>      - "\n"
>      - - str_join:
>        - ' '
>        - get_attr:
>          - "$1"
>          - networks.ctlplane.0
>        - get_attr:
>          - "$1"
>          - name
>    - - NovaCompute0
>      - NovaCompute1
>
> This would be helpful for the instances of resource groups too, as we
> can make sure they return a list. The above then becomes:
>
>
> map:
>    - str_join:
>      - "\n"
>      - - str_join:
>        - ' '
>        - get_attr:
>          - "$1"
>          - networks.ctlplane.0
>        - get_attr:
>          - "$1"
>          - name
>    - get_attr:
>        - NovaComputeGroup
>        - member_resources
>
> Thoughts on this idea? I will throw together an implementation soon but
> wanted to get this idea out there into the hive mind ASAP.

Apparently I read this at the time, but completely forgot about it. 
Sorry about that! Since it has come up again in the context of the 
"TripleO Heat templates and merge.py" thread, allow me to contribute my 2c.

Without expressing an opinion on this proposal specifically, consensus 
within the Heat core team has been heavily -1 on any sort of for-each 
functionality. I'm happy to have the debate again (and TBH I don't 
really know what the right answer is), but I wouldn't consider the lack 
of comment on this as a reliable indicator of lazy consensus in favour; 
equivalent proposals have been considered and rejected on multiple 
occasions.

Since it looks like TripleO will soon be able to move over to using 
AutoscalingGroups (or ResourceGroups, or something) for groups of 
similar servers, maybe we could consider baking this functionality into 
Autoscaling groups instead of as an intrinsic function.

For example, when you do get_attr on an autoscaling resource it could 
fetch the corresponding attribute from each member of the group and 
return them as a list. (It might be wise to prepend "Output." or 
something similar - maybe "Members." - to the attribute names, as 
AWS::CloudFormation::Stack does, so that attributes of the autoscaling 
group itself can remain in a separate namespace.)

Since members of your NovaComputeGroup will be nested stacks anyway 
(using ResourceGroup or some equivalent feature - preferably autoscaling 
with rolling updates), in the case above you'd define in the scaled 
template:

   outputs:
     hosts_entry:
       description: An /etc/hosts entry for the NovaComputeServer
       value:
         - str_join:
           - ' '
           - - get_attr:
               - NovaComputeServer
               - networks
               - ctlplane
               - 0
             - get_attr:
               - NovaComputeServer
               - name

And then in the main template (containing the autoscaling group):

     str_join:
       - "\n"
       - get_attr:
         - NovaComputeGroup
         - Members.hosts_entry

would give the same output as your example would.

IMHO we should do something like this regardless of whether it solves 
your use case, because it's fairly easy, requires no changes to the 
template format, and users have been asking for ways to access e.g. a 
list of IP addresses from a scaling group. That said, it seems very 
likely that making the other changes required for TripleO to get rid of 
merge.py (i.e. switching to scaling groups of templates instead of by 
multiplying resources in templates) will make this a viable solution for 
TripleO's use case as well.

cheers,
Zane.