Open Stack

On Apr 4, 2014, at 1:56 PM, Zane Bitter <zbitter at redhat.com>
 wrote:

> On 19/02/14 02:48, Clint Byrum wrote:
>> Since picking up Heat and trying to think about how to express clusters
>> of things, I've been troubled by how poorly the CFN language supports
>> using lists. There has always been the Fn::Select function for
>> dereferencing arrays and maps, and recently we added a nice enhancement
>> to HOT to allow referencing these directly in get_attr and get_param.
>> 
>> However, this does not help us when we want to do something with all of
>> the members of a list.
>> 
>> In many applications I suspect the template authors will want to do what
>> we want to do now in TripleO. We have a list of identical servers and
>> we'd like to fetch the same attribute from them all, join it with other
>> attributes, and return that as a string.
>> 
>> The specific case is that we need to have all of the hosts in a cluster
>> of machines addressable in /etc/hosts (please, Designate, save us,
>> eventually. ;). The way to do this if we had just explicit resources
>> named NovaCompute0, NovaCompute1, would be:
>> 
>>   str_join:
>>     - "\n"
>>     - - str_join:
>>         - ' '
>>         - get_attr:
>>           - NovaCompute0
>>           - networks.ctlplane.0
>>         - get_attr:
>>           - NovaCompute0
>>           - name
>>       - str_join:
>>         - ' '
>>         - get_attr:
>>           - NovaCompute1
>>           - networks.ctplane.0
>>         - get_attr:
>>           - NovaCompute1
>>           - name
>> 
>> Now, what I'd really like to do is this:
>> 
>> map:
>>   - str_join:
>>     - "\n"
>>     - - str_join:
>>       - ' '
>>       - get_attr:
>>         - "$1"
>>         - networks.ctlplane.0
>>       - get_attr:
>>         - "$1"
>>         - name
>>   - - NovaCompute0
>>     - NovaCompute1
>> 
>> This would be helpful for the instances of resource groups too, as we
>> can make sure they return a list. The above then becomes:
>> 
>> 
>> map:
>>   - str_join:
>>     - "\n"
>>     - - str_join:
>>       - ' '
>>       - get_attr:
>>         - "$1"
>>         - networks.ctlplane.0
>>       - get_attr:
>>         - "$1"
>>         - name
>>   - get_attr:
>>       - NovaComputeGroup
>>       - member_resources
>> 
>> Thoughts on this idea? I will throw together an implementation soon but
>> wanted to get this idea out there into the hive mind ASAP.
> 
> Apparently I read this at the time, but completely forgot about it. Sorry about that! Since it has come up again in the context of the "TripleO Heat templates and merge.py" thread, allow me to contribute my 2c.
> 
> Without expressing an opinion on this proposal specifically, consensus within the Heat core team has been heavily -1 on any sort of for-each functionality. I'm happy to have the debate again (and TBH I don't really know what the right answer is), but I wouldn't consider the lack of comment on this as a reliable indicator of lazy consensus in favour; equivalent proposals have been considered and rejected on multiple occasions.
> 
> Since it looks like TripleO will soon be able to move over to using AutoscalingGroups (or ResourceGroups, or something) for groups of similar servers, maybe we could consider baking this functionality into Autoscaling groups instead of as an intrinsic function.
> 
> For example, when you do get_attr on an autoscaling resource it could fetch the corresponding attribute from each member of the group and return them as a list. (It might be wise to prepend "Output." or something similar - maybe "Members." - to the attribute names, as AWS::CloudFormation::Stack does, so that attributes of the autoscaling group itself can remain in a separate namespace.)

FWIW, ResourceGroup supports this now as well as getting the attribute value from a given indexed member of the group.

> 
> Since members of your NovaComputeGroup will be nested stacks anyway (using ResourceGroup or some equivalent feature - preferably autoscaling with rolling updates), in the case above you'd define in the scaled template:
> 
>  outputs:
>    hosts_entry:
>      description: An /etc/hosts entry for the NovaComputeServer
>      value:
>        - str_join:
>          - ' '
>          - - get_attr:
>              - NovaComputeServer
>              - networks
>              - ctlplane
>              - 0
>            - get_attr:
>              - NovaComputeServer
>              - name
> 
> And then in the main template (containing the autoscaling group):
> 
>    str_join:
>      - "\n"
>      - get_attr:
>        - NovaComputeGroup
>        - Members.hosts_entry
> 
> would give the same output as your example would.
> 
> IMHO we should do something like this regardless of whether it solves your use case, because it's fairly easy, requires no changes to the template format, and users have been asking for ways to access e.g. a list of IP addresses from a scaling group. That said, it seems very likely that making the other changes required for TripleO to get rid of merge.py (i.e. switching to scaling groups of templates instead of by multiplying resources in templates) will make this a viable solution for TripleO's use case as well.
> 
> cheers,
> Zane.
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev