Open Stack

Hi all,

This is a follow-up to Clint Byrum's suggestion to add the `Map`
intrinsic function[0], Zane Bitter's response[1] and Randall Burt's
addendum[2].

Sorry for bringing it up again, but I'd love to reach consensus on this.
The summary of the previous conversation:

1. TripleO is using some functionality currently not supported by Heat
around scaled-out resources
2. Clint proposed a `map` intrinsic function that would solve it
3. Zane said Heat have historically been against a for-loop functionality
4. Randall suggested ResourceGroup's attribute passthrough may do what
we need

I've looked at the ResourceGroup code and experimented a bit. It does do
some of what TripleO needs but not all.

Here's what we're doing with our scaled-out resources (what we'd like to
wrap in a ResourceGroup or similar in the future):


1. Building a coma-separated list of RabbitMQ nodes:

https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/overcloud-source.yaml#L642

This one is easy with ResourceGroup's inner attribute support:

    list_join:
    - ", "
    - {get_attr: [controller_group, name]}

(controller_group is a ResourceGroup of Nova servers)


2. Get the name of the first Controller node:

https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/overcloud-source.yaml#L339

Possible today:

    {get_attr: [controller_group, resource.0.name]}


3. List of IP addresses of all controllers:

https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/overcloud-source.yaml#L405

We cannot do this, because resource group doesn't support extended
attributes.

Would need something like:

    {get_attr: [controller_group, networks, ctlplane, 0]}

(ctlplane is the network controller_group servers are on)


4. IP address of the first node in the resource group:

https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/swift-deploy.yaml#L29

Can't do: extended attributes are not supported for the n-th node for
the group either.

This can be solved by `get_resource` working with resource IDs:

   get_attr:
   - {get_attr: [controller_group, resource.0]}
   - [networks, ctlplane, 0]

(i.e. we get the server's ID from the ResourceGroup and change
`get_attr` to work with the ID's too. Would also work if `get_resource`
understood IDs).


Alternatively, we could extend the ResourceGroup's get_attr behaviour:

    {get_attr: [controller_group, resource.0.networks.ctlplane.0]}

but the former is a bit cleaner and more generic.


---


That was the easy stuff, where we can get by with the current
functionality (plus a few fixes).

What follows are examples that really need new intrinsic functions (or
seriously complicating the ResourceGroup attribute code and syntax).


5. Building a list of {ip: ..., name: ...} dictionaries to configure
haproxy:

https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/overcloud-source.yaml#L478

This really calls for a mapping/for-each kind of functionality. Trying
to invent a ResourceGroup syntax for this would be perverse.

Here's what it could look like under Clint's `map` proposal:

    map:
    - ip: {get_attr: [{get_resource: "$1"}, networks, ctlplane, 0]
      name: {get_attr: [{get_resource: "$1"}, name]}
    - {get_attr: [compute_group, refs]}

(this relies on `get_resource` working with resource IDs. Alternatively,
we could have a `resources` attribute for ResourceGroup that returns
objects that can be used with get_attr.


6. Building the /etc/hosts file

https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/overcloud-source.yaml#L585

Same as above, but also joining two lists together.

We can use nested {list_join: ["\n", [...]} just as we're doing now, but
having a `concat_list` function would make this and some other cases
shorter and clearer.


7. Building the list of Swift devices:

https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/swift-deploy.yaml#L23

In addition to the abowe, we're adding a single element at the beginning
of a list.

Asking for a `cons` support is pushing it, right? ;-)

We could just wrap that in a list and use `concat_list` or keep using
nested `list_join`s as in the /etc/hosts case.


----


So this boils down to 4 features proposals:

1. Support extended attributes in ResourceGroup's members
2. Allow a way to use a Resource ID (e.g. what you get by {get_attr:
[ResourceGroup, refs]} or {get_attr: [ResourceGroup, resource.0]}) with
existing intrinsic functions (get_resource, get_attr)
3. A `map` intrinsic function that turns a list of items to another list
by doing operations on each item
4. A `concat_list` intrinsic function that joins multiple lists into one.

I think the first two are not controversial. What about the other two?
I've shown you some examples where we would find a good use in the
TripleO templates. The lack of `map` actually blocks us from going all-Heat.

The alternative would be to say that this sort of stuff to be done
inside the instance by os-apply-config et al. It would complicate things
for TripleO, but oh well.

Either way, can we come to a clear answer? I would love to convert our
templates to 100% Natural Heat during Juno and I'm willing to put my
time towards coding the necessary features.

Tomas


[0]:
http://lists.openstack.org/pipermail/openstack-dev/2014-February/027536.html
[1]:
http://lists.openstack.org/pipermail/openstack-dev/2014-April/031944.html
[2]:
http://lists.openstack.org/pipermail/openstack-dev/2014-April/032074.html