Open Stack

On 14/08/14 00:00, Tomas Sedovic wrote:
> On 12/08/14 01:06, Steve Baker wrote:
>> On 09/08/14 11:15, Zane Bitter wrote:
>>> On 08/08/14 11:07, Tomas Sedovic wrote:
>>>> On 08/08/14 00:53, Zane Bitter wrote:
>>>>> On 07/08/14 13:22, Tomas Sedovic wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I have a ResourceGroup which wraps a custom resource defined in
>>>>>> another
>>>>>> template:
>>>>>>
>>>>>>       servers:
>>>>>>         type: OS::Heat::ResourceGroup
>>>>>>         properties:
>>>>>>           count: 10
>>>>>>           resource_def:
>>>>>>             type: my_custom_server
>>>>>>             properties:
>>>>>>               prop_1: "..."
>>>>>>               prop_2: "..."
>>>>>>               ...
>>>>>>
>>>>>> And a corresponding provider template and environment file.
>>>>>>
>>>>>> Now I can get say the list of IP addresses or any custom value of each
>>>>>> server from the ResourceGroup by using `{get_attr: [servers,
>>>>>> ip_address]}` and outputs defined in the provider template.
>>>>>>
>>>>>> But I can't figure out how to pass that list back to each server in
>>>>>> the
>>>>>> group.
>>>>>>
>>>>>> This is something we use in TripleO for things like building a MySQL
>>>>>> cluster, where each node in the cluster (the ResourceGroup) needs the
>>>>>> addresses of all the other nodes.
>>>>> Yeah, this is kind of the perpetual problem with clusters. I've been
>>>>> hoping that DNSaaS will show up in OpenStack soon and that that will be
>>>>> a way to fix this issue.
>>>>>
>>>>> The other option is to have the cluster members discover each other
>>>>> somehow (mDNS?), but people seem loath to do that.
>>>>>
>>>>>> Right now, we have the servers ungrouped in the top-level template
>>>>>> so we
>>>>>> can build this list manually. But if we move to ResourceGroups (or any
>>>>>> other scaling mechanism, I think), this is no longer possible.
>>>>> So I believe the current solution is to abuse a Launch Config resource
>>>>> as a store for the data, and then later retrieve it somehow? Possibly
>>>>> you could do something along similar lines, but it's unclear how the
>>>>> 'later retrieval' part would work... presumably it would have to
>>>>> involve
>>>>> something outside of Heat closing the loop :(
>>>> Do you mean AWS::AutoScaling::LaunchConfiguration? I'm having trouble
>>>> figuring out how would that work. LaunchConfig represents an instance,
>>>> right?
>>>>
>>>>>> We can't pass the list to ResourceGroup's `resource_def` section
>>>>>> because
>>>>>> that causes a circular dependency.
>>>>>>
>>>>>> And I'm not aware of a way to attach a SoftwareConfig to a
>>>>>> ResourceGroup. SoftwareDeployment only allows attaching a config to a
>>>>>> single server.
>>>>> Yeah, and that would be a tricky thing to implement well, because a
>>>>> resource group may not be a group of servers (but in many cases it may
>>>>> be a group of nested stacks that each contain one or more servers, and
>>>>> you'd want to be able to handle that too).
>>>> Yeah, I worried about that, too :-(.
>>>>
>>>> Here's a proposal that might actually work, though:
>>>>
>>>> The provider resource exposes the reference to its inner instance by
>>>> declaring it as one of its outputs. A SoftwareDeployment would learn to
>>>> accept a list of Nova servers, too.
>>>>
>>>> Provider template:
>>>>
>>>>      resources:
>>>>        my_server:
>>>>          type: OS::Nova::Server
>>>>          properties:
>>>>            ...
>>>>
>>>>        ... (some other resource hidden in the provider template)
>>>>
>>>>      outputs:
>>>>        inner_server:
>>>>          value: {get_resource: my_server}
>>>>        ip_address:
>>>>          value: {get_attr: [controller_server, networks, private, 0]}
>>>>
>>>> Based on my limited testing, this already makes it possible to use the
>>>> inner server with a SoftwareDeployment from another template that uses
>>>> "my_server" as a provider resource.
>>>>
>>>> E.g.:
>>>>
>>>>      a_cluster_of_my_servers:
>>>>        type: OS::Heat::ResourceGroup
>>>>        properties:
>>>>          count: 10
>>>>          resource_def:
>>>>            type: custom::my_server
>>>>            ...
>>>>
>>>>      some_deploy:
>>>>        type: OS::Heat::StructuredDeployment
>>>>        properties:
>>>>          server: {get_attr: [a_cluster_of_my_servers,
>>>> resource.0.inner_server]}
>>>>          config: {get_resource: some_config}
>>>>
>>>>
>>>> So what if we allowed SoftwareDeployment to accept a list of servers in
>>>> addition to accepting just one server? Or add another resource that does
>>>> that.
>>> I approve of that in principle. Only Steve Baker can tell us for sure
>>> if there are any technical roadblocks in the way of that, but I don't
>>> see any.
>>>
>>> Maybe if we had a new resource type that was internally implemented as
>>> a nested stack... that might give us a way of tracking the individual
>>> deployment statuses for free.
>>>
>>> cheers,
>>> Zane.
>>>
>>>> Then we could do:
>>>>
>>>>      mysql_cluster_deployment:
>>>>        type: OS::Heat::StructuredDeployment
>>>>        properties:
>>>>          server_list: {get_attr: [a_cluster_of_my_servers,
>>>> inner_server]}
>>>>          config: {get_resource: mysql_cluster_config}
>>>>          input_values:
>>>>            cluster_ip_addresses: {get_attr: [a_cluster_of_my_servers,
>>>> ip_address]}
>>>>
>>>> This isn't that different from having a SoftwareDeployment accepting a
>>>> single server and doesn't have any of the problems of allowing a
>>>> ResourceGroup as a SoftwareDeployment target.
>>>>
>>>> What do you think?
>> All the other solutions I can think of will result in circular issues.
>>
>> I'll start looking at a spec to create a resource which is capable of
>> deploying a config to a list of servers.
> Thanks. If there's any way I could help with this, let me know. I'll be
> happy to.
>
I have a full series ready for review [1] and a simple template which
demonstrates deploying cluster IP addresses to the group [2]

The series adds 2 new resources, OS::Heat::SoftwareDeployments and
OS::Heat::StructuredDeployments which extends a (somewhat refactored)
ResourceGroup.

I've successfully used stack-update to scale the template[2] up and down
and the resulting IP list has happily propagated to the servers.

[1]
https://review.openstack.org/#/q/status:open+topic:bp/deployment-multiple-servers,n,z
[2] http://paste.openstack.org/show/101278/