[openstack-dev] [Heat] HOT Software configuration proposal

Angus Salkeld asalkeld at redhat.com
Fri Oct 25 01:48:16 UTC 2013


On 24/10/13 11:54 +0200, Patrick Petit wrote:
>Hi Clint,
>Thank you! I have few replies/questions in-line.
>Cheers,
>Patrick
>On 10/23/13 8:36 PM, Clint Byrum wrote:
>>Excerpts from Patrick Petit's message of 2013-10-23 10:58:22 -0700:
>>>Dear Steve and All,
>>>
>>>If I may add up on this already busy thread to share our experience with
>>>using Heat in large and complex software deployments.
>>>
>>Thanks for sharing Patrick, I have a few replies in-line.
>>
>>>I work on a project which precisely provides additional value at the
>>>articulation point between resource orchestration automation and
>>>configuration management. We rely on Heat and chef-solo respectively for
>>>these base management functions. On top of this, we have developed an
>>>event-driven workflow to manage the life-cycles of complex software
>>>stacks which primary purpose is to support middleware components as
>>>opposed to end-user apps. Our use cases are peculiar in the sense that
>>>software setup (install, config, contextualization) is not a one-time
>>>operation issue but a continuous thing that can happen any time in
>>>life-span of a stack. Users can deploy (and undeploy) apps long time
>>>after the stack is created. Auto-scaling may also result in an
>>>asynchronous apps deployment. More about this latter. The framework we
>>>have designed works well for us. It clearly refers to a PaaS-like
>>>environment which I understand is not the topic of the HOT software
>>>configuration proposal(s) and that's absolutely fine with us. However,
>>>the question for us is whether the separation of software config from
>>>resources would make our life easier or not. I think the answer is
>>>definitely yes but at the condition that the DSL extension preserves
>>>almost everything from the expressiveness of the resource element. In
>>>practice, I think that a strict separation between resource and
>>>component will be hard to achieve because we'll always need a little bit
>>>of application's specific in the resources. Take for example the case of
>>>the SecurityGroups. The ports open in a SecurityGroup are application
>>>specific.
>>>
>>Components can only be made up of the things that are common to all users
>>of said component. Also components would, if I understand the concept
>>correctly, just be for things that are at the sub-resource level.
>That isn't cop
>>Security groups and open ports would be across multiple resources, and
>>thus would be separately specified from your app's component (though it
>>might be useful to allow components to export static values so that the
>>port list can be referred to along with the app component).
>>
>>>Then, designing a Chef or Puppet component type may be harder than it
>>>looks at first glance. Speaking of our use cases we still need a little
>>>bit of scripting in the instance's user-data block to setup a working
>>>chef-solo environment. For example, we run librarian-chef prior to
>>>starting chef-solo to resolve the cookbook dependencies. A cookbook can
>>>present itself as a downloadable tarball but it's not always the case. A
>>>chef component type would have to support getting a cookbook from a
>>>public or private git repo (maybe subversion), handle situations where
>>>there is one cookbook per repo or multiple cookbooks per repo, let the
>>>user choose a particular branch or label, provide ssh keys if it's a
>>>private repo, and so forth. We support all of this scenarios and so we
>>>can provide more detailed requirements if needed.
>>>
>>Correct me if I'm wrong though, all of those scenarios are just variations
>>on standard inputs into chef. So the chef component really just has to
>>allow a way to feed data to chef.
>
>>
>>>I am not sure adding component relations like the 'depends-on' would
>>>really help us since it is the job of config management to handle
>>>software dependencies. Also, it doesn't address the issue of circular
>>>dependencies. Circular dependencies occur in complex software stack
>>>deployments. Example. When we setup a Slum virtual cluster, both the
>>>head node and compute nodes depend on one another to complete their
>>>configuration and so they would wait for each other indefinitely if we
>>>were to rely on the 'depends-on'. In addition, I think it's critical to
>>>distinguish between configuration parameters which are known ahead of
>>>time, like a db name or user name and password, versus contextualization
>>>parameters which are known after the fact generally when the instance is
>>>created. Typically those contextualization parameters are IP addresses
>>>but not only. The fact packages x,y,z have been properly installed and
>>>services a,b,c successfully started is contextualization information
>>>(a.k.a facts) which may be indicative that other components can move on
>>>to the next setup stage.
>>>
>>The form of contextualization you mention above can be handled by a
>>slightly more capable wait condition mechanism than we have now. I've
>>been suggesting that this is the interface that workflow systems should
>>use.
>>
>>>The case of complex deployments with or without circular dependencies is
>>>typically resolved by making the system converge toward the desirable
>>>end-state through running idempotent recipes. This is our approach. The
>>>first configuration phase handles parametrization which in general
>>>brings an instance to CREATE_COMPLETE state. A second phase follows to
>>>handle contextualization at the stack level. As a matter of fact, a new
>>>contextualization should be triggered every time an instance enters or
>>>leave the CREATE_COMPLETE state which may happen any time with
>>>auto-scaling. In that phase, circular dependencies can be resolved
>>>because all contextualization data can be compiled globally. Notice that
>>>Heat doesn't provide a purpose built resource or service like Chef's
>>>data-bag for the storage and retrieval of metadata. This a gap which IMO
>>>should be addressed in the proposal. Currently, we use a kludge that is
>>>to create a fake AWS::AutoScaling::LaunchConfiguration resource to store
>>>contextualization data in the metadata section of that resource.
>>>
>>That is what we use in TripleO as well:
>>
>>http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud-source.yaml#n143
>>
>>We are not doing any updating of that from within our servers though.
>>That is an interesting further use of the capability.
>Right. The problem with that is... that it's a kludge ;-)  Obscure 
>the readability of the code because used for an unintended purpose.
>>>Aside from the HOT software configuration proposal(s). There are two
>>>critical enhancements in Heat that would make software life-cycles
>>>management much easier. In fact, they are actual blockers for us.
>>>
>>>The first one would be to support asynchronous notifications when an
>>>instance is created or deleted as a result of an auto-scaling decision.
>>>As stated earlier, contextualization needs to apply in a stack every
>>>time a instance enters or leaves the CREATE_COMPLETE state. I am not
>>>referring to a Ceilometer notification but a Heat notification that can
>>>be consumed by a Heat client.
>>>
>>I think this fits into something that I want for optimizing
>>os-collect-config as well (our in-instance Heat-aware agent). That is
>>a way for us to wait for notification of changes to Metadata without
>>polling.
>Interesting... If I understand correctly that's kinda replacement of 
>cfn-hup... Do you have a blueprint pointer or something more 
>specific? While I see the benefits of it, in-instance notifications 
>is not really what we are looking for. We are looking for a 
>notification service that exposes an API whereby listeners can 
>register for Heat notifications. AWS Alarming / CloudFormation has 
>that. Why not Ceilometer / Heat? That would be extremely valuable for 
>those who build PaaS-like solutions above Heat. To say it bluntly, 
>I'd like to suggest we explore ways to integrate Heat with Marconi.

Yeah, I am trying to do a PoC of this now. I'll let you know how
it goes.

I am trying to implement the following:

heat_template_version: 2013-05-23
parameters:
   key_name:
     type: String
   flavor:
     type: String
     default: m1.small
   image:
     type: String
     default: fedora-19-i386-heat-cfntools
resources:
   config_server:
     type: OS::Marconi::QueueServer
     properties:
       image: {get_param: image}
       flavor: {get_param: flavor}
       key_name: {get_param: key_name}

   configA:
     type: OS::Heat::OrderedConfig
     properties:
       marconi_server: {get_attr: [config_server, url]}
       hosted_on: {get_resource: serv1}
       script: |
         #!/bin/bash
         logger "1. hello from marconi"

   configB:
     type: OS::Heat::OrderedConfig
     properties:
       marconi_server: {get_attr: [config_server, url]}
       hosted_on: {get_resource: serv1}
       depends_on: {get_resource: configA}
       script: |
         #!/bin/bash
         logger "2. hello from marconi"

   serv1:
     type: OS::Nova::Server
     properties:
       image: {get_param: image}
       flavor: {get_param: flavor}
       key_name: {get_param: key_name}
       user_data: |
         #!/bin/sh
         # poll <marconi url>/v1/queues/{hostname}/messages
         # apply config
         # post a response message with any outputs
         # delete request message

The idea here is that each "os::heat::orderedConfig" does the
following:
create the following marconi queues:
<server id>.{request,respones}
sends the config to the request queue
the vm then posts responses to the response queue

- each vm/server has a "config" queue (named based on "hosted_on")
- we can get attributes/outputs from the vm
- we can depend_on other config resources
- you can monitor progress externally via marconi
- you can have properties other than "script" (puppet/chef/..)
- you could/should have a marconi server running in you infrastructure
   (I am doing it in a vm for ease of testing).

Is this the kind of thing you are after?
  
-Angus

>>
>>>The second one would be to support a new type of AWS::IAM::User (perhaps
>>>OS::IAM::User) resource whereby one could pass Keystone credentials to
>>>be able to specify Ceilometer alarms based on application's specific
>>>metrics (a.k.a KPIs).
>>>
>>It would likely be OS::Keystone::User, and AFAIK this is on the list of
>>de-AWS-ification things.
>Great! As I said. It's a blocker for us and really would like to see 
>it accepted for icehouse.
>>
>>>I hope this is making sense to you and can serve as a basis for further
>>>discussions and refinements.
>>>
>>Really great feedback Patrick, thanks again for sharing!
>>
>>_______________________________________________
>>OpenStack-dev mailing list
>>OpenStack-dev at lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>-- 
>Patrick Petit
>Cloud Computing Principal Architect, Innovative Products
>Bull, Architect of an Open World TM
>Tél : +33 (0)4 76 29 70 31
>Mobile : +33 (0)6 85 22 06 39
>http://www.bull.com
>
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list