Open Stack

Sorry, I clicked the 'send' button too quickly.

On 10/24/13 11:54 AM, Patrick Petit wrote:
> Hi Clint,
> Thank you! I have few replies/questions in-line.
> Cheers,
> Patrick
> On 10/23/13 8:36 PM, Clint Byrum wrote:
>> Excerpts from Patrick Petit's message of 2013-10-23 10:58:22 -0700:
>>> Dear Steve and All,
>>>
>>> If I may add up on this already busy thread to share our experience 
>>> with
>>> using Heat in large and complex software deployments.
>>>
>> Thanks for sharing Patrick, I have a few replies in-line.
>>
>>> I work on a project which precisely provides additional value at the
>>> articulation point between resource orchestration automation and
>>> configuration management. We rely on Heat and chef-solo respectively 
>>> for
>>> these base management functions. On top of this, we have developed an
>>> event-driven workflow to manage the life-cycles of complex software
>>> stacks which primary purpose is to support middleware components as
>>> opposed to end-user apps. Our use cases are peculiar in the sense that
>>> software setup (install, config, contextualization) is not a one-time
>>> operation issue but a continuous thing that can happen any time in
>>> life-span of a stack. Users can deploy (and undeploy) apps long time
>>> after the stack is created. Auto-scaling may also result in an
>>> asynchronous apps deployment. More about this latter. The framework we
>>> have designed works well for us. It clearly refers to a PaaS-like
>>> environment which I understand is not the topic of the HOT software
>>> configuration proposal(s) and that's absolutely fine with us. However,
>>> the question for us is whether the separation of software config from
>>> resources would make our life easier or not. I think the answer is
>>> definitely yes but at the condition that the DSL extension preserves
>>> almost everything from the expressiveness of the resource element. In
>>> practice, I think that a strict separation between resource and
>>> component will be hard to achieve because we'll always need a little 
>>> bit
>>> of application's specific in the resources. Take for example the 
>>> case of
>>> the SecurityGroups. The ports open in a SecurityGroup are application
>>> specific.
>>>
>> Components can only be made up of the things that are common to all 
>> users
>> of said component. Also components would, if I understand the concept
>> correctly, just be for things that are at the sub-resource level.
>> Security groups and open ports would be across multiple resources, and
>> thus would be separately specified from your app's component (though it
>> might be useful to allow components to export static values so that the
>> port list can be referred to along with the app component).
Okay got it. If that's the case then that would work....
>>
>>> Then, designing a Chef or Puppet component type may be harder than it
>>> looks at first glance. Speaking of our use cases we still need a little
>>> bit of scripting in the instance's user-data block to setup a working
>>> chef-solo environment. For example, we run librarian-chef prior to
>>> starting chef-solo to resolve the cookbook dependencies. A cookbook can
>>> present itself as a downloadable tarball but it's not always the 
>>> case. A
>>> chef component type would have to support getting a cookbook from a
>>> public or private git repo (maybe subversion), handle situations where
>>> there is one cookbook per repo or multiple cookbooks per repo, let the
>>> user choose a particular branch or label, provide ssh keys if it's a
>>> private repo, and so forth. We support all of this scenarios and so we
>>> can provide more detailed requirements if needed.
>>>
>> Correct me if I'm wrong though, all of those scenarios are just 
>> variations
>> on standard inputs into chef. So the chef component really just has to
>> allow a way to feed data to chef.
>
That's correct. Boils down to specifying correctly all the constraints 
that apply to deploying a cookbook in an instance from it's component 
description.
>>
>>> I am not sure adding component relations like the 'depends-on' would
>>> really help us since it is the job of config management to handle
>>> software dependencies. Also, it doesn't address the issue of circular
>>> dependencies. Circular dependencies occur in complex software stack
>>> deployments. Example. When we setup a Slum virtual cluster, both the
>>> head node and compute nodes depend on one another to complete their
>>> configuration and so they would wait for each other indefinitely if we
>>> were to rely on the 'depends-on'. In addition, I think it's critical to
>>> distinguish between configuration parameters which are known ahead of
>>> time, like a db name or user name and password, versus 
>>> contextualization
>>> parameters which are known after the fact generally when the 
>>> instance is
>>> created. Typically those contextualization parameters are IP addresses
>>> but not only. The fact packages x,y,z have been properly installed and
>>> services a,b,c successfully started is contextualization information
>>> (a.k.a facts) which may be indicative that other components can move on
>>> to the next setup stage.
>>>
>> The form of contextualization you mention above can be handled by a
>> slightly more capable wait condition mechanism than we have now. I've
>> been suggesting that this is the interface that workflow systems should
>> use.
Okay. I am looking forward to see what a more capable wait condition 
framework would look like.
The risk though is that by wanting to do too much in the boot sequence 
is that we end-up with a spaghetti plate of wait condition relationships 
wired in the template which would make it hard to read and make the 
workflow hard to debug.
>>
>>> The case of complex deployments with or without circular 
>>> dependencies is
>>> typically resolved by making the system converge toward the desirable
>>> end-state through running idempotent recipes. This is our approach. The
>>> first configuration phase handles parametrization which in general
>>> brings an instance to CREATE_COMPLETE state. A second phase follows to
>>> handle contextualization at the stack level. As a matter of fact, a new
>>> contextualization should be triggered every time an instance enters or
>>> leave the CREATE_COMPLETE state which may happen any time with
>>> auto-scaling. In that phase, circular dependencies can be resolved
>>> because all contextualization data can be compiled globally. Notice 
>>> that
>>> Heat doesn't provide a purpose built resource or service like Chef's
>>> data-bag for the storage and retrieval of metadata. This a gap which 
>>> IMO
>>> should be addressed in the proposal. Currently, we use a kludge that is
>>> to create a fake AWS::AutoScaling::LaunchConfiguration resource to 
>>> store
>>> contextualization data in the metadata section of that resource.
>>>
>> That is what we use in TripleO as well:
>>
>> http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud-source.yaml#n143 
>>
>>
>> We are not doing any updating of that from within our servers though.
>> That is an interesting further use of the capability.
> Right. The problem with that is... that currently it's a kludge ;-)  
> Obscure the readability of the code because used for an unintended 
> purpose.
>>> Aside from the HOT software configuration proposal(s). There are two
>>> critical enhancements in Heat that would make software life-cycles
>>> management much easier. In fact, they are actual blockers for us.
>>>
>>> The first one would be to support asynchronous notifications when an
>>> instance is created or deleted as a result of an auto-scaling decision.
>>> As stated earlier, contextualization needs to apply in a stack every
>>> time a instance enters or leaves the CREATE_COMPLETE state. I am not
>>> referring to a Ceilometer notification but a Heat notification that can
>>> be consumed by a Heat client.
>>>
>> I think this fits into something that I want for optimizing
>> os-collect-config as well (our in-instance Heat-aware agent). That is
>> a way for us to wait for notification of changes to Metadata without
>> polling.
> Interesting... If I understand correctly that's kinda replacement of 
> cfn-hup... Do you have a blueprint pointer or something more specific? 
> While I see the benefits of it, in-instance notifications is not 
> really what we are looking for. We are looking for a notification 
> service that exposes an API whereby listeners can register for Heat 
> notifications. AWS Alarming / CloudFormation has that. Why not 
> Ceilometer / Heat? That would be extremely valuable for those who 
> build PaaS-like solutions above Heat. To say it bluntly, I'd like to 
> suggest we explore ways to integrate Heat with Marconi.
>>
>>> The second one would be to support a new type of AWS::IAM::User 
>>> (perhaps
>>> OS::IAM::User) resource whereby one could pass Keystone credentials to
>>> be able to specify Ceilometer alarms based on application's specific
>>> metrics (a.k.a KPIs).
>>>
>> It would likely be OS::Keystone::User, and AFAIK this is on the list of
>> de-AWS-ification things.
> Great! As I said. It's a blocker for us and really would like to see 
> it accepted for icehouse.
>>
>>> I hope this is making sense to you and can serve as a basis for further
>>> discussions and refinements.
>>>
>> Really great feedback Patrick, thanks again for sharing!
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Patrick Petit
Cloud Computing Principal Architect, Innovative Products
Bull, Architect of an Open World TM
Tél : +33 (0)4 76 29 70 31
Mobile : +33 (0)6 85 22 06 39
http://www.bull.com