Open Stack

Hi Patric,

Thank you for such great post! This is very close to the vision I've tried
to propose earlier on software orchestration thread and I'm glad other
people concern about the same issues. However the problem the problem with
PaaS-like approached it that they currently on a little bit higher
abstraction layer than Heat is intended to be. Typical Heat users are more
of DevOps people rather than those who enjoy PaaS-related solutions. Going
that direction would require some major paradigm shift for the Heat which I
think is unnecessary.

I believe there is a place in OpenStack software-orchestration ecosystem
for layers that would sin on top of Heat and provide more high-level
services for software composition, dependency management. Heat is not aimed
to be software-everything. I would suggest you to take a look at Murano
project as it is very very close to what you want to achieve and as every
open-source project it needs community contributions. And I believe that it
is the place in OpenStack ecosystem where your expirience would be most
valuable and appreciated as well as your contributions


On Wed, Oct 23, 2013 at 9:58 PM, Patrick Petit <patrick.petit at bull.net>wrote:

>  Dear Steve and All,
>
> If I may add up on this already busy thread to share our experience with
> using Heat in large and complex software deployments.
>
> I work on a project which precisely provides additional value at the
> articulation point between resource orchestration automation and
> configuration management. We rely on Heat and chef-solo respectively for
> these base management functions. On top of this, we have developed an
> event-driven workflow to manage the life-cycles of complex software stacks
> which primary purpose is to support middleware components as opposed to
> end-user apps. Our use cases are peculiar in the sense that software setup
> (install, config, contextualization) is not a one-time operation issue but
> a continuous thing that can happen any time in life-span of a stack. Users
> can deploy (and undeploy) apps long time after the stack is created.
> Auto-scaling may also result in an asynchronous apps deployment. More about
> this latter. The framework we have designed works well for us. It clearly
> refers to a PaaS-like environment which I understand is not the topic of
> the HOT software configuration proposal(s) and that's absolutely fine with
> us. However, the question for us is whether the separation of software
> config from resources would make our life easier or not. I think the answer
> is definitely yes but at the condition that the DSL extension preserves
> almost everything from the expressiveness of the resource element. In
> practice, I think that a strict separation between resource and component
> will be hard to achieve because we'll always need a little bit of
> application's specific in the resources. Take for example the case of the
> SecurityGroups. The ports open in a SecurityGroup are application specific.
>
> Then, designing a Chef or Puppet component type may be harder than it
> looks at first glance. Speaking of our use cases we still need a little bit
> of scripting in the instance's user-data block to setup a working chef-solo
> environment. For example, we run librarian-chef prior to starting chef-solo
> to resolve the cookbook dependencies. A cookbook can present itself as a
> downloadable tarball but it's not always the case. A chef component type
> would have to support getting a cookbook from a public or private git repo
> (maybe subversion), handle situations where there is one cookbook per repo
> or multiple cookbooks per repo, let the user choose a particular branch or
> label, provide ssh keys if it's a private repo, and so forth. We support
> all of this scenarios and so we can provide more detailed requirements if
> needed.
>
> I am not sure adding component relations like the 'depends-on' would
> really help us since it is the job of config management to handle software
> dependencies. Also, it doesn't address the issue of circular dependencies.
> Circular dependencies occur in complex software stack deployments. Example.
> When we setup a Slum virtual cluster, both the head node and compute nodes
> depend on one another to complete their configuration and so they would
> wait for each other indefinitely if we were to rely on the 'depends-on'. In
> addition, I think it's critical to distinguish between configuration
> parameters which are known ahead of time, like a db name or user name and
> password, versus contextualization parameters which are known after the
> fact generally when the instance is created. Typically those
> contextualization parameters are IP addresses but not only. The fact
> packages x,y,z have been properly installed and services a,b,c successfully
> started is contextualization information (a.k.a facts) which may be
> indicative that other components can move on to the next setup stage.
>
> The case of complex deployments with or without circular dependencies is
> typically resolved by making the system converge toward the desirable
> end-state through running idempotent recipes. This is our approach. The
> first configuration phase handles parametrization which in general brings
> an instance to CREATE_COMPLETE state. A second phase follows to handle
> contextualization at the stack level. As a matter of fact, a new
> contextualization should be triggered every time an instance enters or
> leave the CREATE_COMPLETE state which may happen any time with
> auto-scaling. In that phase, circular dependencies can be resolved because
> all contextualization data can be compiled globally. Notice that Heat
> doesn't provide a purpose built resource or service like Chef's data-bag
> for the storage and retrieval of metadata. This a gap which IMO should be
> addressed in the proposal. Currently, we use a kludge that is to create a
> fake AWS::AutoScaling::LaunchConfiguration resource to store
> contextualization data in the metadata section of that resource.
>
> Aside from the HOT software configuration proposal(s). There are two
> critical enhancements in Heat that would make software life-cycles
> management much easier. In fact, they are actual blockers for us.
>
> The first one would be to support asynchronous notifications when an
> instance is created or deleted as a result of an auto-scaling decision. As
> stated earlier, contextualization needs to apply in a stack every time a
> instance enters or leaves the CREATE_COMPLETE state. I am not referring to
> a Ceilometer notification but a Heat notification that can be consumed by a
> Heat client.
>
> The second one would be to support a new type of AWS::IAM::User (perhaps
> OS::IAM::User) resource whereby one could pass Keystone credentials to be
> able to specify Ceilometer alarms based on application's specific metrics
> (a.k.a KPIs).
>
> I hope this is making sense to you and can serve as a basis for further
> discussions and refinements.
>
> Cheers,
> Patrick
>
>
> On 10/16/13 12:48 AM, Steve Baker wrote:
>
> I've just written some proposals to address Heat's HOT software
> configuration needs, and I'd like to use this thread to get some feedback:
> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config
>
> https://wiki.openstack.org/wiki/Heat/Blueprints/native-tools-bootstrap-config
>
> Please read the proposals and reply to the list with any comments or
> suggestions.
>
> We can spend some time discussing software configuration at tomorrow's
> Heat meeting, but I fully expect we'll still be in the discussion phase at
> Hong Kong.
>
> cheers
>
>
> _______________________________________________
> OpenStack-dev mailing listOpenStack-dev at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Patrick Petit
> Cloud Computing Principal Architect, Innovative Products
> Bull, Architect of an Open World TM
> Tél : +33 (0)4 76 29 70 31
> Mobile : +33 (0)6 85 22 06 39http://www.bull.com
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Sincerely yours
Stanislav (Stan) Lagun
Senior Developer
Mirantis
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
www.mirantis.com
slagun at mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131023/c402bec7/attachment-0001.html>