Open Stack

Apologies in advance for my blunt and somewhat dour response, Matt. I'm
not singling you out at all, and I know you've tried your best to get
the various Chef stakeholders to work together. Also apologies for
top-posting, but there's not a whole lot of use inline posting this.

tl;dr
-----

We need to stop the needless fracturing of the operational knowledge of
the Chef community and try working as a team towards some common goals
instead of creating fork after fork of repos of Chef cookbooks.

There is a ton of wasted effort in this area.

Proposal:

* Get our act together and treat Chef repos (and puppet/juju/whatever)
as we do other OpenStack core and supporting projects -- use Gerrit, use
a CI gating system, do real code reviews on it, and in general treat
them as a supporting OpenStack project
* Mark ALL Chef repos that are not currently maintained with an OBSELETE
marker and/or DELETE the repo on Github
* Consolidate all *cookbooks* into a repository in
github.com/openstack/chef-repo
* Use git submodules to manage cookbooks that are upstreamed in
github.com/opscode/ that have little to no changes in them
* Actually fix the documentation of the dang cookbooks -- right now,
half of them include the documentation from the memcache cookbook, as
they were lazily copy-pasted around, or the standard example doc that is
created when using something like knife.
* Put as much variation in deployment philosophy into *roles* and
attribute overrides/defaults

More/Rant/Details
-----------------

Maybe it's just the open source developer in me, but I don't understand
why there is such an aversion to coordination in the deployment/ops
community around the scripts and deployment
cookbooks/modules/charms/whatever.

Is it that everyone has a different idea of what is best? Is it because
deployers/ops folks think that coordinating with other contributors is
too time-consuming? Is it because the chef repos are not controlled in
the same way as, say, devstack or the core projects, with an automated
patch queue manager and code review system that actually encourages
debate over patches? A combination of all of the above?

Over the last 2 years, I've worked at 3 companies in the OpenStack
ecosystem. All three companies had their own repos of Chef cookbooks
(still do to this day). 50-60% of the content of these cookbooks is
identical. 10-20% of the content of these cookbooks is different -- but
only slightly or cosmetically. And a good portion of the remaining
20-40% are differences that are incorrectly (IMHO) placed in the
cookbooks and recipes instead of where they should be: in roles and
environments, with cookbooks created that deal with variations in
deployments with attributes and the occasional if/else block.

In trying to determine the appropriate Chef repo to use for the TryStack
project, we found the following repo:

https://github.com/osops/chef-repo

to have the most up-to-date. I've since been told this repo is no longer
maintained. This is very frustrating, not because of this particular
repo, but because this is just one in a long line of neglected and
forgotten forks of chef cookbook repositories. The fact that the default
Chef behaviour and Opscode documentation encourages the copy/pasting of
cookbooks all over the place and GitHub itself encourages the random and
promiscuous forking of repos doesn't help.

Let's get real about the deployment/ops code and
cookbooks/modules/charms. Let's treat them the same way we do code in
the core projects and supporting projects.

Thanks for your time,
-jay

On 07/10/2012 11:42 AM, Matt Ray wrote:
> Just a heads up, I'm working on building unified community-driven
> cookbooks over in https://github.com/opscode/openstack-chef-repo (and
> repos for the individual cookbooks). These are forked from Rackspace's
> cookbooks and I'm working with them and others to make reusable,
> well-documented and supported Chef cookbooks for OpenStack. I'll make
> a larger announcement around them once I have a working quickstart
> document for them.
> 
> tl;dr; https://github.com/opscode/openstack-chef-repo
> 
> Thanks,
> Matt Ray
> Senior Technical Evangelist | Opscode Inc.
> matt at opscode.com | (512) 731-2218
> Twitter, IRC, GitHub: mattray
> 
> 
> On Tue, Jul 10, 2012 at 10:22 AM, Jay Pipes <jaypipes at gmail.com> wrote:
>> Gah... probably would be good if you guys either shut down the repo or
>> made a big notice on the README then :(
>>
>> -jay
>>
>> On 07/09/2012 05:25 PM, Joe Breu wrote:
>>> Hi Jay,
>>>
>>> The chef cookbooks at https://github.com/osops are no longer maintained.
>>>  Our current cookbooks are at  https://github.com/rcbops/chef-cookbooks
>>>
>>>
>>>
>>> ---
>>> Joseph Breu
>>> Deployment Engineer
>>> Rackspace Cloud Builders
>>> 210-312-3508
>>>
>>> On Jul 9, 2012, at 8:01 AM, Jay Pipes wrote:
>>>
>>>> Vish and Ron, just getting back to this... see inline continued
>>>> questions for you both.
>>>>
>>>> On 07/02/2012 04:24 PM, Vishvananda Ishaya wrote:
>>>>>
>>>>> On Jul 2, 2012, at 7:28 AM, Jay Pipes wrote:
>>>>>
>>>>>> Hi Ron, cc'ing the openstack ML for extra eyes and opinions...
>>>>>>
>>>>>> So, Nati and I are looking to use either the osops chef-repo or
>>>>>> something similar as the basis of the new TryStack zone chef deployment.
>>>>>> I've been going through the recipes and roles and I have a question on
>>>>>> the nova-compute *role*:
>>>>>>
>>>>>> https://github.com/osops/chef-repo/blob/master/roles/nova-compute.rb
>>>>>>
>>>>>> I'm wondering why the nova-api recipe is in the runlist for
>>>>>> nova-compute?
>>>>>
>>>>> Because metadata needs to run on the compute hosts in the default
>>>>> layout. This should
>>>>> be switched to use nova-api-metadata instead of nova-api, but the
>>>>> split out hasn't been done yet.
>>>>
>>>> OK, I will work on splitting this out a bit more effectively.
>>>>
>>>> One additional question, though. In opening up the
>>>> /cookbooks/nova/recipes/nova/compute.rb file, you will notice this at
>>>> the top:
>>>>
>>>> include_recipe "nova::api"
>>>>
>>>> Therefore, unless I am mistaken, the nova-compute *role*'s runlist
>>>> actually doesn't need to contain both nova-api AND nova-compute since
>>>> apparently the nova-compute *recipe* already includes all of the
>>>> nova-api recipe.
>>>>
>>>> Would you agree with that?
>>>>
>>>>>> In addition, I was wondering if y'all had considered making more use of
>>>>>> roles instead of recipes to allow most of the attribute assignment and
>>>>>> variation to be in the combination of roles assigned to a host, instead
>>>>>> of recipes combined in a role?
>>>>>>
>>>>>> For example, right now, there is a "nova-controller" role that looks
>>>>>> like this:
>>>>>>
>>>>>> name "nova-controller"
>>>>>> description "Nova controller node (vncproxy + rabbit)"
>>>>>> run_list(
>>>>>> "role[base]",
>>>>>> "recipe[nova::controller]"
>>>>>> )
>>>>>>
>>>>>> Because most of the special sauce is in the nova::controller recipe, I
>>>>>> have to go into that recipe to see what the role is composed of:
>>>>>>
>>>>>> include_recipe "mysql::server"
>>>>>> include_recipe "openssh::default"
>>>>>>
>>>>>> include_recipe "rabbitmq::default"
>>>>>> include_recipe "keystone::server"
>>>>>> include_recipe "glance::registry"
>>>>>> include_recipe "glance::api"
>>>>>> include_recipe "nova::nova-setup"
>>>>>> include_recipe "nova::scheduler"
>>>>>> include_recipe "nova::api"
>>>>>>
>>>>>> if platform?(%w{fedora})
>>>>>> # Fedora skipping vncproxy for right now
>>>>>> else
>>>>>> include_recipe "nova::vncproxy"
>>>>>> end
>>>>>>
>>>>>> But what this recipe really is is an opinionated description of a
>>>>>> "controller role". If the role was, instead, structured like so:
>>>>>>
>>>>>> name "nova-controller"
>>>>>> description "Nova Controller - All major API services and control
>>>>>> servers like MySQL and Rabbit"
>>>>>> run_list(
>>>>>> "role[base]",
>>>>>> "recipe[mysql::server]",
>>>>>> "recipe[openssh::default]",
>>>>>> "recipe[rabbitmq::default]",
>>>>>> "recipe[keystone::server]",
>>>>>> "recipe[glance::api]",
>>>>>> "recipe[glance::registry]",
>>>>>> "recipe[nova::scheduler]",
>>>>>> "recipe[nova::api]",
>>>>>> )
>>>>>>
>>>>>> Then the deployer can more easily switch up the way they deploy
>>>>>> OpenStack servers by merely changing the role -- say, removing the
>>>>>> Rabbit service and putting it somewhere else -- WITHOUT having to modify
>>>>>> a recipe in a Git submodule in the upstream cookbooks.
>>>>>>
>>>>>> Furthermore, if we broke out more roles -- such as "control-services"
>>>>>> which might be MySQL and Rabbit only -- than we could make the "super
>>>>>> roles" ,like the nova-controller role above, more of a "put this and
>>>>>> that role together, like so:
>>>>>>
>>>>>> name "nova-controller"
>>>>>> description "Nova Controller - All major API services and control
>>>>>> servers like MySQL and Rabbit"
>>>>>> run_list(
>>>>>> "role[base]",
>>>>>> "role[control_services]",
>>>>>> "recipe[keystone::server]",
>>>>>> "recipe[glance::api]",
>>>>>> "recipe[glance::registry]",
>>>>>> "recipe[nova::scheduler]",
>>>>>> "recipe[nova::api]",
>>>>>> )
>>>>>
>>>>> This all makes sense to me.  Ron?
>>>>
>>>> Ron, any disagreement here?
>>>>
>>>>>> Finally, I've noticed that there are aren't any HA options in the osops
>>>>>> recipes -- specifically around MySQL. Are there plans to do so? We use
>>>>>> heartbeat/Pacemaker/DRBD in the original TryStack cookbooks [1] and
>>>>>> environments to get simple HA solutions up and would love to see those
>>>>>> in the upstream.
>>>>
>>>> Either of you, any thoughts on this front?
>>>>
>>>> Thanks!
>>>> -jay
>>>>
>>>>>> Thanks and all the best guys,
>>>>>> -jay
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/trystack/openstack-chef/tree/stable/diablo/cookbooks/heartbeat
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack at lists.launchpad.net
>>>> <mailto:openstack at lists.launchpad.net>
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack at lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp