Open Stack

Bluntness appreciated, this process is already in motion.
http://opscode.com/openstack was launched 2 weeks ago and I promptly
left for conferences and vacation. I am consolidating GitHub repos
here:

https://github.com/opscode/openstack-chef-repo
https://github.com/opscode-cookbooks/nova
https://github.com/opscode-cookbooks/glance
https://github.com/opscode-cookbooks/horizon
https://github.com/opscode-cookbooks/keystone
https://github.com/opscode-cookbooks/swift

Work is being done in my own repos until it's ready for an initial
release, which will include a Getting Started with Chef and OpenStack
document.
https://github.com/mattray/

I'm working with quite a few folks already, Rackspace, Dell, DreamHost
and others and Intel is sponsoring this work.

Jay and I chatted a bit in IRC, we're quite aligned in how we plan on
working this and the goal will be to get
github.com/openstack/chef-repo gated with Gerrit and CI and pulling
from Opscode's repos soon. Please feel free to join the discussion on
our new mailing list focused on this effort here:
http://groups.google.com/group/opscode-chef-openstack

And an IRC channel:
#openstack-chef on irc.freenode.net

Thanks,
Matt Ray
Senior Technical Evangelist | Opscode Inc.
matt at opscode.com | (512) 731-2218
Twitter, IRC, GitHub: mattray


On Tue, Jul 10, 2012 at 11:25 AM, Jay Pipes <jaypipes at gmail.com> wrote:
> Apologies in advance for my blunt and somewhat dour response, Matt. I'm
> not singling you out at all, and I know you've tried your best to get
> the various Chef stakeholders to work together. Also apologies for
> top-posting, but there's not a whole lot of use inline posting this.
>
> tl;dr
> -----
>
> We need to stop the needless fracturing of the operational knowledge of
> the Chef community and try working as a team towards some common goals
> instead of creating fork after fork of repos of Chef cookbooks.
>
> There is a ton of wasted effort in this area.
>
> Proposal:
>
> * Get our act together and treat Chef repos (and puppet/juju/whatever)
> as we do other OpenStack core and supporting projects -- use Gerrit, use
> a CI gating system, do real code reviews on it, and in general treat
> them as a supporting OpenStack project
> * Mark ALL Chef repos that are not currently maintained with an OBSELETE
> marker and/or DELETE the repo on Github
> * Consolidate all *cookbooks* into a repository in
> github.com/openstack/chef-repo
> * Use git submodules to manage cookbooks that are upstreamed in
> github.com/opscode/ that have little to no changes in them
> * Actually fix the documentation of the dang cookbooks -- right now,
> half of them include the documentation from the memcache cookbook, as
> they were lazily copy-pasted around, or the standard example doc that is
> created when using something like knife.
> * Put as much variation in deployment philosophy into *roles* and
> attribute overrides/defaults
>
> More/Rant/Details
> -----------------
>
> Maybe it's just the open source developer in me, but I don't understand
> why there is such an aversion to coordination in the deployment/ops
> community around the scripts and deployment
> cookbooks/modules/charms/whatever.
>
> Is it that everyone has a different idea of what is best? Is it because
> deployers/ops folks think that coordinating with other contributors is
> too time-consuming? Is it because the chef repos are not controlled in
> the same way as, say, devstack or the core projects, with an automated
> patch queue manager and code review system that actually encourages
> debate over patches? A combination of all of the above?
>
> Over the last 2 years, I've worked at 3 companies in the OpenStack
> ecosystem. All three companies had their own repos of Chef cookbooks
> (still do to this day). 50-60% of the content of these cookbooks is
> identical. 10-20% of the content of these cookbooks is different -- but
> only slightly or cosmetically. And a good portion of the remaining
> 20-40% are differences that are incorrectly (IMHO) placed in the
> cookbooks and recipes instead of where they should be: in roles and
> environments, with cookbooks created that deal with variations in
> deployments with attributes and the occasional if/else block.
>
> In trying to determine the appropriate Chef repo to use for the TryStack
> project, we found the following repo:
>
> https://github.com/osops/chef-repo
>
> to have the most up-to-date. I've since been told this repo is no longer
> maintained. This is very frustrating, not because of this particular
> repo, but because this is just one in a long line of neglected and
> forgotten forks of chef cookbook repositories. The fact that the default
> Chef behaviour and Opscode documentation encourages the copy/pasting of
> cookbooks all over the place and GitHub itself encourages the random and
> promiscuous forking of repos doesn't help.
>
> Let's get real about the deployment/ops code and
> cookbooks/modules/charms. Let's treat them the same way we do code in
> the core projects and supporting projects.
>
> Thanks for your time,
> -jay
>
> On 07/10/2012 11:42 AM, Matt Ray wrote:
>> Just a heads up, I'm working on building unified community-driven
>> cookbooks over in https://github.com/opscode/openstack-chef-repo (and
>> repos for the individual cookbooks). These are forked from Rackspace's
>> cookbooks and I'm working with them and others to make reusable,
>> well-documented and supported Chef cookbooks for OpenStack. I'll make
>> a larger announcement around them once I have a working quickstart
>> document for them.
>>
>> tl;dr; https://github.com/opscode/openstack-chef-repo
>>
>> Thanks,
>> Matt Ray
>> Senior Technical Evangelist | Opscode Inc.
>> matt at opscode.com | (512) 731-2218
>> Twitter, IRC, GitHub: mattray
>>
>>
>> On Tue, Jul 10, 2012 at 10:22 AM, Jay Pipes <jaypipes at gmail.com> wrote:
>>> Gah... probably would be good if you guys either shut down the repo or
>>> made a big notice on the README then :(
>>>
>>> -jay
>>>
>>> On 07/09/2012 05:25 PM, Joe Breu wrote:
>>>> Hi Jay,
>>>>
>>>> The chef cookbooks at https://github.com/osops are no longer maintained.
>>>>  Our current cookbooks are at  https://github.com/rcbops/chef-cookbooks
>>>>
>>>>
>>>>
>>>> ---
>>>> Joseph Breu
>>>> Deployment Engineer
>>>> Rackspace Cloud Builders
>>>> 210-312-3508
>>>>
>>>> On Jul 9, 2012, at 8:01 AM, Jay Pipes wrote:
>>>>
>>>>> Vish and Ron, just getting back to this... see inline continued
>>>>> questions for you both.
>>>>>
>>>>> On 07/02/2012 04:24 PM, Vishvananda Ishaya wrote:
>>>>>>
>>>>>> On Jul 2, 2012, at 7:28 AM, Jay Pipes wrote:
>>>>>>
>>>>>>> Hi Ron, cc'ing the openstack ML for extra eyes and opinions...
>>>>>>>
>>>>>>> So, Nati and I are looking to use either the osops chef-repo or
>>>>>>> something similar as the basis of the new TryStack zone chef deployment.
>>>>>>> I've been going through the recipes and roles and I have a question on
>>>>>>> the nova-compute *role*:
>>>>>>>
>>>>>>> https://github.com/osops/chef-repo/blob/master/roles/nova-compute.rb
>>>>>>>
>>>>>>> I'm wondering why the nova-api recipe is in the runlist for
>>>>>>> nova-compute?
>>>>>>
>>>>>> Because metadata needs to run on the compute hosts in the default
>>>>>> layout. This should
>>>>>> be switched to use nova-api-metadata instead of nova-api, but the
>>>>>> split out hasn't been done yet.
>>>>>
>>>>> OK, I will work on splitting this out a bit more effectively.
>>>>>
>>>>> One additional question, though. In opening up the
>>>>> /cookbooks/nova/recipes/nova/compute.rb file, you will notice this at
>>>>> the top:
>>>>>
>>>>> include_recipe "nova::api"
>>>>>
>>>>> Therefore, unless I am mistaken, the nova-compute *role*'s runlist
>>>>> actually doesn't need to contain both nova-api AND nova-compute since
>>>>> apparently the nova-compute *recipe* already includes all of the
>>>>> nova-api recipe.
>>>>>
>>>>> Would you agree with that?
>>>>>
>>>>>>> In addition, I was wondering if y'all had considered making more use of
>>>>>>> roles instead of recipes to allow most of the attribute assignment and
>>>>>>> variation to be in the combination of roles assigned to a host, instead
>>>>>>> of recipes combined in a role?
>>>>>>>
>>>>>>> For example, right now, there is a "nova-controller" role that looks
>>>>>>> like this:
>>>>>>>
>>>>>>> name "nova-controller"
>>>>>>> description "Nova controller node (vncproxy + rabbit)"
>>>>>>> run_list(
>>>>>>> "role[base]",
>>>>>>> "recipe[nova::controller]"
>>>>>>> )
>>>>>>>
>>>>>>> Because most of the special sauce is in the nova::controller recipe, I
>>>>>>> have to go into that recipe to see what the role is composed of:
>>>>>>>
>>>>>>> include_recipe "mysql::server"
>>>>>>> include_recipe "openssh::default"
>>>>>>>
>>>>>>> include_recipe "rabbitmq::default"
>>>>>>> include_recipe "keystone::server"
>>>>>>> include_recipe "glance::registry"
>>>>>>> include_recipe "glance::api"
>>>>>>> include_recipe "nova::nova-setup"
>>>>>>> include_recipe "nova::scheduler"
>>>>>>> include_recipe "nova::api"
>>>>>>>
>>>>>>> if platform?(%w{fedora})
>>>>>>> # Fedora skipping vncproxy for right now
>>>>>>> else
>>>>>>> include_recipe "nova::vncproxy"
>>>>>>> end
>>>>>>>
>>>>>>> But what this recipe really is is an opinionated description of a
>>>>>>> "controller role". If the role was, instead, structured like so:
>>>>>>>
>>>>>>> name "nova-controller"
>>>>>>> description "Nova Controller - All major API services and control
>>>>>>> servers like MySQL and Rabbit"
>>>>>>> run_list(
>>>>>>> "role[base]",
>>>>>>> "recipe[mysql::server]",
>>>>>>> "recipe[openssh::default]",
>>>>>>> "recipe[rabbitmq::default]",
>>>>>>> "recipe[keystone::server]",
>>>>>>> "recipe[glance::api]",
>>>>>>> "recipe[glance::registry]",
>>>>>>> "recipe[nova::scheduler]",
>>>>>>> "recipe[nova::api]",
>>>>>>> )
>>>>>>>
>>>>>>> Then the deployer can more easily switch up the way they deploy
>>>>>>> OpenStack servers by merely changing the role -- say, removing the
>>>>>>> Rabbit service and putting it somewhere else -- WITHOUT having to modify
>>>>>>> a recipe in a Git submodule in the upstream cookbooks.
>>>>>>>
>>>>>>> Furthermore, if we broke out more roles -- such as "control-services"
>>>>>>> which might be MySQL and Rabbit only -- than we could make the "super
>>>>>>> roles" ,like the nova-controller role above, more of a "put this and
>>>>>>> that role together, like so:
>>>>>>>
>>>>>>> name "nova-controller"
>>>>>>> description "Nova Controller - All major API services and control
>>>>>>> servers like MySQL and Rabbit"
>>>>>>> run_list(
>>>>>>> "role[base]",
>>>>>>> "role[control_services]",
>>>>>>> "recipe[keystone::server]",
>>>>>>> "recipe[glance::api]",
>>>>>>> "recipe[glance::registry]",
>>>>>>> "recipe[nova::scheduler]",
>>>>>>> "recipe[nova::api]",
>>>>>>> )
>>>>>>
>>>>>> This all makes sense to me.  Ron?
>>>>>
>>>>> Ron, any disagreement here?
>>>>>
>>>>>>> Finally, I've noticed that there are aren't any HA options in the osops
>>>>>>> recipes -- specifically around MySQL. Are there plans to do so? We use
>>>>>>> heartbeat/Pacemaker/DRBD in the original TryStack cookbooks [1] and
>>>>>>> environments to get simple HA solutions up and would love to see those
>>>>>>> in the upstream.
>>>>>
>>>>> Either of you, any thoughts on this front?
>>>>>
>>>>> Thanks!
>>>>> -jay
>>>>>
>>>>>>> Thanks and all the best guys,
>>>>>>> -jay
>>>>>>>
>>>>>>> [1]
>>>>>>> https://github.com/trystack/openstack-chef/tree/stable/diablo/cookbooks/heartbeat
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to     : openstack at lists.launchpad.net
>>>>> <mailto:openstack at lists.launchpad.net>
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack at lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>