Open Stack

On 07/10/2012 04:23 PM, Matt Ray wrote:
> Bluntness appreciated, this process is already in motion.
> http://opscode.com/openstack was launched 2 weeks ago and I promptly
> left for conferences and vacation. I am consolidating GitHub repos
> here:

Awesome.

> https://github.com/opscode/openstack-chef-repo
> https://github.com/opscode-cookbooks/nova
> https://github.com/opscode-cookbooks/glance
> https://github.com/opscode-cookbooks/horizon
> https://github.com/opscode-cookbooks/keystone
> https://github.com/opscode-cookbooks/swift
> 
> Work is being done in my own repos until it's ready for an initial
> release, which will include a Getting Started with Chef and OpenStack
> document.
> https://github.com/mattray/
> 
> I'm working with quite a few folks already, Rackspace, Dell, DreamHost
> and others and Intel is sponsoring this work.
> 
> Jay and I chatted a bit in IRC, we're quite aligned in how we plan on
> working this and the goal will be to get
> github.com/openstack/chef-repo gated with Gerrit and CI and pulling
> from Opscode's repos soon.

Only really replying because I saw the word gated. :) I'd love to be
part of any conversations that are being had on this subject, sooner
rather than later.

> Please feel free to join the discussion on
> our new mailing list focused on this effort here:
> http://groups.google.com/group/opscode-chef-openstack
>
> And an IRC channel:
> #openstack-chef on irc.freenode.net
> 
> Thanks,
> Matt Ray
> Senior Technical Evangelist | Opscode Inc.
> matt at opscode.com | (512) 731-2218
> Twitter, IRC, GitHub: mattray
> 
> 
> On Tue, Jul 10, 2012 at 11:25 AM, Jay Pipes <jaypipes at gmail.com> wrote:
>> Apologies in advance for my blunt and somewhat dour response, Matt. I'm
>> not singling you out at all, and I know you've tried your best to get
>> the various Chef stakeholders to work together. Also apologies for
>> top-posting, but there's not a whole lot of use inline posting this.
>>
>> tl;dr
>> -----
>>
>> We need to stop the needless fracturing of the operational knowledge of
>> the Chef community and try working as a team towards some common goals
>> instead of creating fork after fork of repos of Chef cookbooks.
>>
>> There is a ton of wasted effort in this area.
>>
>> Proposal:
>>
>> * Get our act together and treat Chef repos (and puppet/juju/whatever)
>> as we do other OpenStack core and supporting projects -- use Gerrit, use
>> a CI gating system, do real code reviews on it, and in general treat
>> them as a supporting OpenStack project
>> * Mark ALL Chef repos that are not currently maintained with an OBSELETE
>> marker and/or DELETE the repo on Github
>> * Consolidate all *cookbooks* into a repository in
>> github.com/openstack/chef-repo
>> * Use git submodules to manage cookbooks that are upstreamed in
>> github.com/opscode/ that have little to no changes in them
>> * Actually fix the documentation of the dang cookbooks -- right now,
>> half of them include the documentation from the memcache cookbook, as
>> they were lazily copy-pasted around, or the standard example doc that is
>> created when using something like knife.
>> * Put as much variation in deployment philosophy into *roles* and
>> attribute overrides/defaults
>>
>> More/Rant/Details
>> -----------------
>>
>> Maybe it's just the open source developer in me, but I don't understand
>> why there is such an aversion to coordination in the deployment/ops
>> community around the scripts and deployment
>> cookbooks/modules/charms/whatever.
>>
>> Is it that everyone has a different idea of what is best? Is it because
>> deployers/ops folks think that coordinating with other contributors is
>> too time-consuming? Is it because the chef repos are not controlled in
>> the same way as, say, devstack or the core projects, with an automated
>> patch queue manager and code review system that actually encourages
>> debate over patches? A combination of all of the above?
>>
>> Over the last 2 years, I've worked at 3 companies in the OpenStack
>> ecosystem. All three companies had their own repos of Chef cookbooks
>> (still do to this day). 50-60% of the content of these cookbooks is
>> identical. 10-20% of the content of these cookbooks is different -- but
>> only slightly or cosmetically. And a good portion of the remaining
>> 20-40% are differences that are incorrectly (IMHO) placed in the
>> cookbooks and recipes instead of where they should be: in roles and
>> environments, with cookbooks created that deal with variations in
>> deployments with attributes and the occasional if/else block.
>>
>> In trying to determine the appropriate Chef repo to use for the TryStack
>> project, we found the following repo:
>>
>> https://github.com/osops/chef-repo
>>
>> to have the most up-to-date. I've since been told this repo is no longer
>> maintained. This is very frustrating, not because of this particular
>> repo, but because this is just one in a long line of neglected and
>> forgotten forks of chef cookbook repositories. The fact that the default
>> Chef behaviour and Opscode documentation encourages the copy/pasting of
>> cookbooks all over the place and GitHub itself encourages the random and
>> promiscuous forking of repos doesn't help.
>>
>> Let's get real about the deployment/ops code and
>> cookbooks/modules/charms. Let's treat them the same way we do code in
>> the core projects and supporting projects.
>>
>> Thanks for your time,
>> -jay
>>
>> On 07/10/2012 11:42 AM, Matt Ray wrote:
>>> Just a heads up, I'm working on building unified community-driven
>>> cookbooks over in https://github.com/opscode/openstack-chef-repo (and
>>> repos for the individual cookbooks). These are forked from Rackspace's
>>> cookbooks and I'm working with them and others to make reusable,
>>> well-documented and supported Chef cookbooks for OpenStack. I'll make
>>> a larger announcement around them once I have a working quickstart
>>> document for them.
>>>
>>> tl;dr; https://github.com/opscode/openstack-chef-repo
>>>
>>> Thanks,
>>> Matt Ray
>>> Senior Technical Evangelist | Opscode Inc.
>>> matt at opscode.com | (512) 731-2218
>>> Twitter, IRC, GitHub: mattray
>>>
>>>
>>> On Tue, Jul 10, 2012 at 10:22 AM, Jay Pipes <jaypipes at gmail.com> wrote:
>>>> Gah... probably would be good if you guys either shut down the repo or
>>>> made a big notice on the README then :(
>>>>
>>>> -jay
>>>>
>>>> On 07/09/2012 05:25 PM, Joe Breu wrote:
>>>>> Hi Jay,
>>>>>
>>>>> The chef cookbooks at https://github.com/osops are no longer maintained.
>>>>>  Our current cookbooks are at  https://github.com/rcbops/chef-cookbooks
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> Joseph Breu
>>>>> Deployment Engineer
>>>>> Rackspace Cloud Builders
>>>>> 210-312-3508
>>>>>
>>>>> On Jul 9, 2012, at 8:01 AM, Jay Pipes wrote:
>>>>>
>>>>>> Vish and Ron, just getting back to this... see inline continued
>>>>>> questions for you both.
>>>>>>
>>>>>> On 07/02/2012 04:24 PM, Vishvananda Ishaya wrote:
>>>>>>>
>>>>>>> On Jul 2, 2012, at 7:28 AM, Jay Pipes wrote:
>>>>>>>
>>>>>>>> Hi Ron, cc'ing the openstack ML for extra eyes and opinions...
>>>>>>>>
>>>>>>>> So, Nati and I are looking to use either the osops chef-repo or
>>>>>>>> something similar as the basis of the new TryStack zone chef deployment.
>>>>>>>> I've been going through the recipes and roles and I have a question on
>>>>>>>> the nova-compute *role*:
>>>>>>>>
>>>>>>>> https://github.com/osops/chef-repo/blob/master/roles/nova-compute.rb
>>>>>>>>
>>>>>>>> I'm wondering why the nova-api recipe is in the runlist for
>>>>>>>> nova-compute?
>>>>>>>
>>>>>>> Because metadata needs to run on the compute hosts in the default
>>>>>>> layout. This should
>>>>>>> be switched to use nova-api-metadata instead of nova-api, but the
>>>>>>> split out hasn't been done yet.
>>>>>>
>>>>>> OK, I will work on splitting this out a bit more effectively.
>>>>>>
>>>>>> One additional question, though. In opening up the
>>>>>> /cookbooks/nova/recipes/nova/compute.rb file, you will notice this at
>>>>>> the top:
>>>>>>
>>>>>> include_recipe "nova::api"
>>>>>>
>>>>>> Therefore, unless I am mistaken, the nova-compute *role*'s runlist
>>>>>> actually doesn't need to contain both nova-api AND nova-compute since
>>>>>> apparently the nova-compute *recipe* already includes all of the
>>>>>> nova-api recipe.
>>>>>>
>>>>>> Would you agree with that?
>>>>>>
>>>>>>>> In addition, I was wondering if y'all had considered making more use of
>>>>>>>> roles instead of recipes to allow most of the attribute assignment and
>>>>>>>> variation to be in the combination of roles assigned to a host, instead
>>>>>>>> of recipes combined in a role?
>>>>>>>>
>>>>>>>> For example, right now, there is a "nova-controller" role that looks
>>>>>>>> like this:
>>>>>>>>
>>>>>>>> name "nova-controller"
>>>>>>>> description "Nova controller node (vncproxy + rabbit)"
>>>>>>>> run_list(
>>>>>>>> "role[base]",
>>>>>>>> "recipe[nova::controller]"
>>>>>>>> )
>>>>>>>>
>>>>>>>> Because most of the special sauce is in the nova::controller recipe, I
>>>>>>>> have to go into that recipe to see what the role is composed of:
>>>>>>>>
>>>>>>>> include_recipe "mysql::server"
>>>>>>>> include_recipe "openssh::default"
>>>>>>>>
>>>>>>>> include_recipe "rabbitmq::default"
>>>>>>>> include_recipe "keystone::server"
>>>>>>>> include_recipe "glance::registry"
>>>>>>>> include_recipe "glance::api"
>>>>>>>> include_recipe "nova::nova-setup"
>>>>>>>> include_recipe "nova::scheduler"
>>>>>>>> include_recipe "nova::api"
>>>>>>>>
>>>>>>>> if platform?(%w{fedora})
>>>>>>>> # Fedora skipping vncproxy for right now
>>>>>>>> else
>>>>>>>> include_recipe "nova::vncproxy"
>>>>>>>> end
>>>>>>>>
>>>>>>>> But what this recipe really is is an opinionated description of a
>>>>>>>> "controller role". If the role was, instead, structured like so:
>>>>>>>>
>>>>>>>> name "nova-controller"
>>>>>>>> description "Nova Controller - All major API services and control
>>>>>>>> servers like MySQL and Rabbit"
>>>>>>>> run_list(
>>>>>>>> "role[base]",
>>>>>>>> "recipe[mysql::server]",
>>>>>>>> "recipe[openssh::default]",
>>>>>>>> "recipe[rabbitmq::default]",
>>>>>>>> "recipe[keystone::server]",
>>>>>>>> "recipe[glance::api]",
>>>>>>>> "recipe[glance::registry]",
>>>>>>>> "recipe[nova::scheduler]",
>>>>>>>> "recipe[nova::api]",
>>>>>>>> )
>>>>>>>>
>>>>>>>> Then the deployer can more easily switch up the way they deploy
>>>>>>>> OpenStack servers by merely changing the role -- say, removing the
>>>>>>>> Rabbit service and putting it somewhere else -- WITHOUT having to modify
>>>>>>>> a recipe in a Git submodule in the upstream cookbooks.
>>>>>>>>
>>>>>>>> Furthermore, if we broke out more roles -- such as "control-services"
>>>>>>>> which might be MySQL and Rabbit only -- than we could make the "super
>>>>>>>> roles" ,like the nova-controller role above, more of a "put this and
>>>>>>>> that role together, like so:
>>>>>>>>
>>>>>>>> name "nova-controller"
>>>>>>>> description "Nova Controller - All major API services and control
>>>>>>>> servers like MySQL and Rabbit"
>>>>>>>> run_list(
>>>>>>>> "role[base]",
>>>>>>>> "role[control_services]",
>>>>>>>> "recipe[keystone::server]",
>>>>>>>> "recipe[glance::api]",
>>>>>>>> "recipe[glance::registry]",
>>>>>>>> "recipe[nova::scheduler]",
>>>>>>>> "recipe[nova::api]",
>>>>>>>> )
>>>>>>>
>>>>>>> This all makes sense to me.  Ron?
>>>>>>
>>>>>> Ron, any disagreement here?
>>>>>>
>>>>>>>> Finally, I've noticed that there are aren't any HA options in the osops
>>>>>>>> recipes -- specifically around MySQL. Are there plans to do so? We use
>>>>>>>> heartbeat/Pacemaker/DRBD in the original TryStack cookbooks [1] and
>>>>>>>> environments to get simple HA solutions up and would love to see those
>>>>>>>> in the upstream.
>>>>>>
>>>>>> Either of you, any thoughts on this front?
>>>>>>
>>>>>> Thanks!
>>>>>> -jay
>>>>>>
>>>>>>>> Thanks and all the best guys,
>>>>>>>> -jay
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/trystack/openstack-chef/tree/stable/diablo/cookbooks/heartbeat
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to     : openstack at lists.launchpad.net
>>>>>> <mailto:openstack at lists.launchpad.net>
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack at lists.launchpad.net
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>