Open Stack

On 07/11/2012 10:04 AM, Matt Ray wrote:
> I know Rackspace gates their commits with foodcritic
> (https://rubygems.org/gems/foodcritic), which is becoming the standard
> we're recommending to people for cookbooks style. For doing Travis CI
> testing automatically, I plan on adding testing outlined in this post:
> http://nathenharvey.com/blog/2012/05/29/mvt-foodcritic-and-travis-ci/

Great. I imagine running foodcritic on a chef repo owned by openstack
would not be hard to do if that's something we want to have/own/do. (The
Travis CI stuff is obviously not very interesting to me, but I suppose
other people read this list who are not me. weird. :) )

> Opscode also has a "test-kitchen" project in progress that will be
> running CI with Jenkins on a number of platforms automatically for
> community cookbooks, we'll be pushing the OpenStack cookbooks on that
> as well as soon as it's made public and open sourced.

Sweet.

Let me know if there are any things that people are wanting related to
any of these projects from the OpenStack CI infrastructure.
Foodcritic/jsonlint seem pretty easy - deployments on to bare nodes
using the chef stuff similar to our devstack-based installs might take a
little more work and would need to be planned for. :)

> Thanks,
> Matt Ray
> Senior Technical Evangelist | Opscode Inc.
> matt at opscode.com | (512) 731-2218
> Twitter, IRC, GitHub: mattray
> 
> 
> On Wed, Jul 11, 2012 at 9:39 AM, Sullivan, Jon Paul
> <JonPaul.Sullivan at hp.com> wrote:
>>> -----Original Message-----
>>> From: openstack-bounces+jonpaul.sullivan=hp.com at lists.launchpad.net
>>> [mailto:openstack-bounces+jonpaul.sullivan=hp.com at lists.launchpad.net]
>>> On Behalf Of Monty Taylor
>>> Sent: 11 July 2012 14:37
>>> To: Matt Ray
>>> Cc: Nati Ueno; openstack at lists.launchpad.net
>>> Subject: Re: [Openstack] [CHEF] Clarification on osops/chef-
>>> repo/roles/nova-compute.rb
>>>
>>>
>>>
>>> On 07/10/2012 04:23 PM, Matt Ray wrote:
>>>> Bluntness appreciated, this process is already in motion.
>>>> http://opscode.com/openstack was launched 2 weeks ago and I promptly
>>>> left for conferences and vacation. I am consolidating GitHub repos
>>>> here:
>>>
>>> Awesome.
>>>
>>>> https://github.com/opscode/openstack-chef-repo
>>>> https://github.com/opscode-cookbooks/nova
>>>> https://github.com/opscode-cookbooks/glance
>>>> https://github.com/opscode-cookbooks/horizon
>>>> https://github.com/opscode-cookbooks/keystone
>>>> https://github.com/opscode-cookbooks/swift
>>>>
>>>> Work is being done in my own repos until it's ready for an initial
>>>> release, which will include a Getting Started with Chef and OpenStack
>>>> document.
>>>> https://github.com/mattray/
>>>>
>>>> I'm working with quite a few folks already, Rackspace, Dell, DreamHost
>>>> and others and Intel is sponsoring this work.
>>>>
>>>> Jay and I chatted a bit in IRC, we're quite aligned in how we plan on
>>>> working this and the goal will be to get
>>>> github.com/openstack/chef-repo gated with Gerrit and CI and pulling
>>>> from Opscode's repos soon.
>>>
>>> Only really replying because I saw the word gated. :) I'd love to be
>>> part of any conversations that are being had on this subject, sooner
>>> rather than later.
>>
>> Our standard gating tests for chef code are to run "jsonlint" on all json files, "knife cookbook test" on all cookbooks, and then running all chefspec tests for every cookbook via rspec.
>>
>> Suggestions of extra tests that would be worthwhile gratefully received ;-)
>>
>>>
>>>> Please feel free to join the discussion on our new mailing list
>>>> focused on this effort here:
>>>> http://groups.google.com/group/opscode-chef-openstack
>>>>
>>>> And an IRC channel:
>>>> #openstack-chef on irc.freenode.net
>>>>
>>>> Thanks,
>>>> Matt Ray
>>>> Senior Technical Evangelist | Opscode Inc.
>>>> matt at opscode.com | (512) 731-2218
>>>> Twitter, IRC, GitHub: mattray
>>>>
>>>>
>>>> On Tue, Jul 10, 2012 at 11:25 AM, Jay Pipes <jaypipes at gmail.com>
>>> wrote:
>>>>> Apologies in advance for my blunt and somewhat dour response, Matt.
>>>>> I'm not singling you out at all, and I know you've tried your best to
>>>>> get the various Chef stakeholders to work together. Also apologies
>>>>> for top-posting, but there's not a whole lot of use inline posting
>>> this.
>>>>>
>>>>> tl;dr
>>>>> -----
>>>>>
>>>>> We need to stop the needless fracturing of the operational knowledge
>>>>> of the Chef community and try working as a team towards some common
>>>>> goals instead of creating fork after fork of repos of Chef cookbooks.
>>>>>
>>>>> There is a ton of wasted effort in this area.
>>>>>
>>>>> Proposal:
>>>>>
>>>>> * Get our act together and treat Chef repos (and
>>>>> puppet/juju/whatever) as we do other OpenStack core and supporting
>>>>> projects -- use Gerrit, use a CI gating system, do real code reviews
>>>>> on it, and in general treat them as a supporting OpenStack project
>>>>> * Mark ALL Chef repos that are not currently maintained with an
>>>>> OBSELETE marker and/or DELETE the repo on Github
>>>>> * Consolidate all *cookbooks* into a repository in
>>>>> github.com/openstack/chef-repo
>>>>> * Use git submodules to manage cookbooks that are upstreamed in
>>>>> github.com/opscode/ that have little to no changes in them
>>>>> * Actually fix the documentation of the dang cookbooks -- right now,
>>>>> half of them include the documentation from the memcache cookbook, as
>>>>> they were lazily copy-pasted around, or the standard example doc that
>>>>> is created when using something like knife.
>>>>> * Put as much variation in deployment philosophy into *roles* and
>>>>> attribute overrides/defaults
>>>>>
>>>>> More/Rant/Details
>>>>> -----------------
>>>>>
>>>>> Maybe it's just the open source developer in me, but I don't
>>>>> understand why there is such an aversion to coordination in the
>>>>> deployment/ops community around the scripts and deployment
>>>>> cookbooks/modules/charms/whatever.
>>>>>
>>>>> Is it that everyone has a different idea of what is best? Is it
>>>>> because deployers/ops folks think that coordinating with other
>>>>> contributors is too time-consuming? Is it because the chef repos are
>>>>> not controlled in the same way as, say, devstack or the core
>>>>> projects, with an automated patch queue manager and code review
>>>>> system that actually encourages debate over patches? A combination of
>>> all of the above?
>>>>>
>>>>> Over the last 2 years, I've worked at 3 companies in the OpenStack
>>>>> ecosystem. All three companies had their own repos of Chef cookbooks
>>>>> (still do to this day). 50-60% of the content of these cookbooks is
>>>>> identical. 10-20% of the content of these cookbooks is different --
>>>>> but only slightly or cosmetically. And a good portion of the
>>>>> remaining 20-40% are differences that are incorrectly (IMHO) placed
>>>>> in the cookbooks and recipes instead of where they should be: in
>>>>> roles and environments, with cookbooks created that deal with
>>>>> variations in deployments with attributes and the occasional if/else
>>> block.
>>>>>
>>>>> In trying to determine the appropriate Chef repo to use for the
>>>>> TryStack project, we found the following repo:
>>>>>
>>>>> https://github.com/osops/chef-repo
>>>>>
>>>>> to have the most up-to-date. I've since been told this repo is no
>>>>> longer maintained. This is very frustrating, not because of this
>>>>> particular repo, but because this is just one in a long line of
>>>>> neglected and forgotten forks of chef cookbook repositories. The fact
>>>>> that the default Chef behaviour and Opscode documentation encourages
>>>>> the copy/pasting of cookbooks all over the place and GitHub itself
>>>>> encourages the random and promiscuous forking of repos doesn't help.
>>>>>
>>>>> Let's get real about the deployment/ops code and
>>>>> cookbooks/modules/charms. Let's treat them the same way we do code in
>>>>> the core projects and supporting projects.
>>>>>
>>>>> Thanks for your time,
>>>>> -jay
>>>>>
>>>>> On 07/10/2012 11:42 AM, Matt Ray wrote:
>>>>>> Just a heads up, I'm working on building unified community-driven
>>>>>> cookbooks over in https://github.com/opscode/openstack-chef-repo
>>>>>> (and repos for the individual cookbooks). These are forked from
>>>>>> Rackspace's cookbooks and I'm working with them and others to make
>>>>>> reusable, well-documented and supported Chef cookbooks for
>>>>>> OpenStack. I'll make a larger announcement around them once I have a
>>>>>> working quickstart document for them.
>>>>>>
>>>>>> tl;dr; https://github.com/opscode/openstack-chef-repo
>>>>>>
>>>>>> Thanks,
>>>>>> Matt Ray
>>>>>> Senior Technical Evangelist | Opscode Inc.
>>>>>> matt at opscode.com | (512) 731-2218
>>>>>> Twitter, IRC, GitHub: mattray
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 10, 2012 at 10:22 AM, Jay Pipes <jaypipes at gmail.com>
>>> wrote:
>>>>>>> Gah... probably would be good if you guys either shut down the repo
>>>>>>> or made a big notice on the README then :(
>>>>>>>
>>>>>>> -jay
>>>>>>>
>>>>>>> On 07/09/2012 05:25 PM, Joe Breu wrote:
>>>>>>>> Hi Jay,
>>>>>>>>
>>>>>>>> The chef cookbooks at https://github.com/osops are no longer
>>> maintained.
>>>>>>>>  Our current cookbooks are at
>>>>>>>> https://github.com/rcbops/chef-cookbooks
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Joseph Breu
>>>>>>>> Deployment Engineer
>>>>>>>> Rackspace Cloud Builders
>>>>>>>> 210-312-3508
>>>>>>>>
>>>>>>>> On Jul 9, 2012, at 8:01 AM, Jay Pipes wrote:
>>>>>>>>
>>>>>>>>> Vish and Ron, just getting back to this... see inline continued
>>>>>>>>> questions for you both.
>>>>>>>>>
>>>>>>>>> On 07/02/2012 04:24 PM, Vishvananda Ishaya wrote:
>>>>>>>>>>
>>>>>>>>>> On Jul 2, 2012, at 7:28 AM, Jay Pipes wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Ron, cc'ing the openstack ML for extra eyes and opinions...
>>>>>>>>>>>
>>>>>>>>>>> So, Nati and I are looking to use either the osops chef-repo or
>>>>>>>>>>> something similar as the basis of the new TryStack zone chef
>>> deployment.
>>>>>>>>>>> I've been going through the recipes and roles and I have a
>>>>>>>>>>> question on the nova-compute *role*:
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/osops/chef-repo/blob/master/roles/nova-compu
>>>>>>>>>>> te.rb
>>>>>>>>>>>
>>>>>>>>>>> I'm wondering why the nova-api recipe is in the runlist for
>>>>>>>>>>> nova-compute?
>>>>>>>>>>
>>>>>>>>>> Because metadata needs to run on the compute hosts in the
>>>>>>>>>> default layout. This should be switched to use nova-api-metadata
>>>>>>>>>> instead of nova-api, but the split out hasn't been done yet.
>>>>>>>>>
>>>>>>>>> OK, I will work on splitting this out a bit more effectively.
>>>>>>>>>
>>>>>>>>> One additional question, though. In opening up the
>>>>>>>>> /cookbooks/nova/recipes/nova/compute.rb file, you will notice
>>>>>>>>> this at the top:
>>>>>>>>>
>>>>>>>>> include_recipe "nova::api"
>>>>>>>>>
>>>>>>>>> Therefore, unless I am mistaken, the nova-compute *role*'s
>>>>>>>>> runlist actually doesn't need to contain both nova-api AND
>>>>>>>>> nova-compute since apparently the nova-compute *recipe* already
>>>>>>>>> includes all of the nova-api recipe.
>>>>>>>>>
>>>>>>>>> Would you agree with that?
>>>>>>>>>
>>>>>>>>>>> In addition, I was wondering if y'all had considered making
>>>>>>>>>>> more use of roles instead of recipes to allow most of the
>>>>>>>>>>> attribute assignment and variation to be in the combination of
>>>>>>>>>>> roles assigned to a host, instead of recipes combined in a
>>> role?
>>>>>>>>>>>
>>>>>>>>>>> For example, right now, there is a "nova-controller" role that
>>>>>>>>>>> looks like this:
>>>>>>>>>>>
>>>>>>>>>>> name "nova-controller"
>>>>>>>>>>> description "Nova controller node (vncproxy + rabbit)"
>>>>>>>>>>> run_list(
>>>>>>>>>>> "role[base]",
>>>>>>>>>>> "recipe[nova::controller]"
>>>>>>>>>>> )
>>>>>>>>>>>
>>>>>>>>>>> Because most of the special sauce is in the nova::controller
>>>>>>>>>>> recipe, I have to go into that recipe to see what the role is
>>> composed of:
>>>>>>>>>>>
>>>>>>>>>>> include_recipe "mysql::server"
>>>>>>>>>>> include_recipe "openssh::default"
>>>>>>>>>>>
>>>>>>>>>>> include_recipe "rabbitmq::default"
>>>>>>>>>>> include_recipe "keystone::server"
>>>>>>>>>>> include_recipe "glance::registry"
>>>>>>>>>>> include_recipe "glance::api"
>>>>>>>>>>> include_recipe "nova::nova-setup"
>>>>>>>>>>> include_recipe "nova::scheduler"
>>>>>>>>>>> include_recipe "nova::api"
>>>>>>>>>>>
>>>>>>>>>>> if platform?(%w{fedora})
>>>>>>>>>>> # Fedora skipping vncproxy for right now else include_recipe
>>>>>>>>>>> "nova::vncproxy"
>>>>>>>>>>> end
>>>>>>>>>>>
>>>>>>>>>>> But what this recipe really is is an opinionated description of
>>>>>>>>>>> a "controller role". If the role was, instead, structured like
>>> so:
>>>>>>>>>>>
>>>>>>>>>>> name "nova-controller"
>>>>>>>>>>> description "Nova Controller - All major API services and
>>>>>>>>>>> control servers like MySQL and Rabbit"
>>>>>>>>>>> run_list(
>>>>>>>>>>> "role[base]",
>>>>>>>>>>> "recipe[mysql::server]",
>>>>>>>>>>> "recipe[openssh::default]",
>>>>>>>>>>> "recipe[rabbitmq::default]",
>>>>>>>>>>> "recipe[keystone::server]",
>>>>>>>>>>> "recipe[glance::api]",
>>>>>>>>>>> "recipe[glance::registry]",
>>>>>>>>>>> "recipe[nova::scheduler]",
>>>>>>>>>>> "recipe[nova::api]",
>>>>>>>>>>> )
>>>>>>>>>>>
>>>>>>>>>>> Then the deployer can more easily switch up the way they deploy
>>>>>>>>>>> OpenStack servers by merely changing the role -- say, removing
>>>>>>>>>>> the Rabbit service and putting it somewhere else -- WITHOUT
>>>>>>>>>>> having to modify a recipe in a Git submodule in the upstream
>>> cookbooks.
>>>>>>>>>>>
>>>>>>>>>>> Furthermore, if we broke out more roles -- such as "control-
>>> services"
>>>>>>>>>>> which might be MySQL and Rabbit only -- than we could make the
>>>>>>>>>>> "super roles" ,like the nova-controller role above, more of a
>>>>>>>>>>> "put this and that role together, like so:
>>>>>>>>>>>
>>>>>>>>>>> name "nova-controller"
>>>>>>>>>>> description "Nova Controller - All major API services and
>>>>>>>>>>> control servers like MySQL and Rabbit"
>>>>>>>>>>> run_list(
>>>>>>>>>>> "role[base]",
>>>>>>>>>>> "role[control_services]",
>>>>>>>>>>> "recipe[keystone::server]",
>>>>>>>>>>> "recipe[glance::api]",
>>>>>>>>>>> "recipe[glance::registry]",
>>>>>>>>>>> "recipe[nova::scheduler]",
>>>>>>>>>>> "recipe[nova::api]",
>>>>>>>>>>> )
>>>>>>>>>>
>>>>>>>>>> This all makes sense to me.  Ron?
>>>>>>>>>
>>>>>>>>> Ron, any disagreement here?
>>>>>>>>>
>>>>>>>>>>> Finally, I've noticed that there are aren't any HA options in
>>>>>>>>>>> the osops recipes -- specifically around MySQL. Are there plans
>>>>>>>>>>> to do so? We use heartbeat/Pacemaker/DRBD in the original
>>>>>>>>>>> TryStack cookbooks [1] and environments to get simple HA
>>>>>>>>>>> solutions up and would love to see those in the upstream.
>>>>>>>>>
>>>>>>>>> Either of you, any thoughts on this front?
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> -jay
>>>>>>>>>
>>>>>>>>>>> Thanks and all the best guys,
>>>>>>>>>>> -jay
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://github.com/trystack/openstack-chef/tree/stable/diablo/c
>>>>>>>>>>> ookbooks/heartbeat
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>>>>> Post to     : openstack at lists.launchpad.net
>>>>>>>>> <mailto:openstack at lists.launchpad.net>
>>>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>>> Post to     : openstack at lists.launchpad.net
>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack at lists.launchpad.net
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack at lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>>
>> Thanks,
>> Jon-Paul Sullivan ☺ Cloud Services - @hpcloud
>>
>> Postal Address: Hewlett-Packard Galway Limited, Ballybrit Business Park, Galway.
>> Registered Office: Hewlett-Packard Galway Limited, 63-74 Sir John Rogerson's Quay, Dublin 2.
>> Registered Number: 361933
>>
>> The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender.
>>
>> To any recipient of this message within HP, unless otherwise stated, you should consider this message and attachments as "HP CONFIDENTIAL".
>