<span class="Apple-style-span" style="border-collapse: collapse; font-family: arial, sans-serif; font-size: 13px; ">Time for another update....<div><br></div><div>The crowbar is (finally) out, it made some splash at OSCON, but in case you missed it, you can find it here: <a href="https://github.com/dellcloudedge/crowbar" target="_blank" style="color: rgb(0, 0, 204); ">https://github.com/dellcloudedge/crowbar</a></div>
<div> </div><div>In case you really missed it - crowbar provides a bare metal to a fully deployed glance, nova and/or swift install system. It will take care of PXE booting the systems and walking them through the paces (discovery, hw-install and configure, os-install and configuration). Once the base OS is ready for the set of machines to be installed, clusters of compute or storage services are deployed and prep-ed for use.</div>
<div><br></div><div>You can take crowbar for a spin in a virutal environment or on bare metal hw.</div><div>take it for a spin, drop us a note.</div><div><br></div><div>(p.s. it subsumes (by a lot) the previous swift-only recipes from this thread, and includes some bug fixes)</div>
</span><br><div class="gmail_quote">On Mon, May 23, 2011 at 1:41 PM, andi abes <span dir="ltr"><<a href="mailto:andi.abes@gmail.com">andi.abes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div></div><div class="h5"><div class="gmail_quote"><br><br><div>It took a while, but finally:</div><div><a href="https://github.com/dellcloudedge/openstack-swift" target="_blank">https://github.com/dellcloudedge/openstack-swift</a></div>
<div><br></div>
<div>Jay, I've added a swift-proxy-acct and swift-proxy (without account management).</div>
<div><br></div><div>This cookbook is an advanced "leak" of for swift, soon to be followed with a leak of a nova cookbook. The full crowbar that was mentioned is on its way...</div><div><br></div><div>To use these recipes (with default settings) you just need to pick your storage nodes and 1 or more proxies. Then assign the appropriate roles (swift-storage swift-proxy or swift-proxy-acct) using the chef ui or a knife command. Choose one of the nodes and assign it the swift-node-compute. and the swift cluster is built (because of async nature of multi-node deployments, it might require a few chef-client runs while the ring files are generated and pushed around.</div>
<div><br></div><div>have a spin. eager to hear comments.</div><div><div></div><div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><br><br><div class="gmail_quote">On Mon, May 2, 2011 at 11:36 AM, andi abes <span dir="ltr"><<a href="mailto:andi.abes@gmail.com" target="_blank">andi.abes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Jay,<div><br></div><div>hmmm, interesting point about account management in the proxy. Guess you're suggesting that you have 2 flavors of a proxy server - one with account management enabled and one without?</div>
<div>
Is the main concern here security - you'd have more controls on the account management servers? Or is this about something else? </div><div><br></div><div>About ring-compute:</div><div>so there are 2 concerns I was thinking about with rings - a) make sure the ring information is consistent across all the nodes in the cluster, and b) try not to lose the ring info.</div>
<div><br></div><div>The main driver to have only 1 ring compute node was a). the main concern being guaranteeing consistency of the ring data among all nodes without causing too strong coupling to the underlying mechanisms used to build the ring.</div>
<div>For example - if 2 new rings are created independently, then the order in which disks are added to the ring should be consistent (assuming that the disk/partition allocation algorithm is sensitive to ordering). Which implies that the query to chef should always return data in exactly the same order.</div>
<div>If also would require that the ring building (and mandate that it will never be changed) does not use any heuristics that are time or machine dependent (I _think_ that right now that is the case, but I would rather not depend on it).</div>
<div><br></div><div>I was thinking that these restrictions can be avoided easily by making sure that only 1 node computes the ring. To make sure that b) (don't lose the ring) is addressed - the ring is copied around.</div>
<div>If the ring compute node fails, then any other node can be used to seed a new compute ring without any loss.</div><div>Does that make sense?</div><div><br></div><div><br></div><div>Right now I'm using a snapshot deb package built from bzr266. Changing the source of the bits is pretty esay... (and installing the deb includes the utilities you mentioned)</div>
<div><br></div><div><br></div><div>Re: load balancers:</div><div>What you're proposing makes perfect sense. Chef is pretty modular. So the swift configuration recipe focuses on setting up swift - not the whole environment. It would make sense to deploy some load balancer, firewall appliance etc in an environment. However, these would be add-ons to the basic swift configuration. </div>
<div>A simple way to achieve this would be to have a recipe that would query the chef server for all nodes which have the swift-proxy role, and add them as internal addresses for the load balancer of your choice.</div><div>
(e.g. : <a href="http://wiki.opscode.com/display/chef/Search#Search-FindNodeswithaRoleintheExpandedRunList" target="_blank">http://wiki.opscode.com/display/chef/Search#Search-FindNodeswithaRoleintheExpandedRunList</a>)</div>
<div><br></div><font color="#888888">
<div><br></div><div>a.</div></font><div><div></div><div><div><br></div><div><br></div><div><br></div><div><div class="gmail_quote">On Sun, May 1, 2011 at 10:14 AM, Jay Payne <span dir="ltr"><<a href="mailto:letterj@gmail.com" target="_blank">letterj@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Andi,<br>
<br>
This looks great. I do have some thoughts/questions.<br>
<br>
If you are using 1.3, do you have a separate role for the management<br>
functionality in the proxy? It's not a good idea to have all your<br>
proxy servers running in management mode (unless you only have one<br>
proxy).<br>
<br>
Why only 1 ring-compute node? If that node is lost or unavailable do<br>
you loose your ring-builder files?<br>
<br>
When I create an environment I always setup utilities like st,<br>
get-nodes, stats-report, and a simple functional test script on a<br>
server to help troubleshoot and manage the cluster(s).<br>
<br>
Are you using packages or eggs to deploy the swift code? If your<br>
using packages, are you building them yourself or using the ones from<br>
launchpad?<br>
<br>
If you have more than three proxy servers, do you plan on using load balancers?<br>
<br>
<br>
Thanks<br>
<font color="#888888">--J<br>
</font><div><div></div><div><br>
<br>
<br>
<br>
On Sun, May 1, 2011 at 8:37 AM, andi abes <<a href="mailto:andi.abes@gmail.com" target="_blank">andi.abes@gmail.com</a>> wrote:<br>
> Judd,<br>
> Sorry, today I won't be around. I'd love to hear feedback and suggestions<br>
> on what I have so far ( I'm not 100% sure when I can make the fully<br>
> available, but I'm hoping this is very soon). I'm running with swift 1.3 on<br>
> ubuntu 10.10.<br>
> I'm using the environment pattern in chef - when nodes search for their<br>
> peers a predicate comparing the node[:swift][:config][:environment] to the<br>
> corresponding value on the prospective peer. A "default" value is assigned<br>
> to this by the default recipe's attributes, so if only 1 cluster is present,<br>
> all nodes are eligible. For example, when proxy recipe creates the memcached<br>
> list of addresses, it searches for all the other nodes with the swift-proxy<br>
> role assigned to it anded with the above.<br>
> Is that what you meant about having a classifier?<br>
> To the basic roles you've described (base, storage and proxy) I've added 1:<br>
> ring-compute. This role should be assigned to only 1 node on top of either<br>
> the storage or proxy. This server will recompute the rings whenever the set<br>
> of storage servers change (new disks or machines added or existing ones<br>
> removed). It uses information on the affected machines (ip, disk and zone<br>
> assignment) to update the ring. Once the ring is updated, it is rsynced to<br>
> all the other nodes in the cluster.<br>
> [ this is achieved with a LWRP as I mentioned earlier, which first parses<br>
> the current ring info, then compares it to the current set of disks. It<br>
> notifies chef if any changes were made, so that the rsync actions can be<br>
> triggered]<br>
> At this point, all disks are added to all rings. Though it should be easy to<br>
> make this conditional on an attribute on the node/disk or some heuristic.<br>
> The 'base role' is also a bit more extensive than your description, but<br>
> needs a bit more work. The recipe uses a swift_disk LWRP to ensure the<br>
> partition table matches the configuration. The LWRP accepts an array<br>
> describing the desired partition table, and allows using a :remaining token<br>
> to indicate using what's left (should only be used on the last partition).<br>
> At this point it, the recipe is pretty hard coded. It assumes that /dev/sdb<br>
> is dedicated to storage. it just requires a hard coded single partition,<br>
> that uses the whole disk. If it's not present, or different, the LWRP<br>
> creates a BSD label, a single partition and xfs filesystem. Once this is<br>
> done, the available disk is 'published' as node attributes. If the current<br>
> state of the system matches the desired state, nothing is modified.<br>
> The proxy and storage roles, do as you'd expect. install the relevant<br>
> packages (including memcached on the proxy using the opscode recipe) and<br>
> plunk in the server/cluster specific info into the relevant config file<br>
> templates.<br>
> What's totally not yet addressed:<br>
> - starting services<br>
> - prep-ing the authentication<br>
> - injecting disk configuration.<br>
> This cookbook can be used by it self, but it is more powerful when used in<br>
> conjunction with the crowbar proposal mechanism. crowbar basically allows<br>
> you to define the qualifications for a system to be assigned a given role,<br>
> edit the automatic role assignment and configuration of the cluster and then<br>
> materialize the cluster based on these values by driving chef. Part of the<br>
> planned crowbar capability is performing automatic disk/zone allocation,<br>
> based on network topology and connectivity. In the mean time, the allocation<br>
> is done when the storage node is created.<br>
> Currently, a proposal can provide values for the following:<br>
> "cluster_hash", "cluster_admin_pw", "replicas", and user/group to run swift<br>
> as. I'm really curious to hear what other configuration parameters might be<br>
> useful<br>
> I'm really curious to hear some feedback about this.<br>
> and sorry about the timing, I'm in the boston area and it's finally nice<br>
> around here... hence other weekend plans.<br>
><br>
><br>
> On Thu, Apr 28, 2011 at 11:25 PM, Judd Maltin <<a href="mailto:judd@newgoliath.com" target="_blank">judd@newgoliath.com</a>> wrote:<br>
>><br>
>> Hey andi,<br>
>> I'm psyched to collaborate on this. I'm a busy guy, but I'm dedicating<br>
>> Sunday to this. So if you have time Sunday, that would be best to catch up<br>
>> via IRC, IM or voice.<br>
>> Having a node classifier of some sort is critical.<br>
>> -Judd<br>
>><br>
>> Judd Maltin<br>
>> <a href="tel:%2B1%20917%20882%201270" value="+19178821270" target="_blank">+1 917 882 1270</a><br>
>> Happiness is a straight line and a goal. -fn<br>
>> On Apr 28, 2011, at 11:59, andi abes <<a href="mailto:andi.abes@gmail.com" target="_blank">andi.abes@gmail.com</a>> wrote:<br>
>><br>
>> Judd,<br>
>> Ok. Here are some of the thoughts I've had (and have mostly working, but<br>
>> hitting some swift snags..) Maybe we can collaborate on this?<br>
>> Since Chef promotes idempotent operations and cookbooks, I put some effort<br>
>> in making sure that changes are only made if they're required, particularly<br>
>> around destructive or expensive operations.<br>
>> The 2 main cases are:<br>
>> - partitioning disks, which is obviously destructive<br>
>> - building / rebuilding the ring files - rebalancing the ring is<br>
>> relatively expensive.<br>
>> for both cases I've built a LWRP which reads the current state of affairs,<br>
>> and decides if and what are the required changes. For disks, I'm using '<br>
>> 'parted', which produces machine friendly output. For ring files I'm using<br>
>> the output from ring-builder.<br>
>> the LWRP are driven by recipes which inspect databag - very similar to<br>
>> your approach. However, they also utilize inventory information about<br>
>> available resources created by crowbar in chef during the initial bring up<br>
>> of the deployment.<br>
>> (As a side note - Dell has announced plans to opensource most of crowbar,<br>
>> but there's legalese involved)<br>
>><br>
>> I'd be more than happy to elaborate and collaborate on this !<br>
>><br>
>> a<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> On Thu, Apr 28, 2011 at 11:35 AM, Judd Maltin <<a href="mailto:judd@newgoliath.com" target="_blank">judd@newgoliath.com</a>> wrote:<br>
>>><br>
>>> Hi Andi,<br>
>>><br>
>>> Indeed, the swift recipes hadn't been updated since mid 2010, so I pushed<br>
>>> forward with my own.<br>
>>><br>
>>> Thanks!<br>
>>> -judd<br>
>>><br>
>>> On Thu, Apr 28, 2011 at 10:03 AM, andi abes <<a href="mailto:andi.abes@gmail.com" target="_blank">andi.abes@gmail.com</a>> wrote:<br>
>>>><br>
>>>> Judd,<br>
>>>> this is a great idea... actually so great, that some folks @Dell and<br>
>>>> OpsCode, me included, have been working on it.<br>
>>>> Have a peek on<br>
>>>> : <a href="https://github.com/opscode/openstack-cookbooks/tree/master/cookbooks" target="_blank">https://github.com/opscode/openstack-cookbooks/tree/master/cookbooks</a><br>
>>>> This effort is also being included into Crowbar (take a peek<br>
>>>> here: <a href="http://robhirschfeld.com/tag/crowbar/" target="_blank">http://robhirschfeld.com/tag/crowbar/</a>) which adds the steps needed to<br>
>>>> start with bare metal (rather than installed OS), then using chef to get to<br>
>>>> a working Open stack deployment.<br>
>>>> (if you're at the design meeting, there are demos scheduled).<br>
>>>> That said - I'm updating the swift cookbook, and hope to update github<br>
>>>> soon.<br>
>>>> a.<br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> On Wed, Apr 27, 2011 at 9:55 PM, Jay Payne <<a href="mailto:letterj@gmail.com" target="_blank">letterj@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>> Judd,<br>
>>>>><br>
>>>>> I'm not that familiar with Chef (I'll do some research) but I have a<br>
>>>>> couple of questions and some thoughts:<br>
>>>>><br>
>>>>> 1. Is this for a multi-server environment?<br>
>>>>> 2. Are all your proxy nodes going to have "allow_account_management =<br>
>>>>> true" in the configs? It might be a good idea to have a second proxy<br>
>>>>> config for account management only<br>
>>>>> 3. Have you looked at using swauth instead of auth?<br>
>>>>> 4. Have you thought about an admin or client node that has utilities<br>
>>>>> on it like st and stats-report?<br>
>>>>> 5. How where will you do on-going ring management or changes?<br>
>>>>> 6. I would think about including some type of functional test at the<br>
>>>>> end of the deployment process to verify everything was created<br>
>>>>> properly and that all nodes can communicate.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> --J<br>
>>>>><br>
>>>>> On Wed, Apr 27, 2011 at 6:18 PM, Judd Maltin <<a href="mailto:judd@newgoliath.com" target="_blank">judd@newgoliath.com</a>><br>
>>>>> wrote:<br>
>>>>> > Hi Folks,<br>
>>>>> ><br>
>>>>> > I've been hacking away at creating an automated deployment system for<br>
>>>>> > Swift<br>
>>>>> > using Chef. I'd like to drop a design idea on you folks (most of<br>
>>>>> > which I've<br>
>>>>> > already implemented) and get feedback from this esteemed group.<br>
>>>>> ><br>
>>>>> > My end goal is to have a "manifest" (apologies to Puppet) which will<br>
>>>>> > define<br>
>>>>> > an entire swift cluster, deploy it automatically, and allow edits to<br>
>>>>> > the<br>
>>>>> > ingredients to manage the cluster. In this case, a "manifest" is a<br>
>>>>> > combination of a chef databag describing the swift settings, and a<br>
>>>>> > spiceweasel infrastructure.yaml file describing the OS configuration.<br>
>>>>> ><br>
>>>>> > Ingredients:<br>
>>>>> > - swift cookbook with base, proxy and server recipes. proxy nodes<br>
>>>>> > also<br>
>>>>> > (provisionally) contain auth services. storage nodes handle object,<br>
>>>>> > container and account services.<br>
>>>>> > -- Base recipe handles common package install, OS user creation.<br>
>>>>> > Sets up<br>
>>>>> > keys.<br>
>>>>> > -- Proxy recipe handles proxy nodes: network config, package install,<br>
>>>>> > memcache config, proxy and auth package config, user creation, ring<br>
>>>>> > management (including builder file backup), user management<br>
>>>>> > -- Storage recipe handles storage nodes: network config, storage<br>
>>>>> > device<br>
>>>>> > config, package install, ring management.<br>
>>>>> ><br>
>>>>> > - chef databag that describes a swift cluster (eg:<br>
>>>>> > mycluster_databag.json)<br>
>>>>> > -- proxy config settings<br>
>>>>> > -- memcached settings<br>
>>>>> > -- settings for all rings and devices<br>
>>>>> > -- basic user settings<br>
>>>>> > -- account management<br>
>>>>> ><br>
>>>>> > - chef "spiceweasel" file that auto-vivifies the infrastructure: (eg:<br>
>>>>> > mycluster_infra.yaml)<br>
>>>>> > -- uploads cookbooks<br>
>>>>> > -- uploads roles<br>
>>>>> > -- uploads the cluster's databag<br>
>>>>> > -- kicks off node provisioning by requesting from infrastructure API<br>
>>>>> > (ec2 or<br>
>>>>> > what have you) the following:<br>
>>>>> > --- chef roles applied (role[swift:proxy] or role[swift:storage])<br>
>>>>> > --- server flavor<br>
>>>>> > --- storage device configs<br>
>>>>> > --- hostname<br>
>>>>> > --- proxy and storage network details<br>
>>>>> ><br>
>>>>> > By calling this spiceweasel file, the infrastructure can leap into<br>
>>>>> > existence.<br>
>>>>> ><br>
>>>>> > I'm more or less done with all this stuff - and I'd really appreciate<br>
>>>>> > conceptual feedback before I take out all the non-sense code I have<br>
>>>>> > in the<br>
>>>>> > files and publish.<br>
>>>>> ><br>
>>>>> > Many thanks! Happy spring, northern hemispherians!<br>
>>>>> > -judd<br>
>>>>> ><br>
>>>>> > Judd Maltin<br>
>>>>> > T: <a href="tel:917-882-1270" value="+19178821270" target="_blank">917-882-1270</a><br>
>>>>> > F: <a href="tel:501-694-7809" value="+15016947809" target="_blank">501-694-7809</a><br>
>>>>> > A loving heart is never wrong.<br>
>>>>> ><br>
>>>>> ><br>
>>>>> ><br>
>>>>> ><br>
>>>>> > _______________________________________________<br>
>>>>> > Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>> > Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
>>>>> > Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>> > More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
>>>>> ><br>
>>>>> ><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
>>>>> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Judd Maltin<br>
>>> T: <a href="tel:917-882-1270" value="+19178821270" target="_blank">917-882-1270</a><br>
>>> F: <a href="tel:501-694-7809" value="+15016947809" target="_blank">501-694-7809</a><br>
>>> A loving heart is never wrong.<br>
>>><br>
>>><br>
>>><br>
>><br>
><br>
><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br>
</div></div></div><br>
</div></div></blockquote></div><br>