Judd,<div><br></div><div> Sorry, today I won't be around. I'd love to hear feedback and suggestions on what I have so far ( I'm not 100% sure when I can make the fully available, but I'm hoping this is very soon). I'm running with swift 1.3 on ubuntu 10.10.</div>
<div>I'm using the environment pattern in chef - when nodes search for their peers a predicate comparing the node[:swift][:config][:environment] to the corresponding value on the prospective peer. A "default" value is assigned to this by the default recipe's attributes, so if only 1 cluster is present, all nodes are eligible. For example, when proxy recipe creates the memcached list of addresses, it searches for all the other nodes with the swift-proxy role assigned to it anded with the above.</div>
<div> Is that what you meant about having a classifier?</div><div><br></div><div>To the basic roles you've described (base, storage and proxy) I've added 1: ring-compute. This role should be assigned to only 1 node on top of either the storage or proxy. This server will recompute the rings whenever the set of storage servers change (new disks or machines added or existing ones removed). It uses information on the affected machines (ip, disk and zone assignment) to update the ring. Once the ring is updated, it is rsynced to all the other nodes in the cluster.</div>
<div>[ this is achieved with a LWRP as I mentioned earlier, which first parses the current ring info, then compares it to the current set of disks. It notifies chef if any changes were made, so that the rsync actions can be triggered]</div>
<div>At this point, all disks are added to all rings. Though it should be easy to make this conditional on an attribute on the node/disk or some heuristic.</div><div><br></div><div>The 'base role' is also a bit more extensive than your description, but needs a bit more work. The recipe uses a swift_disk LWRP to ensure the partition table matches the configuration. The LWRP accepts an array describing the desired partition table, and allows using a :remaining token to indicate using what's left (should only be used on the last partition).</div>
<div>At this point it, the recipe is pretty hard coded. It assumes that /dev/sdb is dedicated to storage. it just requires a hard coded single partition, that uses the whole disk. If it's not present, or different, the LWRP creates a BSD label, a single partition and xfs filesystem. Once this is done, the available disk is 'published' as node attributes. If the current state of the system matches the desired state, nothing is modified.</div>
<div><br></div><div>The proxy and storage roles, do as you'd expect. install the relevant packages (including memcached on the proxy using the opscode recipe) and plunk in the server/cluster specific info into the relevant config file templates.</div>
<div><br></div><div>What's totally not yet addressed:</div><div>- starting services</div><div>- prep-ing the authentication</div><div>- injecting disk configuration.</div><div><br></div><div>This cookbook can be used by it self, but it is more powerful when used in conjunction with the crowbar proposal mechanism. crowbar basically allows you to define the qualifications for a system to be assigned a given role, edit the automatic role assignment and configuration of the cluster and then materialize the cluster based on these values by driving chef. Part of the planned crowbar capability is performing automatic disk/zone allocation, based on network topology and connectivity. In the mean time, the allocation is done when the storage node is created.</div>
<div><br></div><div>Currently, a proposal can provide values for the following: "cluster_hash", "cluster_admin_pw", "replicas", and user/group to run swift as. I'm really curious to hear what other configuration parameters might be useful</div>
<div><div><br></div></div><div>I'm really curious to hear some feedback about this.</div><div>and sorry about the timing, I'm in the boston area and it's finally nice around here... hence other weekend plans.</div>
<div><br></div><div><br><br><div class="gmail_quote">On Thu, Apr 28, 2011 at 11:25 PM, Judd Maltin <span dir="ltr"><<a href="mailto:judd@newgoliath.com">judd@newgoliath.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="#FFFFFF"><div>Hey andi,</div><div><br></div><div>I'm psyched to collaborate on this. I'm a busy guy, but I'm dedicating Sunday to this. So if you have time Sunday, that would be best to catch up via IRC, IM or voice.</div>
<div><br></div><div>Having a node classifier of some sort is critical.</div><div><br></div><div>-Judd<br><font color="#888888"><br>Judd Maltin<div><a href="tel:%2B1%20917%20882%201270" value="+19178821270" target="_blank">+1 917 882 1270</a></div>
<div>Happiness is a straight line and a goal. -fn</div></font></div><div><div></div><div class="h5"><div><br>On Apr 28, 2011, at 11:59, andi abes <<a href="mailto:andi.abes@gmail.com" target="_blank">andi.abes@gmail.com</a>> wrote:<br>
<br></div><div></div><blockquote type="cite"><div>Judd,<div><br></div><div> Ok. Here are some of the thoughts I've had (and have mostly working, but hitting some swift snags..) Maybe we can collaborate on this?</div>
<div><br></div><div>Since Chef promotes idempotent operations and cookbooks, I put some effort in making sure that changes are only made if they're required, particularly around destructive or expensive operations.</div>
<div>The 2 main cases are:</div><div>- partitioning disks, which is obviously destructive</div><div>- building / rebuilding the ring files - rebalancing the ring is relatively expensive.</div><div><br></div><div>for both cases I've built a LWRP which reads the current state of affairs, and decides if and what are the required changes. For disks, I'm using '</div>
<div>'parted', which produces machine friendly output. For ring files I'm using the output from ring-builder.</div><div><br></div><div>the LWRP are driven by recipes which inspect databag - very similar to your approach. However, they also utilize inventory information about available resources created by crowbar in chef during the initial bring up of the deployment.</div>
<div><br></div><div>(As a side note - Dell has announced plans to opensource most of crowbar, but there's legalese involved)</div><div><br></div><div><br></div><div>I'd be more than happy to elaborate and collaborate on this !</div>
<div><br></div><div><br></div><div>a</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br><br><div class="gmail_quote">On Thu, Apr 28, 2011 at 11:35 AM, Judd Maltin <span dir="ltr"><<a href="mailto:judd@newgoliath.com" target="_blank"></a><a href="mailto:judd@newgoliath.com" target="_blank">judd@newgoliath.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Andi,<br><br>Indeed, the swift recipes hadn't been updated since mid 2010, so I pushed forward with my own.<br>
<br>Thanks!<br><font color="#888888">-judd</font><div><div></div><div><br><br><div class="gmail_quote">On Thu, Apr 28, 2011 at 10:03 AM, andi abes <span dir="ltr"><<a href="mailto:andi.abes@gmail.com" target="_blank"></a><a href="mailto:andi.abes@gmail.com" target="_blank">andi.abes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Judd, <div><br></div><div> this is a great idea... actually so great, that some folks @Dell and OpsCode, me included, have been working on it.</div>
<div><br></div><div>Have a peek on : <a href="https://github.com/opscode/openstack-cookbooks/tree/master/cookbooks" target="_blank"></a><a href="https://github.com/opscode/openstack-cookbooks/tree/master/cookbooks" target="_blank">https://github.com/opscode/openstack-cookbooks/tree/master/cookbooks</a></div>
<div><br></div><div>This effort is also being included into Crowbar (take a peek here: <a href="http://robhirschfeld.com/tag/crowbar/" target="_blank"></a><a href="http://robhirschfeld.com/tag/crowbar/" target="_blank">http://robhirschfeld.com/tag/crowbar/</a>) which adds the steps needed to start with bare metal (rather than installed OS), then using chef to get to a working Open stack deployment.</div>
<div>(if you're at the design meeting, there are demos scheduled).</div><div><br></div><div>That said - I'm updating the swift cookbook, and hope to update github soon.</div><div><br></div><font color="#888888"><div>
a.</div></font><div><div></div><div><div><br></div>
<div><br></div><div><br></div><div><br></div><div><br><div class="gmail_quote">On Wed, Apr 27, 2011 at 9:55 PM, Jay Payne <span dir="ltr"><<a href="mailto:letterj@gmail.com" target="_blank"></a><a href="mailto:letterj@gmail.com" target="_blank">letterj@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Judd,<br>
<br>
I'm not that familiar with Chef (I'll do some research) but I have a<br>
couple of questions and some thoughts:<br>
<br>
1. Is this for a multi-server environment?<br>
2. Are all your proxy nodes going to have "allow_account_management =<br>
true" in the configs? It might be a good idea to have a second proxy<br>
config for account management only<br>
3. Have you looked at using swauth instead of auth?<br>
4. Have you thought about an admin or client node that has utilities<br>
on it like st and stats-report?<br>
5. How where will you do on-going ring management or changes?<br>
6. I would think about including some type of functional test at the<br>
end of the deployment process to verify everything was created<br>
properly and that all nodes can communicate.<br>
<font color="#888888"><br>
<br>
<br>
--J<br>
</font><div><div></div><div><br>
On Wed, Apr 27, 2011 at 6:18 PM, Judd Maltin <<a href="mailto:judd@newgoliath.com" target="_blank"></a><a href="mailto:judd@newgoliath.com" target="_blank">judd@newgoliath.com</a>> wrote:<br>
> Hi Folks,<br>
><br>
> I've been hacking away at creating an automated deployment system for Swift<br>
> using Chef. I'd like to drop a design idea on you folks (most of which I've<br>
> already implemented) and get feedback from this esteemed group.<br>
><br>
> My end goal is to have a "manifest" (apologies to Puppet) which will define<br>
> an entire swift cluster, deploy it automatically, and allow edits to the<br>
> ingredients to manage the cluster. In this case, a "manifest" is a<br>
> combination of a chef databag describing the swift settings, and a<br>
> spiceweasel infrastructure.yaml file describing the OS configuration.<br>
><br>
> Ingredients:<br>
> - swift cookbook with base, proxy and server recipes. proxy nodes also<br>
> (provisionally) contain auth services. storage nodes handle object,<br>
> container and account services.<br>
> -- Base recipe handles common package install, OS user creation. Sets up<br>
> keys.<br>
> -- Proxy recipe handles proxy nodes: network config, package install,<br>
> memcache config, proxy and auth package config, user creation, ring<br>
> management (including builder file backup), user management<br>
> -- Storage recipe handles storage nodes: network config, storage device<br>
> config, package install, ring management.<br>
><br>
> - chef databag that describes a swift cluster (eg: mycluster_databag.json)<br>
> -- proxy config settings<br>
> -- memcached settings<br>
> -- settings for all rings and devices<br>
> -- basic user settings<br>
> -- account management<br>
><br>
> - chef "spiceweasel" file that auto-vivifies the infrastructure: (eg:<br>
> mycluster_infra.yaml)<br>
> -- uploads cookbooks<br>
> -- uploads roles<br>
> -- uploads the cluster's databag<br>
> -- kicks off node provisioning by requesting from infrastructure API (ec2 or<br>
> what have you) the following:<br>
> --- chef roles applied (role[swift:proxy] or role[swift:storage])<br>
> --- server flavor<br>
> --- storage device configs<br>
> --- hostname<br>
> --- proxy and storage network details<br>
><br>
> By calling this spiceweasel file, the infrastructure can leap into<br>
> existence.<br>
><br>
> I'm more or less done with all this stuff - and I'd really appreciate<br>
> conceptual feedback before I take out all the non-sense code I have in the<br>
> files and publish.<br>
><br>
> Many thanks! Happy spring, northern hemispherians!<br>
> -judd<br>
><br>
> Judd Maltin<br>
> T: <a href="tel:917-882-1270" value="+19178821270" target="_blank">917-882-1270</a><br>
> F: <a href="tel:501-694-7809" value="+15016947809" target="_blank">501-694-7809</a><br>
> A loving heart is never wrong.<br>
><br>
><br>
><br>
><br>
</div></div><div><div></div><div>> _______________________________________________<br>
> Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank"></a><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank"></a><a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank"></a><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank"></a><a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
><br>
><br>
<br>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank"></a><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank"></a><a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank"></a><a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank"></a><a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><br></div></div>-- <br><div><div></div><div>Judd Maltin<br>T: <a href="tel:917-882-1270" value="+19178821270" target="_blank">917-882-1270</a><br>F: <a href="tel:501-694-7809" value="+15016947809" target="_blank">501-694-7809</a><br>
A loving heart is never wrong.<br><br><br><br>
</div></div></blockquote></div><br></div>
</div></blockquote></div></div></div></blockquote></div><br></div>