Open Stack

On Thursday, September 25, 2014 at 7:30 PM, Joe Topjian wrote:
> Hi Jay,
>  
> Hiera takes the cake for my love/hate of Puppet. I try really hard to keep the number of hierarchies small and even then I find it awkward sometimes. I love the concept of Hiera, but I find it can be unintuitive. Similar to the other replies, I have a "common" hierarchy where 90% of the data is stored. The other hierarchies either override "common" or append to it. When I need to know where a parameter is ultimately configured, I find myself thinking "is that parameter common across everything or specific to a certain location or node, and if so, why did I make it specific?", then doing a "grep -R" to find where it's located, and finally thinking "oh right - that's why it's there".
>  
> Another area of Puppet that I'm finding difficult to work with is configuring HA environments. There are two main pain points here and they're pretty applicable to using Puppet with OpenStack:
>  
> The first is standing up a cluster of servers. For example MySQL and Galera. The first server in the cluster gets a blank gcomm setting. This means that I either need to have a blank gcomm setting in Hiera just for the first node or I need to remove the list of nodes from Hiera temporarily and add it back after the first node is set up. puppetdbquery (https://github.com/dalen/puppet-puppetdbquery) (very awesome module) can also be used. There are pros and cons to all of these.
>  
> After the first node, the rest of the servers are trickier because Puppet will usually start the MySQL service before all settings are in place. Then after all settings are in place, MySQL will restart. But the timing of ensuring all settings are in place for the new node to connect to the other nodes and sync with Galera is rarely right. This is fixable by doing a lot of forced ordering in Puppet, but (at least in my current setup), that would require a custom MySQL module or modifying the PuppetLabs MySQL module.
>  
> And finally, just coordinating Puppet runs of multiple cluster members is a bear. It can take up to 3 Puppet runs for two nodes to connect. Think syn, syn/ack, ack here.
I know this sounds like crazy talk, but I actually use ansible here to orchestrate puppet/chef.  The eventual consistency model really bugs me, and probably a source of ~60% of my problems.  I personally find it easier to grok an over state file[1].

I much prefer ansible controlling most everything including the ancillary openstack services (rabbitmq, percona, etc..), and relying on the solid stackforge chef/puppet community simply for openstack.  Again, I know I’m crazy.

[1] https://github.com/blueboxgroup/ursula/blob/master/site.yml  
>  
> The other HA pain point is creating many-to-one configurations, for example putting those MySQL nodes behind HAProxy. I want HAProxy to have an entry for MySQL that lists all nodes. Listing the nodes is easy: each MySQL server can simply export itself as a valid node, but what about the rest of the HAProxy MySQL config that's "singular" (for example, the "listen" and "option" settings)? Only one node can host that data or Puppet will complain about duplicate data. I don't want one node responsible for the data because I feel that creates a "pet" environment vs a "cattle" one. There are two hacks I've done to get around this:
>  
> The first is to create resources in each MySQL server with unique resource names but the same value. A simple example is:
>  
> file_line { '/etc/foobar.conf for node-1':
>   path => '/etc/foobar.conf',
>   line => 'Hello, World!',
> }
>  
> file_line { '/etc/foobar.conf for node-2':
>   path => '/etc/foobar.conf',
>   line => 'Hello, World!',
> }
>  
>  
> Now if these resources are exported and collected on a central server, Puppet will not complain about duplicate resources and the single line will exist in the foobar.conf file.
>  
> The second hack is more involved and it would take a lot of detail to show a full example. In summary, the Puppet stdlib module has a function called ensure_resource  (https://github.com/puppetlabs/puppetlabs-stdlib#ensure_resource)that will only apply a resource if it doesn't already exist. If several nodes export the same ensure_resource with the exact same settings, the single resource will be built on the server that imports them. The problem here is that if the resource is ever modified, then I must comment out the resource, run "puppet apply --noop" so it's removed from PuppetDB, make the change, then run Puppet again. If not, then the first node that runs with the new setting will export the modified resource and Puppet will complain because two resources exist with different data. This might sound very confusing and awkward and it is, but doing this allows me to store resources like firewall rules, MySQL users, HAProxy entries, etc on multiple nodes when those settings should only be applied once in the entire environment.
>  
> I think a cleaner way of doing this is to introduce service discovery into my environment, but I haven't had time to look into this in more detail.
>  
> I should mention that some of these HA pains can be resolved by just moving all of the data to the HAProxy nodes themselves. So when I want to add a new service, such as RabbitMQ, to HAProxy, I add the RabbitMQ settings to the HAProxy role/profiles. But I want HAProxy to be "dumb" about what it's hosting. I want to be able to use it in a Juju-like fashion where I can introduce any arbitrary service and HAProxy configures itself without prior knowledge of the new service.
>  
> I say *some* pains, because this isn't applicable to, say, a MySQL cluster where users and databases are shared across the entire cluster.
>   
> There are probably one or two other areas I'm forgetting, but typing the above has cleared my head. :)
>  
> In general, though, I really enjoy working with Puppet. Our current Puppet configurations allow us to stand up test OpenStack environments with little manual input as well as upgrade to newer releases of OpenStack with very little effort. For example, short of our compute nodes, all other OpenStack components run in LXC containers. Our Icehouse upgrade procedure consisted of killing those containers, making new ones based on Ubuntu 14.04, and letting Puppet do the rest.
>  
> Let me know if you'd like more details or clarification.
>  
> Thanks,
> Joe
>  
>  
> On Thu, Sep 25, 2014 at 12:12 PM, Jay Pipes <jaypipes at gmail.com (mailto:jaypipes at gmail.com)> wrote:
> > On 09/25/2014 11:45 AM, Joe Topjian wrote:
> > > Hi Mathieu,
> > >  
> > > My setup is very similar to yours. Node definitions are in site.pp and
> > > Hiera is used for all configuration. The Hiera hierarchies are also very
> > > similar.
> > >  
> > > Overall, I have a love/hate relationship with the setup. I could go on
> > > in detail, but it'd be all Puppet-specific rather than OpenStack. I'd be
> > > happy to discuss off-list.
> > >  
> > > Of if there's enough interest, I can post it here. I just don't want to
> > > muddy up this list with non-OpenStack things.
> >  
> > I certainly would enjoy a good discussion here about your love-hate relationship with your deployment tooling. I have the same relationship with the Chef work we did at AT&T and am curious whether there are Puppetisms that similarly cause grief as some Chefisms.
> >  
> > /me readies the popcorn.
> >  
> > Best,
> > -jay
> >  
> >  
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org (mailto:OpenStack-operators at lists.openstack.org)
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>  
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org (mailto:OpenStack-operators at lists.openstack.org)
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>  
>  


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140925/eb16f69f/attachment.html>