Open Stack

On Mon, Nov 2, 2015 at 10:49 AM, Paul Belanger <pabelanger at redhat.com>
wrote:

> On Mon, Nov 02, 2015 at 10:27:26AM -0800, Clark Boylan wrote:
> > On Mon, Nov 2, 2015, at 07:58 AM, Paul Belanger wrote:
> > > Greetings,
> > >
> > > I'd like to start a thread talking about the effort to upgrade our
> > > version of
> > > puppet-mysql to a newer / latest version. I know there has been some
> talk
> > > on this already, would somebody mind adding some information?
> > >
> > > I have heard 3 things:
> > >
> > >   1. Remove database support from current modules, and only use
> > >   sql_connection
> > >      strings.
> > >   2. Move everything to trove
> > >   3. Setup mysql cluster
> > I think this is mostly orthogonal to updating the puppet-mysql module
> > because 2 + 1ish (use sql_connection) is basically what we do everywhere
> > but Jenkins slave image builds, paste.openstack.org, and
> > wiki.openstack.org.
> > >
> > > Again, I don't know if there are true or not, just things I have seen
> > > people
> > > talk about.
> > >
> > > The obviously part that is missing, is _how_ we are going to do the
> > > upgrade. I
> > > know some people (clarkb?) already have some ideas on that.
> > >
> > > The reason for me asking, 2 weeks ago I offered to help the infra-cloud
> > > move
> > > forward and upgrading puppet-mysql was one of the items discussed. So,
> > > here I
> > > am, offering to do the grunt work, just need some understanding on what
> > > people
> > > want to do.
> > I was hoping that there was a forward and backward compatible
> > intermediate step we could make where old and new configs were supported
> > but I am told that isn't possible. As a result we will likely need to
> > update puppet-mysql, then all three of the above uses of puppet-mysql
> > semi atomically to keep everything working.
> >
> > Testing that image builds work before the update is simple as we can
> > just run
> > openstack-infra/project-config/nodepool/scripts/prepare_node_bare.sh to
> > see if that works. Paste.o.o is simple and has gone from local Drizzle
> > to Trove to local MySQL without much fuss so I doubt it will have much
> > trouble but a test deployment is easy if we are worried.
> >
> > The tricky one is going to be wiki.openstack.org and this is maybe where
> > the above list is relevant to the discussion. We could move it to an off
> > host database hosted by trove and not worry about updating puppet-mysql
> > in that module at all. Or we can test a deployment of wiki using the
> > newer mysql module before prod deployment. In either case we should
> > probably announce a wiki downtime prior to the upgrade, stop apache/php,
> > perform a database backup, switch to trove/run puppet with newer module,
> > restart apache/php.
> >
> So one of the big things I see, is once we bump puppet-mysql to the newer /
> latest version, all our current nodes will start consuming it (unless we
> stop
> puppet on each server).
>
> One thought I had was some like this:
>
>   1. Create /opt/puppet/modules/old, install current puppet-mysql module
> into
>      it.
>   2. Remove /etc/puppet/modules/mysql and append --modulepath to now
> include
>      /opt/puppet/modules/old
>   3. Add dynamic logic to build modulepath for puppet-mysql based on flag
> and
>      force all nodes to it.
>   4. install puppet-mysql latest into /etc/puppet/modules.
>   5. Start migration process, paste.o.o for example.
>   6. Work through all nodes.
>   7. Disable dynamic modulepath logic (can be used for other module
> upgrades).
>   8. Delete /opt/puppet/modules/old/mysql
>
> Something like this, would allow us to some how control which nodes start
> consuming the newer version of puppet-mysql.  Instead of a massive cutover
> for
> all our nodes.
>
> A possible simpler solution would be to use the load_module_metadata()
function from the stdlib module[1]. This means writing logic into each of
the modules that create a mysql::server to check for the module version and
do the right thing accordingly, e.g.

  $mysql_module_metadata = load_module_metadata('mysql', true)
  if empty($mysql_module_metadata) { #mysql module is too old to support
metadata.json
    class { 'mysql::server':
      config_hash => { 'root_password' => 'secret' }
    }
  } else { # we could check the actual version with
$mysql_module_metadata['version'] or just assume it's new
    class { 'mysql::server':
      root_password => 'secret'
    }
  }

Right now the ability to not fail when metadata.json isn't found isn't
released[2] so we'd have to update stdlib to the latest commit in master or
pressure Puppet Labs into releasing sooner than they plan to.

Colleen

[1] https://github.com/puppetlabs/puppetlabs-stdlib#load_module_metadata
[2] https://github.com/puppetlabs/puppetlabs-stdlib/pull/537
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20151102/b8174f5f/attachment-0001.html>