Open Stack

On Mon, Nov 02, 2015 at 10:27:26AM -0800, Clark Boylan wrote:
> On Mon, Nov 2, 2015, at 07:58 AM, Paul Belanger wrote:
> > Greetings,
> > 
> > I'd like to start a thread talking about the effort to upgrade our
> > version of
> > puppet-mysql to a newer / latest version. I know there has been some talk
> > on this already, would somebody mind adding some information?
> > 
> > I have heard 3 things:
> > 
> >   1. Remove database support from current modules, and only use
> >   sql_connection
> >      strings.
> >   2. Move everything to trove
> >   3. Setup mysql cluster
> I think this is mostly orthogonal to updating the puppet-mysql module
> because 2 + 1ish (use sql_connection) is basically what we do everywhere
> but Jenkins slave image builds, paste.openstack.org, and
> wiki.openstack.org.
> > 
> > Again, I don't know if there are true or not, just things I have seen
> > people
> > talk about.
> > 
> > The obviously part that is missing, is _how_ we are going to do the
> > upgrade. I
> > know some people (clarkb?) already have some ideas on that.
> > 
> > The reason for me asking, 2 weeks ago I offered to help the infra-cloud
> > move
> > forward and upgrading puppet-mysql was one of the items discussed. So,
> > here I
> > am, offering to do the grunt work, just need some understanding on what
> > people
> > want to do.
> I was hoping that there was a forward and backward compatible
> intermediate step we could make where old and new configs were supported
> but I am told that isn't possible. As a result we will likely need to
> update puppet-mysql, then all three of the above uses of puppet-mysql
> semi atomically to keep everything working.
> 
> Testing that image builds work before the update is simple as we can
> just run
> openstack-infra/project-config/nodepool/scripts/prepare_node_bare.sh to
> see if that works. Paste.o.o is simple and has gone from local Drizzle
> to Trove to local MySQL without much fuss so I doubt it will have much
> trouble but a test deployment is easy if we are worried.
> 
> The tricky one is going to be wiki.openstack.org and this is maybe where
> the above list is relevant to the discussion. We could move it to an off
> host database hosted by trove and not worry about updating puppet-mysql
> in that module at all. Or we can test a deployment of wiki using the
> newer mysql module before prod deployment. In either case we should
> probably announce a wiki downtime prior to the upgrade, stop apache/php,
> perform a database backup, switch to trove/run puppet with newer module,
> restart apache/php.
> 
So one of the big things I see, is once we bump puppet-mysql to the newer /
latest version, all our current nodes will start consuming it (unless we stop
puppet on each server).

One thought I had was some like this:

  1. Create /opt/puppet/modules/old, install current puppet-mysql module into
     it.
  2. Remove /etc/puppet/modules/mysql and append --modulepath to now include
     /opt/puppet/modules/old
  3. Add dynamic logic to build modulepath for puppet-mysql based on flag and
     force all nodes to it.
  4. install puppet-mysql latest into /etc/puppet/modules.
  5. Start migration process, paste.o.o for example.
  6. Work through all nodes.
  7. Disable dynamic modulepath logic (can be used for other module upgrades).
  8. Delete /opt/puppet/modules/old/mysql

Something like this, would allow us to some how control which nodes start
consuming the newer version of puppet-mysql.  Instead of a massive cutover for
all our nodes.

> Hope this helps,
> Clark
> 
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra