Open Stack

I agree with the strategy put forward by Clark and Jeremy. As for the
dynamic control Paul is talking about creating, I think it does have
value, but we can get away with this upgrade without it. After we're on
puppet apply, building automation so that different nodes can use
different modules makes more sense.

It is my goal to work on puppet apply before the puppet-mysql upgrade.
We have elasticsearch and gerrit upgrades coming down the pipe as well.
So I'm not sure when we can do this work or who can do it.

-- 
  Spencer Krum
  nibz at spencerkrum.com

On Mon, Nov 2, 2015, at 10:49 AM, Paul Belanger wrote:
> On Mon, Nov 02, 2015 at 10:27:26AM -0800, Clark Boylan wrote:
> > On Mon, Nov 2, 2015, at 07:58 AM, Paul Belanger wrote:
> > > Greetings,
> > > 
> > > I'd like to start a thread talking about the effort to upgrade our
> > > version of
> > > puppet-mysql to a newer / latest version. I know there has been some talk
> > > on this already, would somebody mind adding some information?
> > > 
> > > I have heard 3 things:
> > > 
> > >   1. Remove database support from current modules, and only use
> > >   sql_connection
> > >      strings.
> > >   2. Move everything to trove
> > >   3. Setup mysql cluster
> > I think this is mostly orthogonal to updating the puppet-mysql module
> > because 2 + 1ish (use sql_connection) is basically what we do everywhere
> > but Jenkins slave image builds, paste.openstack.org, and
> > wiki.openstack.org.
> > > 
> > > Again, I don't know if there are true or not, just things I have seen
> > > people
> > > talk about.
> > > 
> > > The obviously part that is missing, is _how_ we are going to do the
> > > upgrade. I
> > > know some people (clarkb?) already have some ideas on that.
> > > 
> > > The reason for me asking, 2 weeks ago I offered to help the infra-cloud
> > > move
> > > forward and upgrading puppet-mysql was one of the items discussed. So,
> > > here I
> > > am, offering to do the grunt work, just need some understanding on what
> > > people
> > > want to do.
> > I was hoping that there was a forward and backward compatible
> > intermediate step we could make where old and new configs were supported
> > but I am told that isn't possible. As a result we will likely need to
> > update puppet-mysql, then all three of the above uses of puppet-mysql
> > semi atomically to keep everything working.
> > 
> > Testing that image builds work before the update is simple as we can
> > just run
> > openstack-infra/project-config/nodepool/scripts/prepare_node_bare.sh to
> > see if that works. Paste.o.o is simple and has gone from local Drizzle
> > to Trove to local MySQL without much fuss so I doubt it will have much
> > trouble but a test deployment is easy if we are worried.
> > 
> > The tricky one is going to be wiki.openstack.org and this is maybe where
> > the above list is relevant to the discussion. We could move it to an off
> > host database hosted by trove and not worry about updating puppet-mysql
> > in that module at all. Or we can test a deployment of wiki using the
> > newer mysql module before prod deployment. In either case we should
> > probably announce a wiki downtime prior to the upgrade, stop apache/php,
> > perform a database backup, switch to trove/run puppet with newer module,
> > restart apache/php.
> > 
> So one of the big things I see, is once we bump puppet-mysql to the newer
> /
> latest version, all our current nodes will start consuming it (unless we
> stop
> puppet on each server).
> 
> One thought I had was some like this:
> 
>   1. Create /opt/puppet/modules/old, install current puppet-mysql module
>   into
>      it.
>   2. Remove /etc/puppet/modules/mysql and append --modulepath to now
>   include
>      /opt/puppet/modules/old
>   3. Add dynamic logic to build modulepath for puppet-mysql based on flag
>   and
>      force all nodes to it.
>   4. install puppet-mysql latest into /etc/puppet/modules.
>   5. Start migration process, paste.o.o for example.
>   6. Work through all nodes.
>   7. Disable dynamic modulepath logic (can be used for other module
>   upgrades).
>   8. Delete /opt/puppet/modules/old/mysql
> 
> Something like this, would allow us to some how control which nodes start
> consuming the newer version of puppet-mysql.  Instead of a massive
> cutover for
> all our nodes.
> 
> > Hope this helps,
> > Clark
> > 
> > _______________________________________________
> > OpenStack-Infra mailing list
> > OpenStack-Infra at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> 
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra