Open Stack

On Mon, Nov 02, 2015 at 11:05:04AM -0800, Spencer Krum wrote:
> I agree with the strategy put forward by Clark and Jeremy. As for the
> dynamic control Paul is talking about creating, I think it does have
> value, but we can get away with this upgrade without it. After we're on
> puppet apply, building automation so that different nodes can use
> different modules makes more sense.
> 
I am not sure how we can pass the gate with out it.  Once we bump puppet-mysql
to 3.6.1, everything will break.  Depends-On won't help us, since we'll have a
bunch of circular dependencies.

> It is my goal to work on puppet apply before the puppet-mysql upgrade.
> We have elasticsearch and gerrit upgrades coming down the pipe as well.
> So I'm not sure when we can do this work or who can do it.
> 
Right, so waiting on puppet apply changes we are blocking the puppet-mysql
upgrade path. What do you think your timelines are for fully depolying puppet
apply into production?

I know crinkle had another patch up[1] that sandboxes infra-cloud modules into
a new directory, but not sure status of that. If infra-roots are fine with that,
that will at least allow infra-cloud to move forward with the newer version of
puppet-mysql. This method is the first step of the dynamic modulepath too.

[1] https://review.openstack.org/#/c/209617/
> -- 
>   Spencer Krum
>   nibz at spencerkrum.com
> 
> On Mon, Nov 2, 2015, at 10:49 AM, Paul Belanger wrote:
> > On Mon, Nov 02, 2015 at 10:27:26AM -0800, Clark Boylan wrote:
> > > On Mon, Nov 2, 2015, at 07:58 AM, Paul Belanger wrote:
> > > > Greetings,
> > > > 
> > > > I'd like to start a thread talking about the effort to upgrade our
> > > > version of
> > > > puppet-mysql to a newer / latest version. I know there has been some talk
> > > > on this already, would somebody mind adding some information?
> > > > 
> > > > I have heard 3 things:
> > > > 
> > > >   1. Remove database support from current modules, and only use
> > > >   sql_connection
> > > >      strings.
> > > >   2. Move everything to trove
> > > >   3. Setup mysql cluster
> > > I think this is mostly orthogonal to updating the puppet-mysql module
> > > because 2 + 1ish (use sql_connection) is basically what we do everywhere
> > > but Jenkins slave image builds, paste.openstack.org, and
> > > wiki.openstack.org.
> > > > 
> > > > Again, I don't know if there are true or not, just things I have seen
> > > > people
> > > > talk about.
> > > > 
> > > > The obviously part that is missing, is _how_ we are going to do the
> > > > upgrade. I
> > > > know some people (clarkb?) already have some ideas on that.
> > > > 
> > > > The reason for me asking, 2 weeks ago I offered to help the infra-cloud
> > > > move
> > > > forward and upgrading puppet-mysql was one of the items discussed. So,
> > > > here I
> > > > am, offering to do the grunt work, just need some understanding on what
> > > > people
> > > > want to do.
> > > I was hoping that there was a forward and backward compatible
> > > intermediate step we could make where old and new configs were supported
> > > but I am told that isn't possible. As a result we will likely need to
> > > update puppet-mysql, then all three of the above uses of puppet-mysql
> > > semi atomically to keep everything working.
> > > 
> > > Testing that image builds work before the update is simple as we can
> > > just run
> > > openstack-infra/project-config/nodepool/scripts/prepare_node_bare.sh to
> > > see if that works. Paste.o.o is simple and has gone from local Drizzle
> > > to Trove to local MySQL without much fuss so I doubt it will have much
> > > trouble but a test deployment is easy if we are worried.
> > > 
> > > The tricky one is going to be wiki.openstack.org and this is maybe where
> > > the above list is relevant to the discussion. We could move it to an off
> > > host database hosted by trove and not worry about updating puppet-mysql
> > > in that module at all. Or we can test a deployment of wiki using the
> > > newer mysql module before prod deployment. In either case we should
> > > probably announce a wiki downtime prior to the upgrade, stop apache/php,
> > > perform a database backup, switch to trove/run puppet with newer module,
> > > restart apache/php.
> > > 
> > So one of the big things I see, is once we bump puppet-mysql to the newer
> > /
> > latest version, all our current nodes will start consuming it (unless we
> > stop
> > puppet on each server).
> > 
> > One thought I had was some like this:
> > 
> >   1. Create /opt/puppet/modules/old, install current puppet-mysql module
> >   into
> >      it.
> >   2. Remove /etc/puppet/modules/mysql and append --modulepath to now
> >   include
> >      /opt/puppet/modules/old
> >   3. Add dynamic logic to build modulepath for puppet-mysql based on flag
> >   and
> >      force all nodes to it.
> >   4. install puppet-mysql latest into /etc/puppet/modules.
> >   5. Start migration process, paste.o.o for example.
> >   6. Work through all nodes.
> >   7. Disable dynamic modulepath logic (can be used for other module
> >   upgrades).
> >   8. Delete /opt/puppet/modules/old/mysql
> > 
> > Something like this, would allow us to some how control which nodes start
> > consuming the newer version of puppet-mysql.  Instead of a massive
> > cutover for
> > all our nodes.
> > 
> > > Hope this helps,
> > > Clark
> > > 
> > > _______________________________________________
> > > OpenStack-Infra mailing list
> > > OpenStack-Infra at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > 
> > _______________________________________________
> > OpenStack-Infra mailing list
> > OpenStack-Infra at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> 
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra