<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi,<div><br></div><div>We are running a 5 node galera cluster with haproxy in front. Haproxy is installed on the same node as cinder and its configuration sends all reads and writes to the first node. From what I can tell the galera and mariadb rpms did not come from the RDO repository.<br><div><br></div><div><br></div><div>/etc/cinder/cinder.conf<br><div>[database]</div><div>connection = mysql+pymysql://<a href="http://cinder:xxxxxxxx@127.0.0.1/cinder">cinder:xxxxxxxx@127.0.0.1/cinder</a></div></div><div><br></div><div><br></div><div>/etc/haproxy/haproxy.conf</div><div>listen galera <a href="http://127.0.0.1:3306">127.0.0.1:3306</a></div><div> maxconn 10000</div><div> mode tcp</div><div> option tcpka</div><div> option tcplog</div><div> option mysql-check user haproxy</div><div> server db1 <a href="http://10.252.173.54:3306">10.252.173.54:3306</a> check maxconn 10000</div><div> server db2 <a href="http://10.252.173.55:3306">10.252.173.55:3306</a> check backup maxconn 10000</div><div> server db3 <a href="http://10.252.173.56:3306">10.252.173.56:3306</a> check backup maxconn 10000</div><div> server db4 <a href="http://10.252.173.57:3306">10.252.173.57:3306</a> check backup maxconn 10000</div><div> server db5 <a href="http://10.252.173.58:3306">10.252.173.58:3306</a> check backup maxconn 10000</div><div><br></div><br></div><div><div>Name : haproxy</div><div>Version : 1.5.18</div><div>Release : 7.el7</div><div>Architecture: x86_64</div><div>Install Date: Wed 09 Jan 2019 07:09:01 PM GMT</div><div>Group : System Environment/Daemons</div><div>Size : 2689838</div><div>License : GPLv2+</div><div>Signature : RSA/SHA256, Wed 25 Apr 2018 11:04:31 AM GMT, Key ID 24c6a8a7f4a80eb5</div><div>Source RPM : haproxy-1.5.18-7.el7.src.rpm</div><div>Build Date : Wed 11 Apr 2018 04:28:42 AM GMT</div><div>Build Host : <a href="http://x86-01.bsys.centos.org">x86-01.bsys.centos.org</a></div><div>Relocations : (not relocatable)</div><div>Packager : CentOS BuildSystem <<a href="http://bugs.centos.org">http://bugs.centos.org</a>></div><div>Vendor : CentOS</div><div>URL : <a href="http://www.haproxy.org/">http://www.haproxy.org/</a></div><div>Summary : TCP/HTTP proxy and load balancer for high availability environments</div></div><div><br></div><div><br></div><div><div>Name : galera</div><div>Version : 25.3.20</div><div>Release : 1.rhel7.el7.centos</div><div>Architecture: x86_64</div><div>Install Date: Wed 09 Jan 2019 07:07:52 PM GMT</div><div>Group : System Environment/Libraries</div><div>Size : 36383325</div><div>License : GPL-2.0</div><div>Signature : DSA/SHA1, Tue 02 May 2017 04:20:52 PM GMT, Key ID cbcb082a1bb943db</div><div>Source RPM : galera-25.3.20-1.rhel7.el7.centos.src.rpm</div><div>Build Date : Thu 27 Apr 2017 12:58:55 PM GMT</div><div>Build Host : centos70-x86-64</div><div>Relocations : (not relocatable)</div><div>Packager : Codership Oy</div><div>Vendor : Codership Oy</div><div>URL : <a href="http://www.codership.com/">http://www.codership.com/</a></div><div>Summary : Galera: a synchronous multi-master wsrep provider (replication engine)</div></div><div><br></div><div><br></div><div><div>Name : MariaDB-server</div><div>Version : 10.3.2</div><div>Release : 1.el7.centos</div><div>Architecture: x86_64</div><div>Install Date: Wed 09 Jan 2019 07:08:11 PM GMT</div><div>Group : Applications/Databases</div><div>Size : 511538370</div><div>License : GPLv2</div><div>Signature : DSA/SHA1, Sat 07 Oct 2017 05:51:08 PM GMT, Key ID cbcb082a1bb943db</div><div>Source RPM : MariaDB-server-10.3.2-1.el7.centos.src.rpm</div><div>Build Date : Fri 06 Oct 2017 01:51:16 PM GMT</div><div>Build Host : centos70-x86-64</div><div>Relocations : (not relocatable)</div><div>Vendor : MariaDB Foundation</div><div>URL : <a href="http://mariadb.org">http://mariadb.org</a></div><div>Summary : MariaDB: a very fast and robust SQL database server</div></div><div><br></div><div>Thanks</div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jan 14, 2019 at 8:23 AM Mike Bayer <<a href="mailto:mike_mp@zzzcomputing.com">mike_mp@zzzcomputing.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Jan 14, 2019 at 7:38 AM Gorka Eguileor <<a href="mailto:geguileo@redhat.com" target="_blank">geguileo@redhat.com</a>> wrote:<br>
><br>
> On 11/01, Brandon Caulder wrote:<br>
> > Hi,<br>
> ><br>
> > The steps were...<br>
> > - purge<br>
> > - shutdown cinder-scheduler, cinder-api<br>
> > - upgrade software<br>
> > - restart cinder-volume<br>
><br>
> Hi,<br>
><br>
> You should not restart cinder volume services before doing the DB sync,<br>
> otherwise the Cinder service is likely to fail.<br>
><br>
> > - sync (upgrade fails and stops at v114)<br>
> > - sync again (db upgrades to v117)<br>
> > - restart cinder-volume<br>
> > - stacktrace observed in volume.log<br>
> ><br>
><br>
> At this point this could be a DB issue:<br>
><br>
> <a href="https://bugs.mysql.com/bug.php?id=67926" rel="noreferrer" target="_blank">https://bugs.mysql.com/bug.php?id=67926</a><br>
> <a href="https://jira.mariadb.org/browse/MDEV-10558" rel="noreferrer" target="_blank">https://jira.mariadb.org/browse/MDEV-10558</a><br>
<br>
that's a scary issue, can the reporter please list what MySQL /<br>
MariaDB version is running and if this is Galera/HA or single node?<br>
<br>
<br>
><br>
> Cheers,<br>
> Gorka.<br>
><br>
> > Thanks<br>
> ><br>
> > On Fri, Jan 11, 2019 at 7:23 AM Gorka Eguileor <<a href="mailto:geguileo@redhat.com" target="_blank">geguileo@redhat.com</a>> wrote:<br>
> ><br>
> > > On 10/01, Brandon Caulder wrote:<br>
> > > > Hi Iain,<br>
> > > ><br>
> > > > There are 424 rows in volumes which drops down to 185 after running<br>
> > > > cinder-manage db purge 1. Restarting the volume service after package<br>
> > > > upgrade and running sync again does not remediate the problem, although<br>
> > > > running db sync a second time does bump the version up to 117, the<br>
> > > > following appears in the volume.log...<br>
> > > ><br>
> > > > <a href="http://paste.openstack.org/show/Gfbe94mSAqAzAp4Ycwlz/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/Gfbe94mSAqAzAp4Ycwlz/</a><br>
> > > ><br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > If I understand correctly the steps were:<br>
> > ><br>
> > > - Run DB sync --> Fail<br>
> > > - Run DB purge<br>
> > > - Restart volume services<br>
> > > - See the log error<br>
> > > - Run DB sync --> version proceeds to 117<br>
> > ><br>
> > > If that is the case, could you restart the services again now that the<br>
> > > migration has been moved to version 117?<br>
> > ><br>
> > > If the cinder-volume service is able to restart please run the online<br>
> > > data migrations with the service running.<br>
> > ><br>
> > > Cheers,<br>
> > > Gorka.<br>
> > ><br>
> > ><br>
> > > > Thanks<br>
> > > ><br>
> > > > On Thu, Jan 10, 2019 at 11:15 AM iain MacDonnell <<br>
> > > <a href="mailto:iain.macdonnell@oracle.com" target="_blank">iain.macdonnell@oracle.com</a>><br>
> > > > wrote:<br>
> > > ><br>
> > > > ><br>
> > > > > Different issue, I believe (DB sync vs. online migrations) - it just<br>
> > > > > happens that both pertain to shared targets.<br>
> > > > ><br>
> > > > > Brandon, might you have a very large number of rows in your volumes<br>
> > > > > table? Have you been purging soft-deleted rows?<br>
> > > > ><br>
> > > > > ~iain<br>
> > > > ><br>
> > > > ><br>
> > > > > On 1/10/19 11:01 AM, Jay Bryant wrote:<br>
> > > > > > Brandon,<br>
> > > > > ><br>
> > > > > > I am thinking you are hitting this bug:<br>
> > > > > ><br>
> > > > ><br>
> > > <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.launchpad.net_cinder_-2Bbug_1806156&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=RxYkIjeLZPK2frXV_wEUCq8d3wvUIvDPimUcunMwbMs&m=FHjmiBaQPWLNzGreplNmZfCZ0MkpV5GLaqD2hcs5hwg&s=AvAoszuVyGkd2_1hyCnQjwGEw9dUNfEoqsUcxdHYZqU&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.launchpad.net_cinder_-2Bbug_1806156&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=RxYkIjeLZPK2frXV_wEUCq8d3wvUIvDPimUcunMwbMs&m=FHjmiBaQPWLNzGreplNmZfCZ0MkpV5GLaqD2hcs5hwg&s=AvAoszuVyGkd2_1hyCnQjwGEw9dUNfEoqsUcxdHYZqU&e=</a><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > I think you can work around it by retrying the migration with the<br>
> > > volume<br>
> > > > > > service running. You may, however, want to check with Iain<br>
> > > MacDonnell<br>
> > > > > > as he has been looking at this for a while.<br>
> > > > > ><br>
> > > > > > Thanks!<br>
> > > > > > Jay<br>
> > > > > ><br>
> > > > > ><br>
> > > > > > On 1/10/2019 12:34 PM, Brandon Caulder wrote:<br>
> > > > > >> Hi,<br>
> > > > > >><br>
> > > > > >> I am receiving the following error when performing an offline<br>
> > > upgrade<br>
> > > > > >> of cinder from RDO openstack-cinder-1:11.1.0-1.el7 to<br>
> > > > > >> openstack-cinder-1:12.0.3-1.el7.<br>
> > > > > >><br>
> > > > > >> # cinder-manage db version<br>
> > > > > >> 105<br>
> > > > > >><br>
> > > > > >> # cinder-manage --debug db sync<br>
> > > > > >> Error during database migration: (pymysql.err.OperationalError)<br>
> > > (2013,<br>
> > > > > >> 'Lost connection to MySQL server during query') [SQL: u'UPDATE<br>
> > > volumes<br>
> > > > > >> SET shared_targets=%(shared_targets)s'] [parameters:<br>
> > > > > >> {'shared_targets': 1}]<br>
> > > > > >><br>
> > > > > >> # cinder-manage db version<br>
> > > > > >> 114<br>
> > > > > >><br>
> > > > > >> The db version does not upgrade to queens version 117. Any help<br>
> > > would<br>
> > > > > >> be appreciated.<br>
> > > > > >><br>
> > > > > >> Thank you<br>
> > > > > ><br>
> > > > ><br>
> > > > ><br>
> > ><br>
><br>
</blockquote></div>