Open Stack

On 11/01, Brandon Caulder wrote:
> Hi,
>
> The steps were...
> - purge
> - shutdown cinder-scheduler, cinder-api
> - upgrade software
> - restart cinder-volume

Hi,

You should not restart cinder volume services before doing the DB sync,
otherwise the Cinder service is likely to fail.

> - sync (upgrade fails and stops at v114)
> - sync again (db upgrades to v117)
> - restart cinder-volume
> - stacktrace observed in volume.log
>

At this point this could be a DB issue:

https://bugs.mysql.com/bug.php?id=67926
https://jira.mariadb.org/browse/MDEV-10558

Cheers,
Gorka.

> Thanks
>
> On Fri, Jan 11, 2019 at 7:23 AM Gorka Eguileor <geguileo at redhat.com> wrote:
>
> > On 10/01, Brandon Caulder wrote:
> > > Hi Iain,
> > >
> > > There are 424 rows in volumes which drops down to 185 after running
> > > cinder-manage db purge 1.  Restarting the volume service after package
> > > upgrade and running sync again does not remediate the problem, although
> > > running db sync a second time does bump the version up to 117, the
> > > following appears in the volume.log...
> > >
> > > http://paste.openstack.org/show/Gfbe94mSAqAzAp4Ycwlz/
> > >
> >
> > Hi,
> >
> > If I understand correctly the steps were:
> >
> > - Run DB sync --> Fail
> > - Run DB purge
> > - Restart volume services
> > - See the log error
> > - Run DB sync --> version proceeds to 117
> >
> > If that is the case, could you restart the services again now that the
> > migration has been moved to version 117?
> >
> > If the cinder-volume service is able to restart please run the online
> > data migrations with the service running.
> >
> > Cheers,
> > Gorka.
> >
> >
> > > Thanks
> > >
> > > On Thu, Jan 10, 2019 at 11:15 AM iain MacDonnell <
> > iain.macdonnell at oracle.com>
> > > wrote:
> > >
> > > >
> > > > Different issue, I believe (DB sync vs. online migrations) - it just
> > > > happens that both pertain to shared targets.
> > > >
> > > > Brandon, might you have a very large number of rows in your volumes
> > > > table? Have you been purging soft-deleted rows?
> > > >
> > > >      ~iain
> > > >
> > > >
> > > > On 1/10/19 11:01 AM, Jay Bryant wrote:
> > > > > Brandon,
> > > > >
> > > > > I am thinking you are hitting this bug:
> > > > >
> > > >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.launchpad.net_cinder_-2Bbug_1806156&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=RxYkIjeLZPK2frXV_wEUCq8d3wvUIvDPimUcunMwbMs&m=FHjmiBaQPWLNzGreplNmZfCZ0MkpV5GLaqD2hcs5hwg&s=AvAoszuVyGkd2_1hyCnQjwGEw9dUNfEoqsUcxdHYZqU&e=
> > > > >
> > > > >
> > > > > I think you can work around it by retrying the migration with the
> > volume
> > > > > service running.  You may, however, want to check with Iain
> > MacDonnell
> > > > > as he has been looking at this for a while.
> > > > >
> > > > > Thanks!
> > > > > Jay
> > > > >
> > > > >
> > > > > On 1/10/2019 12:34 PM, Brandon Caulder wrote:
> > > > >> Hi,
> > > > >>
> > > > >> I am receiving the following error when performing an offline
> > upgrade
> > > > >> of cinder from RDO openstack-cinder-1:11.1.0-1.el7 to
> > > > >> openstack-cinder-1:12.0.3-1.el7.
> > > > >>
> > > > >> # cinder-manage db version
> > > > >> 105
> > > > >>
> > > > >> # cinder-manage --debug db sync
> > > > >> Error during database migration: (pymysql.err.OperationalError)
> > (2013,
> > > > >> 'Lost connection to MySQL server during query') [SQL: u'UPDATE
> > volumes
> > > > >> SET shared_targets=%(shared_targets)s'] [parameters:
> > > > >> {'shared_targets': 1}]
> > > > >>
> > > > >> # cinder-manage db version
> > > > >> 114
> > > > >>
> > > > >> The db version does not upgrade to queens version 117.  Any help
> > would
> > > > >> be appreciated.
> > > > >>
> > > > >> Thank you
> > > > >
> > > >
> > > >
> >