On 11/01, Brandon Caulder wrote:
Hi,
The steps were... - purge - shutdown cinder-scheduler, cinder-api - upgrade software - restart cinder-volume
Hi, You should not restart cinder volume services before doing the DB sync, otherwise the Cinder service is likely to fail.
- sync (upgrade fails and stops at v114) - sync again (db upgrades to v117) - restart cinder-volume - stacktrace observed in volume.log
At this point this could be a DB issue: https://bugs.mysql.com/bug.php?id=67926 https://jira.mariadb.org/browse/MDEV-10558 Cheers, Gorka.
Thanks
On Fri, Jan 11, 2019 at 7:23 AM Gorka Eguileor <geguileo@redhat.com> wrote:
On 10/01, Brandon Caulder wrote:
Hi Iain,
There are 424 rows in volumes which drops down to 185 after running cinder-manage db purge 1. Restarting the volume service after package upgrade and running sync again does not remediate the problem, although running db sync a second time does bump the version up to 117, the following appears in the volume.log...
Hi,
If I understand correctly the steps were:
- Run DB sync --> Fail - Run DB purge - Restart volume services - See the log error - Run DB sync --> version proceeds to 117
If that is the case, could you restart the services again now that the migration has been moved to version 117?
If the cinder-volume service is able to restart please run the online data migrations with the service running.
Cheers, Gorka.
Thanks
On Thu, Jan 10, 2019 at 11:15 AM iain MacDonnell < iain.macdonnell@oracle.com> wrote:
Different issue, I believe (DB sync vs. online migrations) - it just happens that both pertain to shared targets.
Brandon, might you have a very large number of rows in your volumes table? Have you been purging soft-deleted rows?
~iain
On 1/10/19 11:01 AM, Jay Bryant wrote:
Brandon,
I am thinking you are hitting this bug:
I think you can work around it by retrying the migration with the
volume
service running. You may, however, want to check with Iain MacDonnell as he has been looking at this for a while.
Thanks! Jay
On 1/10/2019 12:34 PM, Brandon Caulder wrote:
Hi,
I am receiving the following error when performing an offline upgrade of cinder from RDO openstack-cinder-1:11.1.0-1.el7 to openstack-cinder-1:12.0.3-1.el7.
# cinder-manage db version 105
# cinder-manage --debug db sync Error during database migration: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'UPDATE volumes SET shared_targets=%(shared_targets)s'] [parameters: {'shared_targets': 1}]
# cinder-manage db version 114
The db version does not upgrade to queens version 117. Any help would be appreciated.
Thank you