Open Stack

On Thu, 2020-06-18 at 10:11 +0000, Alexander Dibbo - UKRI STFC wrote:
> Hi,
> 
> I’ve just worked through this issue myself, the issue is due to `nova-mange db sync` not completing correctly.
> 
> In my case the issue was caused by us having upgraded from MariaDB 10.1 to 10.3:
> 
> The issue is described here for another instance:
> https://bugs.launchpad.net/kolla-ansible/+bug/1856296
> 
> And here is the instruction on how to fix it:
> https://lxadm.com/MySQL:_changing_ROW_FORMAT_to_DYNAMIC_or_COMPRESSED
> 
> For me I had to alter:
> Nova.instances
> Nova.shadow_instances
> Nova_cell0.instances
> Nova_cell0.shadow_instances
> 
> I hope this helps
> 
> Regards
> 
> Alexander Dibbo – Cloud Architect
> For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/>;
> To raise a support ticket with the cloud team please email cloud-support at gridpp.rl.ac.uk<mailto:
> cloud-support at gridpp.rl.ac.uk>
> To receive notifications about the service please subscribe to our mailing list at: 
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD
> To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/
> 
> From: Amjad Kotobi <kotobi at dkrz.de>
> Sent: 18 June 2020 10:40
> To: openstack-discuss at lists.openstack.org
> Subject: [nova] stein to train upgrade
> 
> Hi all,
> 
> I tried to upgrade nova-compute from stein -> train release and “openstack-nova-compute” couldn’t able to start and
> from log here is coming up
> 
> ###################
> 
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service Traceback (most recent call last):
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service   File "/usr/lib/python2.7/site-
> packages/oslo_service/service.py", line 810, in run_service
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service     service.start()
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/service.py",
> line 174, in start
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service     self.manager.init_host()
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service   File "/usr/lib/python2.7/site-
> packages/nova/compute/manager.py", line 1337, in init_host
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service     expected_attrs=['info_cache', 'metadata',
> 'numa_topology'])
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service   File "/usr/lib/python2.7/site-
> packages/oslo_versionedobjects/base.py", line 177, in wrapper
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service     args, kwargs)
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service   File "/usr/lib/python2.7/site-
> packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service     args=args, kwargs=kwargs)
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service   File "/usr/lib/python2.7/site-
> packages/oslo_messaging/rpc/client.py", line 181, in call
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service     transport_options=self.transport_options)
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service   File "/usr/lib/python2.7/site-
> packages/oslo_messaging/transport.py", line 129, in _send
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service     transport_options=transport_options)
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service   File "/usr/lib/python2.7/site-
> packages/oslo_messaging/_drivers/amqpdriver.py", line 646, in send
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service     transport_options=transport_options)
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service   File "/usr/lib/python2.7/site-
> packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service     raise result
> 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service RemoteError: Remote error: DBError
> (pymysql.err.InternalError) (1054, u"Unknown column 'instances.hidden' in 'field list
> …….
> ###################
> 
> I only took beginning part, the field doesn’t exist in DB, or?
> I also upgrade rest of the nova components[api-scheduler-conductor,…] but that didn’t go away.
also just a note you should always update the controller first and computes second.
if you do otherwise it can case issues with rpc.

but yes this is caused by a failure of the db migration to run which is done
via "nova-manage db sync" on a contoler node. the compute nodes should not have
db access or creds in there conf files.
> 
> Any idea how to tackle this?
> 
> 
> 
> This email and any attachments are intended solely for the use of the named recipients. If you are not the intended
> recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the
> sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every
> reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the
> recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any
> liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions,
> conclusions or other information in this message and attachments that are not related directly to UKRI business are
> solely those of the author and do not represent the views of UKRI.