Hi, I’ve just worked through this issue myself, the issue is due to `nova-mange db sync` not completing correctly. In my case the issue was caused by us having upgraded from MariaDB 10.1 to 10.3: The issue is described here for another instance: https://bugs.launchpad.net/kolla-ansible/+bug/1856296 And here is the instruction on how to fix it: https://lxadm.com/MySQL:_changing_ROW_FORMAT_to_DYNAMIC_or_COMPRESSED For me I had to alter: Nova.instances Nova.shadow_instances Nova_cell0.instances Nova_cell0.shadow_instances I hope this helps Regards Alexander Dibbo – Cloud Architect For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ From: Amjad Kotobi <kotobi@dkrz.de> Sent: 18 June 2020 10:40 To: openstack-discuss@lists.openstack.org Subject: [nova] stein to train upgrade Hi all, I tried to upgrade nova-compute from stein -> train release and “openstack-nova-compute” couldn’t able to start and from log here is coming up ################### 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service Traceback (most recent call last): 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 810, in run_service 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service service.start() 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 174, in start 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service self.manager.init_host() 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1337, in init_host 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service expected_attrs=['info_cache', 'metadata', 'numa_topology']) 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service args, kwargs) 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service args=args, kwargs=kwargs) 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 181, in call 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=self.transport_options) 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 129, in _send 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=transport_options) 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 646, in send 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=transport_options) 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service raise result 2020-06-18 11:31:08.037 303344 ERROR oslo_service.service RemoteError: Remote error: DBError (pymysql.err.InternalError) (1054, u"Unknown column 'instances.hidden' in 'field list ……. ################### I only took beginning part, the field doesn’t exist in DB, or? I also upgrade rest of the nova components[api-scheduler-conductor,…] but that didn’t go away. Any idea how to tackle this? This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.