On 18. Jun 2020, at 12:50, Sean Mooney <smooney@redhat.com> wrote:On Thu, 2020-06-18 at 10:11 +0000, Alexander Dibbo - UKRI STFC wrote:Hi,also just a note you should always update the controller first and computes second.
I’ve just worked through this issue myself, the issue is due to `nova-mange db sync` not completing correctly.
In my case the issue was caused by us having upgraded from MariaDB 10.1 to 10.3:
The issue is described here for another instance:
https://bugs.launchpad.net/kolla-ansible/+bug/1856296
And here is the instruction on how to fix it:
https://lxadm.com/MySQL:_changing_ROW_FORMAT_to_DYNAMIC_or_COMPRESSED
For me I had to alter:
Nova.instances
Nova.shadow_instances
Nova_cell0.instances
Nova_cell0.shadow_instances
I hope this helps
Regards
Alexander Dibbo – Cloud Architect
For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/>;
To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:
cloud-support@gridpp.rl.ac.uk>
To receive notifications about the service please subscribe to our mailing list at:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD
To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/
From: Amjad Kotobi <kotobi@dkrz.de>
Sent: 18 June 2020 10:40
To: openstack-discuss@lists.openstack.org
Subject: [nova] stein to train upgrade
Hi all,
I tried to upgrade nova-compute from stein -> train release and “openstack-nova-compute” couldn’t able to start and
from log here is coming up
###################
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service Traceback (most recent call last):
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-
packages/oslo_service/service.py", line 810, in run_service
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service service.start()
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py",
line 174, in start
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service self.manager.init_host()
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-
packages/nova/compute/manager.py", line 1337, in init_host
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service expected_attrs=['info_cache', 'metadata',
'numa_topology'])
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-
packages/oslo_versionedobjects/base.py", line 177, in wrapper
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service args, kwargs)
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-
packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service args=args, kwargs=kwargs)
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-
packages/oslo_messaging/rpc/client.py", line 181, in call
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=self.transport_options)
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-
packages/oslo_messaging/transport.py", line 129, in _send
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=transport_options)
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-
packages/oslo_messaging/_drivers/amqpdriver.py", line 646, in send
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=transport_options)
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-
packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service raise result
2020-06-18 11:31:08.037 303344 ERROR oslo_service.service RemoteError: Remote error: DBError
(pymysql.err.InternalError) (1054, u"Unknown column 'instances.hidden' in 'field list
…….
###################
I only took beginning part, the field doesn’t exist in DB, or?
I also upgrade rest of the nova components[api-scheduler-conductor,…] but that didn’t go away.
if you do otherwise it can case issues with rpc.
but yes this is caused by a failure of the db migration to run which is done
via "nova-manage db sync" on a contoler node. the compute nodes should not have
db access or creds in there conf files.
Any idea how to tackle this?
This email and any attachments are intended solely for the use of the named recipients. If you are not the intended
recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the
sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every
reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the
recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any
liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions,
conclusions or other information in this message and attachments that are not related directly to UKRI business are
solely those of the author and do not represent the views of UKRI.