<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Thank you all, It did help me out and solved the issue.<div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On 18. Jun 2020, at 12:50, Sean Mooney <<a href="mailto:smooney@redhat.com" class="">smooney@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">On Thu, 2020-06-18 at 10:11 +0000, Alexander Dibbo - UKRI STFC wrote:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Hi,<br class=""><br class="">I’ve just worked through this issue myself, the issue is due to `nova-mange db sync` not completing correctly.<br class=""><br class="">In my case the issue was caused by us having upgraded from MariaDB 10.1 to 10.3:<br class=""><br class="">The issue is described here for another instance:<br class=""><a href="https://bugs.launchpad.net/kolla-ansible/+bug/1856296" class="">https://bugs.launchpad.net/kolla-ansible/+bug/1856296</a><br class=""><br class="">And here is the instruction on how to fix it:<br class="">https://lxadm.com/MySQL:_changing_ROW_FORMAT_to_DYNAMIC_or_COMPRESSED<br class=""><br class="">For me I had to alter:<br class="">Nova.instances<br class="">Nova.shadow_instances<br class="">Nova_cell0.instances<br class="">Nova_cell0.shadow_instances<br class=""><br class="">I hope this helps<br class=""><br class="">Regards<br class=""><br class="">Alexander Dibbo – Cloud Architect<br class="">For STFC Cloud Documentation visit<span class="Apple-converted-space"> </span><a href="https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/>" class="">https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/></a>;<br class="">To raise a support ticket with the cloud team please email<span class="Apple-converted-space"> </span><a href="mailto:cloud-support@gridpp.rl.ac.uk" class="">cloud-support@gridpp.rl.ac.uk</a><mailto:<br class="">cloud-support@gridpp.rl.ac.uk><br class="">To receive notifications about the service please subscribe to our mailing list at:<span class="Apple-converted-space"> </span><br class="">https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD<br class="">To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/<br class=""><br class="">From: Amjad Kotobi <kotobi@dkrz.de><br class="">Sent: 18 June 2020 10:40<br class="">To: openstack-discuss@lists.openstack.org<br class="">Subject: [nova] stein to train upgrade<br class=""><br class="">Hi all,<br class=""><br class="">I tried to upgrade nova-compute from stein -> train release and “openstack-nova-compute” couldn’t able to start and<br class="">from log here is coming up<br class=""><br class="">###################<br class=""><br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service Traceback (most recent call last):<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-<br class="">packages/oslo_service/service.py", line 810, in run_service<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service service.start()<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py",<br class="">line 174, in start<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service self.manager.init_host()<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-<br class="">packages/nova/compute/manager.py", line 1337, in init_host<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service expected_attrs=['info_cache', 'metadata',<br class="">'numa_topology'])<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-<br class="">packages/oslo_versionedobjects/base.py", line 177, in wrapper<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service args, kwargs)<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-<br class="">packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service args=args, kwargs=kwargs)<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-<br class="">packages/oslo_messaging/rpc/client.py", line 181, in call<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=self.transport_options)<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-<br class="">packages/oslo_messaging/transport.py", line 129, in _send<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=transport_options)<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-<br class="">packages/oslo_messaging/_drivers/amqpdriver.py", line 646, in send<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=transport_options)<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-<br class="">packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service raise result<br class="">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service RemoteError: Remote error: DBError<br class="">(pymysql.err.InternalError) (1054, u"Unknown column 'instances.hidden' in 'field list<br class="">…….<br class="">###################<br class=""><br class="">I only took beginning part, the field doesn’t exist in DB, or?<br class="">I also upgrade rest of the nova components[api-scheduler-conductor,…] but that didn’t go away.<br class=""></blockquote><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">also just a note you should always update the controller first and computes second.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">if you do otherwise it can case issues with rpc.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">but yes this is caused by a failure of the db migration to run which is done</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">via "nova-manage db sync" on a contoler node. the compute nodes should not have</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">db access or creds in there conf files.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class="">Any idea how to tackle this?<br class=""><br class=""><br class=""><br class="">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended<br class="">recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the<br class="">sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every<br class="">reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the<br class="">recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any<br class="">liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions,<br class="">conclusions or other information in this message and attachments that are not related directly to UKRI business are<br class="">solely those of the author and do not represent the views of UKRI.</blockquote></div></blockquote></div><br class=""></div></body></html>