<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Helvetica Neue";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I’ve just worked through this issue myself, the issue is due to `nova-mange db sync` not completing correctly.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">In my case the issue was caused by us having upgraded from MariaDB 10.1 to 10.3:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">The issue is described here for another instance:<o:p></o:p></span></p>
<p class="MsoNormal"><a href="https://bugs.launchpad.net/kolla-ansible/+bug/1856296">https://bugs.launchpad.net/kolla-ansible/+bug/1856296</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And here is the instruction on how to fix it:<o:p></o:p></p>
<p class="MsoNormal"><a href="https://lxadm.com/MySQL:_changing_ROW_FORMAT_to_DYNAMIC_or_COMPRESSED">https://lxadm.com/MySQL:_changing_ROW_FORMAT_to_DYNAMIC_or_COMPRESSED</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">For me I had to alter:<o:p></o:p></p>
<p class="MsoNormal">Nova.instances<o:p></o:p></p>
<p class="MsoNormal">Nova.shadow_instances<o:p></o:p></p>
<p class="MsoNormal">Nova_cell0.instances<o:p></o:p></p>
<p class="MsoNormal">Nova_cell0.shadow_instances<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I hope this helps<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Regards<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Alexander Dibbo – Cloud Architect<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">For STFC Cloud Documentation visit
<a href="https://stfc-cloud-docs.readthedocs.io/"><span style="color:#0563C1">https://stfc-cloud-docs.readthedocs.io</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To raise a support ticket with the cloud team please email
<a href="mailto:cloud-support@gridpp.rl.ac.uk"><span style="color:#0563C1">cloud-support@gridpp.rl.ac.uk</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To receive notifications about the service please subscribe to our mailing list at:
<a href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD"><span style="color:#0563C1">https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To receive fast notifications or to discuss usage of the cloud please join our Slack:
<a href="https://stfc-cloud.slack.com/"><span style="color:#0563C1">https://stfc-cloud.slack.com/</span></a>
<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Amjad Kotobi <kotobi@dkrz.de>
<br>
<b>Sent:</b> 18 June 2020 10:40<br>
<b>To:</b> openstack-discuss@lists.openstack.org<br>
<b>Subject:</b> [nova] stein to train upgrade<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I tried to upgrade nova-compute from stein -> train release and “openstack-nova-compute” couldn’t able to start and from log here is coming up<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">###################<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service Traceback (most recent call last):<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 810, in run_service<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service service.start()<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 174, in start<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service self.manager.init_host()<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1337, in init_host<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service expected_attrs=['info_cache', 'metadata', 'numa_topology'])<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service args, kwargs)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service args=args, kwargs=kwargs)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 181, in call<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=self.transport_options)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 129, in _send<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=transport_options)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 646, in send<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service transport_options=transport_options)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service raise result<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">2020-06-18 11:31:08.037 303344 ERROR oslo_service.service RemoteError: Remote error: DBError (pymysql.err.InternalError) (1054, u"Unknown column 'instances.hidden' in 'field list<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">…….<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">###################<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">I only took beginning part, the field doesn’t exist in DB, or?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">I also upgrade rest of the nova components[api-scheduler-conductor,…] but that didn’t go away.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue"">Any idea how to tackle this?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica Neue""><o:p> </o:p></span></p>
</div>
</div>
<p><span style="font-size: 6pt;">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.</span></p></body>
</html>