<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Cascadia Mono";
panose-1:2 11 6 9 2 0 0 2 0 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi Bharat,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I actually just resolved this issue by adding
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">[clients]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Endpoint_type = internalURL<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">To /etc/heat/heat.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I now have a different issue:<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#0C0C0C"><span style="font-family:"Cascadia Mono";color:#CCCCCC;background:#0C0C0C">2020-07-24 12:53:06.206 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does
not have output_key api_address<br>
2020-07-24 12:53:06.207 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key kube_masters<br>
2020-07-24 12:53:06.695 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key api_address<br>
2020-07-24 12:53:06.696 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key kube_minions<br>
2020-07-24 12:53:26.271 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address<br>
2020-07-24 12:53:26.272 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_masters<br>
2020-07-24 12:53:26.321 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address<br>
2020-07-24 12:53:26.322 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_masters<br>
2020-07-24 12:53:26.348 24280 ERROR magnum.drivers.heat.driver [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] Nodegroup error, stack status: CREATE_FAILED, stack_id: 77bed453-17f4-4<br>
779-9883-b36a82d26578, reason: Resource CREATE failed: AuthorizationFailure: resources.kube_masters.resources[0].resources.kube-master: Authorization failed.<br>
2020-07-24 12:53:26.670 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address<br>
2020-07-24 12:53:26.671 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_minions<br>
2020-07-24 12:53:26.727 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address<br>
2020-07-24 12:53:26.728 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_minions<br>
2020-07-24 12:53:26.755 24280 ERROR magnum.drivers.heat.driver [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] Nodegroup error, stack status: CREATE_FAILED, stack_id: 77bed453-17f4-4<br>
779-9883-b36a82d26578, reason: Resource CREATE failed: AuthorizationFailure: resources.kube_masters.resources[0].resources.kube-master: Authorization failed.</span><span style="font-family:"Cascadia Mono""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Regards<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">For STFC Cloud Documentation visit
<a href="https://stfc-cloud-docs.readthedocs.io/"><span style="color:#0563C1">https://stfc-cloud-docs.readthedocs.io</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To raise a support ticket with the cloud team please email
<a href="mailto:cloud-support@gridpp.rl.ac.uk"><span style="color:#0563C1">cloud-support@gridpp.rl.ac.uk</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To receive notifications about the service please subscribe to our mailing list at:
<a href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD"><span style="color:#0563C1">https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To receive fast notifications or to discuss usage of the cloud please join our Slack:
<a href="https://stfc-cloud.slack.com/"><span style="color:#0563C1">https://stfc-cloud.slack.com/</span></a>
<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Bharat Kunwar <bharat@stackhpc.com>
<br>
<b>Sent:</b> 24 July 2020 14:11<br>
<b>To:</b> Dibbo, Alexander (STFC,RAL,SC) <alexander.dibbo@stfc.ac.uk><br>
<b>Cc:</b> openstack-discuss@lists.openstack.org<br>
<b>Subject:</b> Re: Magnum: invalid format of client version<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Interesting, what version of Heat and Nova are you using? That looks like a nova client error which I guess Heat is using to make calls to Nova.<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On 24 Jul 2020, at 12:52, Alexander Dibbo - UKRI STFC <<a href="mailto:alexander.dibbo@stfc.ac.uk">alexander.dibbo@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Ok, This definitely seems to be a heat issue. A heat stack created with a server group gives the same error</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Regards</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">For STFC Cloud Documentation visit<span class="apple-converted-space"> </span><a href="https://stfc-cloud-docs.readthedocs.io/"><span style="color:#0563C1">https://stfc-cloud-docs.readthedocs.io</span></a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To raise a support ticket with the cloud team please email<span class="apple-converted-space"> </span><a href="mailto:cloud-support@gridpp.rl.ac.uk"><span style="color:#0563C1">cloud-support@gridpp.rl.ac.uk</span></a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To receive notifications about the service please subscribe to our mailing list at:<a href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD"><span style="color:#0563C1">https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD</span></a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To receive fast notifications or to discuss usage of the cloud please join our Slack:<a href="https://stfc-cloud.slack.com/"><span style="color:#0563C1">https://stfc-cloud.slack.com/</span></a></span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Alexander
Dibbo - UKRI STFC <<a href="mailto:alexander.dibbo@stfc.ac.uk">alexander.dibbo@stfc.ac.uk</a>><span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>24 July 2020 09:58<br>
<b>To:</b><span class="apple-converted-space"> </span>Bharat Kunwar <<a href="mailto:bharat@stackhpc.com">bharat@stackhpc.com</a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span><a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>RE: Magnum: invalid format of client version</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">From doing some more digging I see this in the debug logs on heat</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:#0C0C0C"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">2020-07-24 08:33:10.618 2167043 INFO heat.engine.stack [req-249dfd80-48f4-44f5-9f0e-7c9ac743f047 - admin - default default] Exception
in stack validation<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack Traceback (most recent call last):<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 909, in validate<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack result = res.validate()<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server_group.py", line 69, in validate<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack MICROVERSION_SOFT_POLICIES)<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/nova.py", line 107, in is_version_supported<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack self.get_max_microversion())<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 236, in get_api_version<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack api_version = APIVersion(version_string)<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 77, in __init__<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack raise exceptions.UnsupportedVersion(msg)<br>
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack UnsupportedVersion: Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of ve<br>
rsion.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">So it looks like heat is trying to not specify a microversion which is causing an issue with creating server groups. I have tried and I am able to create a server
group and assign members with the cli.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Regards</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">For STFC Cloud Documentation visit<span class="apple-converted-space"> </span></span><a href="https://stfc-cloud-docs.readthedocs.io/"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#0563C1">https://stfc-cloud-docs.readthedocs.io</span></a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To raise a support ticket with the cloud team please email<span class="apple-converted-space"> </span></span><a href="mailto:cloud-support@gridpp.rl.ac.uk"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#0563C1">cloud-support@gridpp.rl.ac.uk</span></a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To receive notifications about the service please subscribe to our mailing list at:</span><a href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#0563C1">https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD</span></a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To receive fast notifications or to discuss usage of the cloud please join our Slack:</span><a href="https://stfc-cloud.slack.com/"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#0563C1">https://stfc-cloud.slack.com/</span></a><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:6.0pt">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose,
copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments
containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions,
conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:6.0pt">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify
the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own
virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments
that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.</span><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p><span style="font-size: 6pt;">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.</span></p></body>
</html>