Hi Bharat, I actually just resolved this issue by adding [clients] Endpoint_type = internalURL To /etc/heat/heat.conf I now have a different issue: 2020-07-24 12:53:06.206 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key api_address 2020-07-24 12:53:06.207 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key kube_masters 2020-07-24 12:53:06.695 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key api_address 2020-07-24 12:53:06.696 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key kube_minions 2020-07-24 12:53:26.271 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.272 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_masters 2020-07-24 12:53:26.321 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.322 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_masters 2020-07-24 12:53:26.348 24280 ERROR magnum.drivers.heat.driver [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] Nodegroup error, stack status: CREATE_FAILED, stack_id: 77bed453-17f4-4 779-9883-b36a82d26578, reason: Resource CREATE failed: AuthorizationFailure: resources.kube_masters.resources[0].resources.kube-master: Authorization failed. 2020-07-24 12:53:26.670 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.671 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_minions 2020-07-24 12:53:26.727 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.728 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_minions 2020-07-24 12:53:26.755 24280 ERROR magnum.drivers.heat.driver [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] Nodegroup error, stack status: CREATE_FAILED, stack_id: 77bed453-17f4-4 779-9883-b36a82d26578, reason: Resource CREATE failed: AuthorizationFailure: resources.kube_masters.resources[0].resources.kube-master: Authorization failed. Regards Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ From: Bharat Kunwar <bharat@stackhpc.com> Sent: 24 July 2020 14:11 To: Dibbo, Alexander (STFC,RAL,SC) <alexander.dibbo@stfc.ac.uk> Cc: openstack-discuss@lists.openstack.org Subject: Re: Magnum: invalid format of client version Interesting, what version of Heat and Nova are you using? That looks like a nova client error which I guess Heat is using to make calls to Nova. On 24 Jul 2020, at 12:52, Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk<mailto:alexander.dibbo@stfc.ac.uk>> wrote: Ok, This definitely seems to be a heat issue. A heat stack created with a server group gives the same error Regards Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at:https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack:https://stfc-cloud.slack.com/ From: Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk<mailto:alexander.dibbo@stfc.ac.uk>> Sent: 24 July 2020 09:58 To: Bharat Kunwar <bharat@stackhpc.com<mailto:bharat@stackhpc.com>> Cc: openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> Subject: RE: Magnum: invalid format of client version From doing some more digging I see this in the debug logs on heat 2020-07-24 08:33:10.618 2167043 INFO heat.engine.stack [req-249dfd80-48f4-44f5-9f0e-7c9ac743f047 - admin - default default] Exception in stack validation 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack Traceback (most recent call last): 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 909, in validate 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack result = res.validate() 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server_group.py", line 69, in validate 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack MICROVERSION_SOFT_POLICIES) 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/nova.py", line 107, in is_version_supported 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack self.get_max_microversion()) 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 236, in get_api_version 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack api_version = APIVersion(version_string) 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 77, in __init__ 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack raise exceptions.UnsupportedVersion(msg) 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack UnsupportedVersion: Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of ve rsion. So it looks like heat is trying to not specify a microversion which is causing an issue with creating server groups. I have tried and I am able to create a server group and assign members with the cli. Regards Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at:https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack:https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.