Hi Bharat,

 

I actually just resolved this issue by adding

[clients]

Endpoint_type = internalURL

 

To /etc/heat/heat.conf

 

I now have a different issue:

2020-07-24 12:53:06.206 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key api_address
2020-07-24 12:53:06.207 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key kube_masters
2020-07-24 12:53:06.695 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key api_address
2020-07-24 12:53:06.696 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key kube_minions
2020-07-24 12:53:26.271 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address
2020-07-24 12:53:26.272 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_masters
2020-07-24 12:53:26.321 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address
2020-07-24 12:53:26.322 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_masters
2020-07-24 12:53:26.348 24280 ERROR magnum.drivers.heat.driver [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] Nodegroup error, stack status: CREATE_FAILED, stack_id: 77bed453-17f4-4
779-9883-b36a82d26578, reason: Resource CREATE failed: AuthorizationFailure: resources.kube_masters.resources[0].resources.kube-master: Authorization failed.
2020-07-24 12:53:26.670 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address
2020-07-24 12:53:26.671 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_minions
2020-07-24 12:53:26.727 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address
2020-07-24 12:53:26.728 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_minions
2020-07-24 12:53:26.755 24280 ERROR magnum.drivers.heat.driver [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] Nodegroup error, stack status: CREATE_FAILED, stack_id: 77bed453-17f4-4
779-9883-b36a82d26578, reason: Resource CREATE failed: AuthorizationFailure: resources.kube_masters.resources[0].resources.kube-master: Authorization failed.

 

 

 

Regards

 

Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader

For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io

To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk

To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD

To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/

 

From: Bharat Kunwar <bharat@stackhpc.com>
Sent: 24 July 2020 14:11
To: Dibbo, Alexander (STFC,RAL,SC) <alexander.dibbo@stfc.ac.uk>
Cc: openstack-discuss@lists.openstack.org
Subject: Re: Magnum: invalid format of client version

 

Interesting, what version of Heat and Nova are you using? That looks like a nova client error which I guess Heat is using to make calls to Nova.



On 24 Jul 2020, at 12:52, Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk> wrote:

 

Ok, This definitely seems to be a heat issue. A heat stack created with a server group gives the same error

 

Regards

 

Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader

For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io

To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk

To receive notifications about the service please subscribe to our mailing list at:https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD

To receive fast notifications or to discuss usage of the cloud please join our Slack:https://stfc-cloud.slack.com/

 

From: Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk> 
Sent: 24 July 2020 09:58
To: Bharat Kunwar <bharat@stackhpc.com>
Cc: openstack-discuss@lists.openstack.org
Subject: RE: Magnum: invalid format of client version

 

From doing some more digging I see this in the debug logs on heat

2020-07-24 08:33:10.618 2167043 INFO heat.engine.stack [req-249dfd80-48f4-44f5-9f0e-7c9ac743f047 - admin - default default] Exception in stack validation
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack Traceback (most recent call last):
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 909, in validate
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack     result = res.validate()
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server_group.py", line 69, in validate
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack     MICROVERSION_SOFT_POLICIES)
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/nova.py", line 107, in is_version_supported
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack     self.get_max_microversion())
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 236, in get_api_version
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack     api_version = APIVersion(version_string)
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 77, in __init__
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack     raise exceptions.UnsupportedVersion(msg)
2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack UnsupportedVersion: Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of ve
rsion.

 

So it looks like heat is trying to not specify a microversion which is causing an issue with creating server groups. I have tried and I am able to create a server group and assign members with the cli.

 

Regards

 

Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader

For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io

To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk

To receive notifications about the service please subscribe to our mailing list at:https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD

To receive fast notifications or to discuss usage of the cloud please join our Slack:https://stfc-cloud.slack.com/

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

 

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.