Magnum: invalid format of client version
Hi, I have just deployed magnum into my train enviroment and am seeing the following error when creating any kind of cluster: This is a Train environment deployed from RDO packages (9.4.0-1). Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server [req-aa9ce18b-64eb-40ad-b1c0-b7c312402780 - - - - -] Exception during message handling: InvalidParameterValue: ERROR: UnsupportedVersion: : resources.worker_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version. Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server Traceback (most recent call last): Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/magnum/conductor/handlers/cluster_conductor.py", line 95, in cluster_create Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server raise e Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server InvalidParameterValue: ERROR: UnsupportedVersion: : resources.worker_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version. Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server #033[00m When logging in debug, I see a dump of a huge heat template and a 400 bad request from heatclient (as below) immediately before the above log excerpt: {"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": {"message": "UnsupportedVersion: : resources.master_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version.", "traceback": null, "type": "StackValidationFailed"}, "title": "Bad Request"} log_http_response /usr/lib/python2.7/site-packages/heatclient/common/http.py:157 More details are available in my question here: https://ask.openstack.org/en/question/128520/magnum-invalid-format-of-client... Any suggestions on where to look to set the client version it is complaining about would be much appreciated? Thanks Alex Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.
Hi Alex, Could you please let me know what's the cluster image you're using Fedora Atomic or Fedora CoreOS? And can you pls show me your cluster template? On 22/07/20 12:53 am, Alexander Dibbo - UKRI STFC wrote:
-- Cheers & Best regards, Feilong Wang (王飞龙) Head of R&D Catalyst Cloud - Cloud Native New Zealand -------------------------------------------------------------------------- Tel: +64-48032246 Email: flwang@catalyst.net.nz Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
Hi, We are using fedora atomic latest from Fedora-Atomic-27-20180419.0.x86_64.qcow2 +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | checksum | a8c45037711181872809eb13431402e6 | | container_format | bare | | created_at | 2019-09-09T12:37:41Z | | disk_format | qcow2 | | file | /v2/images/722db87f-10e2-4656-b7b8-0b78c2ef5aa8/file | | id | 722db87f-10e2-4656-b7b8-0b78c2ef5aa8 | | min_disk | 0 | | min_ram | 0 | | name | fedora-atomic-latest | | owner | c9aee696c4b54f12a645af2c951327dc | | properties | direct_url='rbd://55c25f6d-9e06-4a6a-8202-b087073ea8d6/cloud-dev/722db87f-10e2-4656-b7b8-0b78c2ef5aa8/snap', locations='[{u'url': u'rbd://55c25f6d-9e06-4a6a-8202-b087073ea8d6/cloud-dev/722db87f-10e2-4656-b7b8-0b78c2ef5aa8/snap', u'metadata': {}}]', os_distro='fedora-atomic', os_hash_algo=, os_hash_value=, os_hidden='False' | | protected | False | | schema | /v2/schemas/image | | size | 642286592 | | status | active | | tags | | | updated_at | 2019-09-09T12:37:54Z | | virtual_size | None | | visibility | private | +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ The cluster template looks like this: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | insecure_registry | - | | labels | {} | | updated_at | - | | floating_ip_enabled | True | | fixed_subnet | - | | master_flavor_id | m1.small | | uuid | a81fe72b-e699-44b8-a7d0-cfc7525a42fd | | no_proxy | - | | https_proxy | - | | tls_disabled | False | | keypair_id | - | | public | False | | http_proxy | - | | docker_volume_size | 3 | | server_type | vm | | external_network_id | External | | cluster_distro | fedora-atomic | | image_id | fedora-atomic-latest | | volume_driver | - | | registry_enabled | False | | docker_storage_driver | devicemapper | | apiserver_port | - | | name | kubernetes-cluster-template | | created_at | 2020-07-21T09:25:26+00:00 | | network_driver | flannel | | fixed_network | - | | coe | kubernetes | | flavor_id | m1.small | | master_lb_enabled | False | | dns_nameserver | 8.8.8.8 | | hidden | False | +-----------------------+--------------------------------------+ Regards Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ From: Feilong Wang <feilong@catalyst.net.nz> Sent: 21 July 2020 22:42 To: openstack-discuss@lists.openstack.org Subject: Re: Magnum: invalid format of client version Hi Alex, Could you please let me know what's the cluster image you're using Fedora Atomic or Fedora CoreOS? And can you pls show me your cluster template? On 22/07/20 12:53 am, Alexander Dibbo - UKRI STFC wrote: Hi, I have just deployed magnum into my train enviroment and am seeing the following error when creating any kind of cluster: This is a Train environment deployed from RDO packages (9.4.0-1). Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server [req-aa9ce18b-64eb-40ad-b1c0-b7c312402780 - - - - -] Exception during message handling: InvalidParameterValue: ERROR: UnsupportedVersion: : resources.worker_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version. Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server Traceback (most recent call last): Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/magnum/conductor/handlers/cluster_conductor.py", line 95, in cluster_create Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server raise e Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server InvalidParameterValue: ERROR: UnsupportedVersion: : resources.worker_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version. Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server #033[00m When logging in debug, I see a dump of a huge heat template and a 400 bad request from heatclient (as below) immediately before the above log excerpt: {"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": {"message": "UnsupportedVersion: : resources.master_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version.", "traceback": null, "type": "StackValidationFailed"}, "title": "Bad Request"} log_http_response /usr/lib/python2.7/site-packages/heatclient/common/http.py:157 More details are available in my question here: https://ask.openstack.org/en/question/128520/magnum-invalid-format-of-client... Any suggestions on where to look to set the client version it is complaining about would be much appreciated? Thanks Alex Regards Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI. -- Cheers & Best regards, Feilong Wang (王飞龙) Head of R&D Catalyst Cloud - Cloud Native New Zealand -------------------------------------------------------------------------- Tel: +64-48032246 Email: flwang@catalyst.net.nz<mailto:flwang@catalyst.net.nz> Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
Hi Alex Looks like the following errors are being emitted from Nova via Heat. Would you mind ensuring that you are able to spin up servers using nova CLI and then creating a simple heat stack via heat CLI. Cheers Bharat
On 21 Jul 2020, at 13:53, Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk> wrote:
Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server InvalidParameterValue: ERROR: UnsupportedVersion: : resources.worker_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version.
Hi Bharat, Creating a VM from the CLI works fine, as does the heat stack. In both cases using the same image, flavour and keypair/ Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ From: Bharat Kunwar <bharat@stackhpc.com> Sent: 22 July 2020 15:04 To: Dibbo, Alexander (STFC,RAL,SC) <alexander.dibbo@stfc.ac.uk> Cc: openstack-discuss@lists.openstack.org Subject: Re: Magnum: invalid format of client version Hi Alex Looks like the following errors are being emitted from Nova via Heat. Would you mind ensuring that you are able to spin up servers using nova CLI and then creating a simple heat stack via heat CLI. Cheers Bharat On 21 Jul 2020, at 13:53, Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk<mailto:alexander.dibbo@stfc.ac.uk>> wrote: Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server InvalidParameterValue: ERROR: UnsupportedVersion: : resources.worker_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.
Hi All, Any other ideas of what to try? Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ From: Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk> Sent: 22 July 2020 15:28 To: Bharat Kunwar <bharat@stackhpc.com> Cc: openstack-discuss@lists.openstack.org Subject: RE: Magnum: invalid format of client version Hi Bharat, Creating a VM from the CLI works fine, as does the heat stack. In both cases using the same image, flavour and keypair/ Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ From: Bharat Kunwar <bharat@stackhpc.com<mailto:bharat@stackhpc.com>> Sent: 22 July 2020 15:04 To: Dibbo, Alexander (STFC,RAL,SC) <alexander.dibbo@stfc.ac.uk<mailto:alexander.dibbo@stfc.ac.uk>> Cc: openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> Subject: Re: Magnum: invalid format of client version Hi Alex Looks like the following errors are being emitted from Nova via Heat. Would you mind ensuring that you are able to spin up servers using nova CLI and then creating a simple heat stack via heat CLI. Cheers Bharat On 21 Jul 2020, at 13:53, Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk<mailto:alexander.dibbo@stfc.ac.uk>> wrote: Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server InvalidParameterValue: ERROR: UnsupportedVersion: : resources.worker_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.
So it looks like heat is trying to not specify a microversion which is causing an issue with creating server groups. I have tried and I am able to create a server group and assign members with the cli. Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.
Ok, This definitely seems to be a heat issue. A heat stack created with a server group gives the same error Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ From: Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk> Sent: 24 July 2020 09:58 To: Bharat Kunwar <bharat@stackhpc.com> Cc: openstack-discuss@lists.openstack.org Subject: RE: Magnum: invalid format of client version
So it looks like heat is trying to not specify a microversion which is causing an issue with creating server groups. I have tried and I am able to create a server group and assign members with the cli. Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.
Hi Bharat, I actually just resolved this issue by adding [clients] Endpoint_type = internalURL To /etc/heat/heat.conf I now have a different issue: 2020-07-24 12:53:06.206 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key api_address 2020-07-24 12:53:06.207 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key kube_masters 2020-07-24 12:53:06.695 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key api_address 2020-07-24 12:53:06.696 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key kube_minions 2020-07-24 12:53:26.271 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.272 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_masters 2020-07-24 12:53:26.321 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.322 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_masters 2020-07-24 12:53:26.348 24280 ERROR magnum.drivers.heat.driver [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] Nodegroup error, stack status: CREATE_FAILED, stack_id: 77bed453-17f4-4 779-9883-b36a82d26578, reason: Resource CREATE failed: AuthorizationFailure: resources.kube_masters.resources[0].resources.kube-master: Authorization failed. 2020-07-24 12:53:26.670 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.671 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_minions 2020-07-24 12:53:26.727 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.728 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_minions 2020-07-24 12:53:26.755 24280 ERROR magnum.drivers.heat.driver [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] Nodegroup error, stack status: CREATE_FAILED, stack_id: 77bed453-17f4-4 779-9883-b36a82d26578, reason: Resource CREATE failed: AuthorizationFailure: resources.kube_masters.resources[0].resources.kube-master: Authorization failed. Regards Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ From: Bharat Kunwar <bharat@stackhpc.com> Sent: 24 July 2020 14:11 To: Dibbo, Alexander (STFC,RAL,SC) <alexander.dibbo@stfc.ac.uk> Cc: openstack-discuss@lists.openstack.org Subject: Re: Magnum: invalid format of client version Interesting, what version of Heat and Nova are you using? That looks like a nova client error which I guess Heat is using to make calls to Nova. On 24 Jul 2020, at 12:52, Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk<mailto:alexander.dibbo@stfc.ac.uk>> wrote: Ok, This definitely seems to be a heat issue. A heat stack created with a server group gives the same error Regards Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at:https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack:https://stfc-cloud.slack.com/ From: Alexander Dibbo - UKRI STFC <alexander.dibbo@stfc.ac.uk<mailto:alexander.dibbo@stfc.ac.uk>> Sent: 24 July 2020 09:58 To: Bharat Kunwar <bharat@stackhpc.com<mailto:bharat@stackhpc.com>> Cc: openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> Subject: RE: Magnum: invalid format of client version From doing some more digging I see this in the debug logs on heat 2020-07-24 08:33:10.618 2167043 INFO heat.engine.stack [req-249dfd80-48f4-44f5-9f0e-7c9ac743f047 - admin - default default] Exception in stack validation 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack Traceback (most recent call last): 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 909, in validate 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack result = res.validate() 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server_group.py", line 69, in validate 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack MICROVERSION_SOFT_POLICIES) 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/nova.py", line 107, in is_version_supported 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack self.get_max_microversion()) 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 236, in get_api_version 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack api_version = APIVersion(version_string) 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 77, in __init__ 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack raise exceptions.UnsupportedVersion(msg) 2020-07-24 08:33:10.618 2167043 ERROR heat.engine.stack UnsupportedVersion: Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of ve rsion. So it looks like heat is trying to not specify a microversion which is causing an issue with creating server groups. I have tried and I am able to create a server group and assign members with the cli. Regards Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at:https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack:https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.
Hi Bharat, I actually just resolved this issue by adding [clients] Endpoint_type = internalURL To /etc/heat/heat.conf I now have a different issue: 2020-07-24 12:53:06.206 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key api_address 2020-07-24 12:53:06.207 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key kube_masters 2020-07-24 12:53:06.695 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key api_address 2020-07-24 12:53:06.696 24280 WARNING magnum.drivers.heat.template_def [req-9c29405a-d31d-4bd8-8f56-34dc910ff5ce - - - - -] stack does not have output_key kube_minions 2020-07-24 12:53:26.271 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.272 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_masters 2020-07-24 12:53:26.321 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.322 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_masters 2020-07-24 12:53:26.348 24280 ERROR magnum.drivers.heat.driver [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] Nodegroup error, stack status: CREATE_FAILED, stack_id: 77bed453-17f4-4 779-9883-b36a82d26578, reason: Resource CREATE failed: AuthorizationFailure: resources.kube_masters.resources[0].resources.kube-master: Authorization failed. 2020-07-24 12:53:26.670 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.671 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_minions 2020-07-24 12:53:26.727 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key api_address 2020-07-24 12:53:26.728 24280 WARNING magnum.drivers.heat.template_def [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] stack does not have output_key kube_minions 2020-07-24 12:53:26.755 24280 ERROR magnum.drivers.heat.driver [req-5a19296c-754f-4443-99ce-a8377a927b44 - - - - -] Nodegroup error, stack status: CREATE_FAILED, stack_id: 77bed453-17f4-4 779-9883-b36a82d26578, reason: Resource CREATE failed: AuthorizationFailure: resources.kube_masters.resources[0].resources.kube-master: Authorization failed. Regards Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ From: Bharat Kunwar <bharat@stackhpc.com> Sent: 24 July 2020 14:11 To: Dibbo, Alexander (STFC,RAL,SC) <alexander.dibbo@stfc.ac.uk> Cc: openstack-discuss@lists.openstack.org Subject: Re: Magnum: invalid format of client version Interesting, what version of Heat and Nova are you using? That looks like a nova client error which I guess Heat is using to make calls to Nova. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.
participants (3)
-
Alexander Dibbo - UKRI STFC
-
Bharat Kunwar
-
Feilong Wang