Hi, I have just deployed magnum into my train enviroment and am seeing the following error when creating any kind of cluster: This is a Train environment deployed from RDO packages (9.4.0-1). Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server [req-aa9ce18b-64eb-40ad-b1c0-b7c312402780 - - - - -] Exception during message handling: InvalidParameterValue: ERROR: UnsupportedVersion: : resources.worker_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version. Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server Traceback (most recent call last): Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/magnum/conductor/handlers/cluster_conductor.py", line 95, in cluster_create Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server raise e Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server InvalidParameterValue: ERROR: UnsupportedVersion: : resources.worker_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version. Jul 17 14:08:24 host-172-16-103-43 magnum-conductor: 2020-07-17 14:08:24.942 6251 ERROR oslo_messaging.rpc.server #033[00m When logging in debug, I see a dump of a huge heat template and a 400 bad request from heatclient (as below) immediately before the above log excerpt: {"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": {"message": "UnsupportedVersion: : resources.master_nodes_server_group: : Invalid format of client version ''. Expected format 'X.Y', where X is a major part and Y is a minor part of version.", "traceback": null, "type": "StackValidationFailed"}, "title": "Bad Request"} log_http_response /usr/lib/python2.7/site-packages/heatclient/common/http.py:157 Here is my config file: [DEFAULT] auth_strategy = keystone debug = true memcached_servers = dev-service1.nubes.rl.ac.uk:11211 my_ip = 172.16.103.43 rpc_backend = rabbit stack_domain_admin = magnum stack_domain_admin_password = MAGNUM_PASS stack_user_domain_name = magnum transport_url = rabbit://openstack:rabbit-pass@dev-rabbit4.nubes.rl.ac.uk:5672,openstack:rabbit-pass@dev-rabbit5.nubes.rl.ac.uk:5672,openstack:rabbit-pass@dev-rabbit6.nubes.rl.ac.uk:5672/ verbose = true [api] host = 172.16.103.43 [barbican_client] endpoint_type = public region_name = RegionOne [cache] backend = oslo_cache.memcache_pool enabled = true memcache_servers = dev-service1.nubes.rl.ac.uk:11211 [certificates] cert_manager_type = x509keypair [cinder_client] endpoint_type = public region_name = RegionOne [database] connection = mysql+pymysql://magnum:MAGNUM_DBPASS@dev-openstack.stfc.ac.uk:3306/magnum connection_recycle_time = 3600 [glance_client] endpoint_type = public region_name = RegionOne [heat_client] endpoint_type = public region_name = RegionOne [keystone_authtoken] admin_password = MAGNUM_PASS admin_tenant_name = service admin_user = magnum auth_plugin = password auth_type = password auth_uri = https://dev-openstack.stfc.ac.uk:5000 auth_url = https://dev-openstack.stfc.ac.uk:5000 insecure = false password = MAGNUM_PASS project_domain_name = default project_name = service user_domain_name = default username = magnum www_authenticate_uri = https://dev-openstack.stfc.ac.uk:5000 [magnum_client] endpoint_type = public region_name = RegionOne [neutron_client] endpoint_type = public region_name = RegionOne [nova_client] endpoint_type = public region_name = RegionOne [octavia_client] endpoint_type = public region_name = RegionOne [oslo_concurrency] lock_path = /var/lib/magnum/tmp [oslo_middleware] enable_proxy_headers_parsing = true [oslo_messaging_notifications] driver = messaging [trust] trustee_domain_admin_name = magnum_domain_admin trustee_domain_admin_password = MAGNUM_DOMAIN_ADMIN_PASS trustee_domain_name = magnum trustee_keystone_interface = public Any suggestions on where to look to set the client version it is complaining about would be much appreciated? Thanks Alex Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support@gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.