Hi Ammad It seems to be the same as the cause of the bug report I issued to Heat, but there has been no response from the Heat team. https://storyboard.openstack.org/#!/story/2009164 To: Heat team Could you please confirm this problem? Thanks. Regards, Ueha From: Ammad Syed <syedammad83@gmail.com> Sent: Friday, October 8, 2021 4:14 PM To: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: [heat] xena stack deployment failed Hi, I have upgraded my heat from wallaby to xena. When I am trying to create magnum cluster its giving below error in heat engine logs. Currently in whole stack, I have upgraded heat and magnum to latest release. Before upgrading heat from xena to wallaby, the stack deployment was successful. 2021-10-08 12:06:14.107 2064 DEBUG novaclient.v2.client [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] REQ: curl -g -i -X GET http://controller-khi04.rapid.pk:8774/v2.1/98687873a146418eaeeb54a01693669f -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}ef71182888b7ad3b8719ee8d48e37c6cd526e1f25f8395f733770917aead9b7b" -H "X-OpenStack-Nova-API-Version: 2.1" _http_log_request /usr/lib/python3/dist-packages/keystoneauth1/session.py:519 2021-10-08 12:06:14.121 2064 DEBUG novaclient.v2.client [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] RESP: [404] Connection: keep-alive Content-Length: 112 Content-Type: application/json Date: Fri, 08 Oct 2021 07:06:14 GMT X-Compute-Request-Id: req-d07e25b0-375f-48e6-ac4b-d76b41848e6a X-Openstack-Request-Id: req-d07e25b0-375f-48e6-ac4b-d76b41848e6a _http_log_response /usr/lib/python3/dist-packages/keystoneauth1/session.py:550 2021-10-08 12:06:14.121 2064 DEBUG novaclient.v2.client [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] RESP BODY: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"} _http_log_response /usr/lib/python3/dist-packages/keystoneauth1/session.py:582 2021-10-08 12:06:14.122 2064 DEBUG novaclient.v2.client [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] GET call to compute for http://controller-khi04.rapid.pk:8774/v2.1/98687873a146418eaeeb54a01693669f used request id req-d07e25b0-375f-48e6-ac4b-d76b41848e6a request /usr/lib/python3/dist-packages/keystoneauth1/session.py:954 2021-10-08 12:06:14.122 2064 DEBUG novaclient.v2.client [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] REQ: curl -g -i -X GET http://controller-khi04.rapid.pk:8774/v2.1/ -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}ef71182888b7ad3b8719ee8d48e37c6cd526e1f25f8395f733770917aead9b7b" -H "X-OpenStack-Nova-API-Version: 2.1" _http_log_request /usr/lib/python3/dist-packages/keystoneauth1/session.py:519 2021-10-08 12:06:14.128 2064 DEBUG novaclient.v2.client [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] RESP: [200] Connection: keep-alive Content-Length: 399 Content-Type: application/json Date: Fri, 08 Oct 2021 07:06:14 GMT Openstack-Api-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-9b1f2443-fdee-41dd-9139-57376afd7bef X-Openstack-Nova-Api-Version: 2.1 X-Openstack-Request-Id: req-9b1f2443-fdee-41dd-9139-57376afd7bef _http_log_response /usr/lib/python3/dist-packages/keystoneauth1/session.py:550 2021-10-08 12:06:14.128 2064 DEBUG novaclient.v2.client [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] RESP BODY: {"version": {"id": "v2.1", "status": "CURRENT", "version": "2.88", "min_version": "2.1", "updated": "2013-07-23T11:33:21Z", "links": [{"rel": "self", "href": "http://controller-khi04.rapid.pk:8774/v2.1/"}, {"rel": "describedby", "type": "text/html", "href": "http://docs.openstack.org/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}]}} _http_log_response /usr/lib/python3/dist-packages/keystoneauth1/session.py:582 2021-10-08 12:06:14.128 2064 DEBUG novaclient.v2.client [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] GET call to compute for http://controller-khi04.rapid.pk:8774/v2.1/ used request id req-9b1f2443-fdee-41dd-9139-57376afd7bef request /usr/lib/python3/dist-packages/keystoneauth1/session.py:954 2021-10-08 12:06:14.129 2064 INFO heat.engine.resource [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] CREATE: ServerGroup "worker_nodes_server_group" Stack "k8s-cluster6-7cnnuz4hrfrz" [5501d6a4-59a6-4f76-b25e-ec43e0822361] 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource Traceback (most recent call last): 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 916, in _action_recorder 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource yield 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 1028, in _do_action 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource yield from self.action_handler_task(action, args=handler_args) 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resource.py", line 970, in action_handler_task 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource handler_data = handler(*args) 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/heat/engine/resources/openstack/nova/server_group.py", line 98, in handle_create 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource server_group = client.server_groups.create(name=name, 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource File "/usr/lib/python3/dist-packages/novaclient/api_versions.py", line 393, in substitution 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource return methods[-1].func(obj, *args, **kwargs) 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource TypeError: create() got an unexpected keyword argument 'policies' 2021-10-08 12:06:14.129 2064 ERROR heat.engine.resource 2021-10-08 12:06:14.143 2064 INFO heat.engine.stack [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] Stack CREATE FAILED (k8s-cluster6-7cnnuz4hrfrz): Resource CREATE failed: TypeError: resources.worker_nodes_server_group: create() got an unexpected keyword argument 'policies' 2021-10-08 12:06:14.146 2064 DEBUG heat.engine.stack [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] Persisting stack k8s-cluster6-7cnnuz4hrfrz status CREATE FAILED _send_notification_and_add_event /usr/lib/python3/dist-packages/heat/engine/stack.py:1109 2021-10-08 12:06:15.009 2061 INFO heat.engine.stack [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] Triggering resource 2434 for update 2021-10-08 12:06:15.040 2063 DEBUG heat.engine.worker [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] [88ec8f6f-7e10-4c6a-be97-03de82938eb2] Traversal cancelled; re-trigerring. check_resource /usr/lib/python3/dist-packages/heat/engine/worker.py:193 2021-10-08 12:06:16.016 2061 INFO heat.engine.stack [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] Triggering resource 2437 for update 2021-10-08 12:06:16.048 2061 DEBUG heat.engine.worker [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] [88ec8f6f-7e10-4c6a-be97-03de82938eb2] Traversal cancelled; re-trigerring. check_resource /usr/lib/python3/dist-packages/heat/engine/worker.py:193 2021-10-08 12:06:17.026 2061 INFO heat.engine.stack [req-31d11ee8-fc47-4399-b424-97adabf31331 admin admin - - -] Triggering resource 2443 for update 2021-10-08 12:06:17.066 2062 DEBUG heat.engine.worker [req-61cb3eba-0cf0-47f7-8fdb-8e9375888dc4 - - - - -] [88ec8f6f-7e10-4c6a-be97-03de82938eb2] Traversal cancelled; re-trigerring. check_resource /usr/lib/python3/dist-packages/heat/engine/worker.py:193 - Ammad