After doing some experiments I realize that this is unrelated to heat. If I manually build 20 volumes, 2 of them will be stuck in "creating" status. (openstack) [abraden@chrnc-dev-os-build-01 kolla_multinode]$ for x in {1..20}; do echo $x; openstack volume create --image 42d2a2db-d741-4080-8e8a-a713696de7c2 --size 50 atest$x;done (openstack) [abraden@chrnc-dev-os-build-01 kolla_multinode]$ ovl|grep atest | 1ab37b22-e06f-4360-83b5-e8272a906746 | atest20 | available | 50 | | | 641e8edf-26c1-4c7c-a02f-a199891adcb2 | atest19 | available | 50 | | | 7687e401-1ea1-4eb7-99d5-232ac33ea6eb | atest18 | available | 50 | | | 64c14f24-69c6-42f2-90ad-60f464c5fd3f | atest17 | available | 50 | | | 0bcd4e8e-4d53-4f74-9d7c-001e6aaeef88 | atest16 | creating | 50 | | | 9748a5ad-7a09-4f2f-9abd-12d69a27ab90 | atest15 | available | 50 | | | 85e84467-eef9-447f-9ba9-2545bef857ef | atest14 | available | 50 | | | c55c17c4-3471-4a06-8cc1-497a7f17ec72 | atest13 | available | 50 | | | 7d43cce3-fc4e-4697-9a8f-ca38226ca6cf | atest12 | available | 50 | | | f5a2062a-f6ee-4475-841e-d2efe5b9d4af | atest11 | available | 50 | | | 2b223e8a-c252-4306-8372-e2642b7fbf33 | atest10 | available | 50 | | | 5ce06f71-0169-4e9c-8133-7b1f826573c4 | atest9 | available | 50 | | | 1b40e297-3c54-4260-a4e7-535e183c8d74 | atest8 | available | 50 | | | e1ce1693-ba65-4f12-a85d-8b54e9259ef5 | atest7 | available | 50 | | | 0e6e5d95-a0c0-4daf-b21e-ebd5c8d6cb69 | atest6 | creating | 50 | | | 520d6912-535f-40e7-9da9-9f717f1eba2b | atest5 | available | 50 | | | 4d970654-e83f-4b6a-9c19-f917c4ec2825 | atest4 | available | 50 | | | ad77c9cd-e7bf-428b-8b01-c847fd818c9c | atest3 | available | 50 | | | c7d34554-467d-4312-9e81-5e8d00dcc7be | atest2 | available | 50 | | | 30ea4cdf-9d1e-471b-822b-1dc81e38ff96 | atest1 | available | 50 | | From: Braden, Albert <C-Albert.Braden@charter.com> Sent: Wednesday, February 24, 2021 1:40 PM To: openstack-discuss@lists.openstack.org Subject: [EXTERNAL] RE: [heat] [kolla] Stack creation fails CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Turns out the failed key occurs when the previous keypair fails to delete with the stack, preventing the new keypair with the same name from being created, and I can fix it by manually deleting that keypair. I'm still looking for answers on [1]. From: Braden, Albert <C-Albert.Braden@charter.com<mailto:C-Albert.Braden@charter.com>> Sent: Wednesday, February 24, 2021 1:13 PM To: openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> Subject: [EXTERNAL] [heat] [kolla] Stack creation fails CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. I'm using a heat template to try and build an Openstack cluster on VMs and it is failing in 2 different ways. It starts out fine for both: (openstack) [abraden@chrnc-dev-os-build-01 kolla_multinode]$ os stack create -t kolla_multinode.yaml upgrade-stack +---------------------+----------------------------------------------------------------------------------------+ | Field | Value | +---------------------+----------------------------------------------------------------------------------------+ | id | 7b4a49d2-0418-453a-abde-5e510b588b71 | | stack_name | upgrade-stack | | description | Deploy virtualized multi-node OpenStack and Ceph using kolla-ansible and ceph-ansible. | | | | | creation_time | 2021-02-23T21:17:02Z | | updated_time | None | | stack_status | CREATE_IN_PROGRESS | | stack_status_reason | Stack CREATE started | +---------------------+----------------------------------------------------------------------------------------+ Sometimes it stays in CREATE_IN_PROGRESS status for a long time, and when I look at volumes I see one or more volumes stuck in "creating" status [1]. If I set the stuck volumes to "error" status then heat retries and the cluster builds successfully. Other times the keypair fails. First it goes to DELETE_COMPLETE status and then to CREATE_FAILED [2], and the the stack goes to CREATE_FAILED status. When this happens no volumes are created, and I haven't found a workaround for this failure. Where should I be looking for the issue? Nothing in the log is jumping out at me. RMQ and keystone seem fine. Searching for the id of the failed stack or the stuck volume in /var/log/kolla/* doesn't find any errors or warnings. [1] volumes<https://paste.ubuntu.com/p/ZBhn7TQ2Hv/> [2] failed key<https://paste.ubuntu.com/p/gGwDRyT2Nd/> I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.