<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">After doing some experiments I realize that this is unrelated to heat. If I manually build 20 volumes, 2 of them will be stuck in “creating” status.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">(openstack) [abraden@chrnc-dev-os-build-01 kolla_multinode]$ for x in {1..20}; do echo $x; openstack volume create --image 42d2a2db-d741-4080-8e8a-a713696de7c2 --size 50 atest$x;done<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">(openstack) [abraden@chrnc-dev-os-build-01 kolla_multinode]$ ovl|grep atest<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 1ab37b22-e06f-4360-83b5-e8272a906746 | atest20 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 641e8edf-26c1-4c7c-a02f-a199891adcb2 | atest19 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 7687e401-1ea1-4eb7-99d5-232ac33ea6eb | atest18 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 64c14f24-69c6-42f2-90ad-60f464c5fd3f | atest17 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 0bcd4e8e-4d53-4f74-9d7c-001e6aaeef88 | atest16 | creating | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 9748a5ad-7a09-4f2f-9abd-12d69a27ab90 | atest15 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 85e84467-eef9-447f-9ba9-2545bef857ef | atest14 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| c55c17c4-3471-4a06-8cc1-497a7f17ec72 | atest13 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 7d43cce3-fc4e-4697-9a8f-ca38226ca6cf | atest12 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| f5a2062a-f6ee-4475-841e-d2efe5b9d4af | atest11 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 2b223e8a-c252-4306-8372-e2642b7fbf33 | atest10 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 5ce06f71-0169-4e9c-8133-7b1f826573c4 | atest9 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 1b40e297-3c54-4260-a4e7-535e183c8d74 | atest8 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| e1ce1693-ba65-4f12-a85d-8b54e9259ef5 | atest7 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 0e6e5d95-a0c0-4daf-b21e-ebd5c8d6cb69 | atest6 | creating | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 520d6912-535f-40e7-9da9-9f717f1eba2b | atest5 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 4d970654-e83f-4b6a-9c19-f917c4ec2825 | atest4 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| ad77c9cd-e7bf-428b-8b01-c847fd818c9c | atest3 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| c7d34554-467d-4312-9e81-5e8d00dcc7be | atest2 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">| 30ea4cdf-9d1e-471b-822b-1dc81e38ff96 | atest1 | available | 50 | |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Braden, Albert <C-Albert.Braden@charter.com> <br>
<b>Sent:</b> Wednesday, February 24, 2021 1:40 PM<br>
<b>To:</b> openstack-discuss@lists.openstack.org<br>
<b>Subject:</b> [EXTERNAL] RE: [heat] [kolla] Stack creation fails<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:solid #5A5A5A 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:#235C70"><strong><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:white">CAUTION:</span></strong><span style="font-size:10.0pt;color:white"> The e-mail below is from an external source.
Please exercise caution before opening attachments, clicking links, or following guidance.
<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D">Turns out the failed key occurs when the previous keypair fails to delete with the stack, preventing the new keypair with the same name from being created, and I can fix it by manually deleting that keypair.
I’m still looking for answers on [1].<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Braden, Albert <<a href="mailto:C-Albert.Braden@charter.com">C-Albert.Braden@charter.com</a>>
<br>
<b>Sent:</b> Wednesday, February 24, 2021 1:13 PM<br>
<b>To:</b> <a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a><br>
<b>Subject:</b> [EXTERNAL] [heat] [kolla] Stack creation fails<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:solid #5A5A5A 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:#235C70"><strong><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:white">CAUTION:</span></strong><span style="font-size:10.0pt;color:white"> The e-mail below is from an external source.
Please exercise caution before opening attachments, clicking links, or following guidance.
<o:p></o:p></span></p>
</div>
<p class="MsoNormal">I’m using a heat template to try and build an Openstack cluster on VMs and it is failing in 2 different ways. It starts out fine for both:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">(openstack) [abraden@chrnc-dev-os-build-01 kolla_multinode]$ os stack create -t kolla_multinode.yaml upgrade-stack<o:p></o:p></p>
<p class="MsoNormal">+---------------------+----------------------------------------------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal">| Field | Value |<o:p></o:p></p>
<p class="MsoNormal">+---------------------+----------------------------------------------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal">| id | 7b4a49d2-0418-453a-abde-5e510b588b71 |<o:p></o:p></p>
<p class="MsoNormal">| stack_name | upgrade-stack |<o:p></o:p></p>
<p class="MsoNormal">| description | Deploy virtualized multi-node OpenStack and Ceph using kolla-ansible and ceph-ansible. |<o:p></o:p></p>
<p class="MsoNormal">| | |<o:p></o:p></p>
<p class="MsoNormal">| creation_time | 2021-02-23T21:17:02Z |<o:p></o:p></p>
<p class="MsoNormal">| updated_time | None |<o:p></o:p></p>
<p class="MsoNormal">| stack_status | CREATE_IN_PROGRESS |<o:p></o:p></p>
<p class="MsoNormal">| stack_status_reason | Stack CREATE started |<o:p></o:p></p>
<p class="MsoNormal">+---------------------+----------------------------------------------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Sometimes it stays in CREATE_IN_PROGRESS status for a long time, and when I look at volumes I see one or more volumes stuck in “creating” status [1]. If I set the stuck volumes to “error” status then heat retries and the cluster builds
successfully.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Other times the keypair fails. First it goes to DELETE_COMPLETE status and then to CREATE_FAILED [2], and the the stack goes to CREATE_FAILED status. When this happens no volumes are created, and I haven’t found a workaround for this failure.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Where should I be looking for the issue? Nothing in the log is jumping out at me. RMQ and keystone seem fine. Searching for the id of the failed stack or the stuck volume in /var/log/kolla/* doesn’t find any errors or warnings.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[1] <a href="https://paste.ubuntu.com/p/ZBhn7TQ2Hv/">volumes</a><o:p></o:p></p>
<p class="MsoNormal">[2] <a href="https://paste.ubuntu.com/p/gGwDRyT2Nd/">failed key</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">The contents of this e-mail message and
<br>
any attachments are intended solely for the <br>
addressee(s) and may contain confidential <br>
and/or legally privileged information. If you<br>
are not the intended recipient of this message<br>
or if this message has been addressed to you <br>
in error, please immediately alert the sender<br>
by reply e-mail and then delete this message <br>
and any attachments. If you are not the <br>
intended recipient, you are notified that <br>
any use, dissemination, distribution, copying,<br>
or storage of this message or any attachment <br>
is strictly prohibited.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">The contents of this e-mail message and
<br>
any attachments are intended solely for the <br>
addressee(s) and may contain confidential <br>
and/or legally privileged information. If you<br>
are not the intended recipient of this message<br>
or if this message has been addressed to you <br>
in error, please immediately alert the sender<br>
by reply e-mail and then delete this message <br>
and any attachments. If you are not the <br>
intended recipient, you are notified that <br>
any use, dissemination, distribution, copying,<br>
or storage of this message or any attachment <br>
is strictly prohibited.<o:p></o:p></span></p>
</div>
The contents of this e-mail message and <br>any attachments are intended solely for the <br>addressee(s) and may contain confidential <br>and/or legally privileged information. If you<br>are not the intended recipient of this message<br>or if this message has been addressed to you <br>in error, please immediately alert the sender<br>by reply e-mail and then delete this message <br>and any attachments. If you are not the <br>intended recipient, you are notified that <br>any use, dissemination, distribution, copying,<br>or storage of this message or any attachment <br>is strictly prohibited.</body>
</html>