<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.pre
{mso-style-name:pre;}
span.EstiloCorreo24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:248851171;
mso-list-template-ids:-188975494;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1617903099;
mso-list-template-ids:-357808256;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ES" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Did you check your keys files in nova, kvm and cinder generated by ceph to auth?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Example to generate key file and then copy to compute:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">ceph auth get-or-create compute mon ‘allow r’ osd ‘allow class-read object_prefix rbd_children, allow allow rwx pool=compute, allow allow rwx pool=volumes, allow rx pool=images’ -o /ceph.client.compute.keyring<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hope this help you.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Regards.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>De:</b> Braden, Albert <C-Albert.Braden@charter.com> <br>
<b>Enviado el:</b> lunes, 26 de abril de 2021 15:46<br>
<b>Para:</b> openstack-discuss@lists.openstack.org<br>
<b>Asunto:</b> RE: [kolla] VM build fails after Train-Ussuri upgrade<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><strong><span lang="EN-US" style="color:#FF3333">everis Security Awareness </span>
</strong><span lang="EN-US">- This is an incoming mail from an EXTERNAL DOMAIN. Please verify sender before you open attachments or access links.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Can anyone help with this upgrade issue?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Braden, Albert
<br>
<b>Sent:</b> Monday, April 19, 2021 8:20 AM<br>
<b>To:</b> <a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a><br>
<b>Subject:</b> [kolla] VM build fails after Train-Ussuri upgrade<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I upgraded my Train test cluster to Ussuri following these instructions:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#upgrade-procedure">OpenStack Docs: Operating Kolla</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The upgrade completed successfully with no failures, and the existing VMs are fine, but new VM build fails with
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:black;background:white">rados.Rados.connect\nrados.PermissionDeniedError:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:black;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://paste.ubuntu.com/p/MJhzgjYbW8/">Ubuntu Pastebin</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I’m running external ceph so I looked at this document:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html">OpenStack Docs: External Ceph</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">It says that I need the following in </span>
<span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333;background:white"> </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black;background:white">/etc/kolla/config/glance/ceph.conf:</span></b></span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black;background:white"><o:p></o:p></span></b></span></p>
<p class="MsoNormal"><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black;background:white"><o:p> </o:p></span></b></span></p>
<p class="MsoNormal"><span lang="EN-US">auth_cluster_required = cephx</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">auth_service_required = cephx<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">auth_client_required = cephx<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I didn’t have that, so I added it and then redeployed, but still can’t build VMs. I tried adding the same to all copies of ceph.conf and redeployed again, but that didn’t help. Does anything else need to change in my
ceph config when upgrading from Train to Ussuri? I see some cryptic talk about ceph in the release notes but it’s not obvious what I’m being asked to change:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://docs.openstack.org/releasenotes/kolla-ansible/ussuri.html#relnotes-10-0-0-stable-ussuri-upgrade-notes">OpenStack Docs: Ussuri Series Release Notes</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I read the bug that it refers to: <a href="https://bugs.launchpad.net/kolla-ansible/+bug/1904062">
Bug #1904062 “external ceph cinder volume config breaks volumes ...” : Bugs : kolla-ansible (launchpad.net)</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">But I already have “backend_host=rbd:volumes” so I don’t think I’m hitting that.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Also I read these sections but I don’t see anything obvious here that needs to be changed:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li style="color:#333333;margin-top:0cm;margin-bottom:7.5pt;mso-list:l0 level1 lfo2;background:white">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;background:white">For cinder (</span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black;background:white">cinder-volume</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;background:white"> and </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black;background:white">cinder-backup</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;background:white">), </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black;background:white">glance-api</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;background:white"> and </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black;background:white">manila</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;background:white"> keyrings
behavior has changed and Kolla Ansible deployment will not copy those keys using wildcards (</span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black;background:white">ceph.*</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;background:white">),
instead will use newly introduced variables. Your environment may render unusable after an upgrade if your keys in </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black;background:white">/etc/kolla/config</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;background:white"> do
not match default values for introduced variables.</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"><o:p></o:p></span></li><li style="color:#333333;margin-top:0cm;margin-bottom:7.5pt;mso-list:l0 level1 lfo2;background:white">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">The default behavior for generating the </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">cinder.conf</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> template
has changed. An </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">rbd-1</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> section will be generated when external
Ceph functionality is used, i.e. </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">cinder_backend_ceph</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> is
set to </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">true</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">. Previously it was only included when Kolla
Ansible internal Ceph deployment mechanism was used.<o:p></o:p></span></li></ul>
<ul style="margin-top:0cm" type="disc">
<li style="color:#333333;margin-top:0cm;margin-bottom:7.5pt;mso-list:l1 level1 lfo4;background:white">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">The </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">rbd</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> section
of </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">nova.conf</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> for </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">nova-compute</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> is
now generated when </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">nova_backend</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> is set to </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">"rbd"</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">.
Previously it was only generated when both </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">enable_ceph</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> was </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">"yes"</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> and </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">nova_backend</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> was
set to </span><span class="pre"><b><span lang="EN-US" style="font-size:10.5pt;font-family:Consolas;color:black">"rbd"</span></b></span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">.<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">My ceph keys have the default name and are in the default locations. I have cinder_backend_ceph: "yes". I don’t have a nova_backend setting but I have nova_backend_ceph: "yes"<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I added nova_backend: "rbd" and redeployed and now I get a different error: rados.Rados.connect\nrados.ObjectNotFound<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://paste.ubuntu.com/p/2kdKCW7npw/">Ubuntu Pastebin</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The contents of this e-mail message and <br>
any attachments are intended solely for the <br>
addressee(s) and may contain confidential <br>
and/or legally privileged information. If you<br>
are not the intended recipient of this message<br>
or if this message has been addressed to you <br>
in error, please immediately alert the sender<br>
by reply e-mail and then delete this message <br>
and any attachments. If you are not the <br>
intended recipient, you are notified that <br>
any use, dissemination, distribution, copying,<br>
or storage of this message or any attachment <br>
is strictly prohibited.<o:p></o:p></span></p>
</div>
</body>
</html>