<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.pre
{mso-style-name:pre;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:248851171;
mso-list-template-ids:-188975494;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1617903099;
mso-list-template-ids:-357808256;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Can anyone help with this upgrade issue?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Braden, Albert <br>
<b>Sent:</b> Monday, April 19, 2021 8:20 AM<br>
<b>To:</b> openstack-discuss@lists.openstack.org<br>
<b>Subject:</b> [kolla] VM build fails after Train-Ussuri upgrade<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I upgraded my Train test cluster to Ussuri following these instructions:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#upgrade-procedure">OpenStack Docs: Operating Kolla</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The upgrade completed successfully with no failures, and the existing VMs are fine, but new VM build fails with
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:black;background:white">
rados.Rados.connect\nrados.PermissionDeniedError:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:black;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><a href="https://paste.ubuntu.com/p/MJhzgjYbW8/">Ubuntu Pastebin</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m running external ceph so I looked at this document:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html">OpenStack Docs: External Ceph</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It says that I need the following in <span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333;background:white">
</span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black;background:white">/etc/kolla/config/glance/ceph.conf:<o:p></o:p></span></b></span></p>
<p class="MsoNormal"><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black;background:white"><o:p> </o:p></span></b></span></p>
<p class="MsoNormal">auth_cluster_required = cephx<o:p></o:p></p>
<p class="MsoNormal">auth_service_required = cephx<o:p></o:p></p>
<p class="MsoNormal">auth_client_required = cephx<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I didn’t have that, so I added it and then redeployed, but still can’t build VMs. I tried adding the same to all copies of ceph.conf and redeployed again, but that didn’t help. Does anything else need to change in my ceph config when upgrading
from Train to Ussuri? I see some cryptic talk about ceph in the release notes but it’s not obvious what I’m being asked to change:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://docs.openstack.org/releasenotes/kolla-ansible/ussuri.html#relnotes-10-0-0-stable-ussuri-upgrade-notes">OpenStack Docs: Ussuri Series Release Notes</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I read the bug that it refers to: <a href="https://bugs.launchpad.net/kolla-ansible/+bug/1904062">
Bug #1904062 “external ceph cinder volume config breaks volumes ...” : Bugs : kolla-ansible (launchpad.net)</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But I already have “backend_host=rbd:volumes” so I don’t think I’m hitting that.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Also I read these sections but I don’t see anything obvious here that needs to be changed:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:7.5pt;margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;color:#333333"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333;background:white">For cinder (</span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black;background:white">cinder-volume</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333;background:white"> and </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black;background:white">cinder-backup</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333;background:white">), </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black;background:white">glance-api</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333;background:white"> and </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black;background:white">manila</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333;background:white"> keyrings
behavior has changed and Kolla Ansible deployment will not copy those keys using wildcards (</span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black;background:white">ceph.*</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333;background:white">),
instead will use newly introduced variables. Your environment may render unusable after an upgrade if your keys in </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black;background:white">/etc/kolla/config</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333;background:white"> do
not match default values for introduced variables.</span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"><o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:7.5pt;margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo2;background:white;box-sizing: border-box">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;color:#333333"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333">The default behavior for generating the </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">cinder.conf</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"> template
has changed. An </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">rbd-1</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"> section will be generated when external Ceph
functionality is used, i.e. </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">cinder_backend_ceph</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"> is set to </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">true</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333">.
Previously it was only included when Kolla Ansible internal Ceph deployment mechanism was used.<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:7.5pt;margin-left:.5in;text-indent:-.25in;mso-list:l1 level1 lfo4;background:white;box-sizing: border-box">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;color:#333333"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333">The </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">rbd</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"> section
of </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">nova.conf</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"> for </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">nova-compute</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"> is
now generated when </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">nova_backend</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"> is set to </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">"rbd"</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333">.
Previously it was only generated when both </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">enable_ceph</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"> was </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">"yes"</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"> and </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">nova_backend</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333"> was
set to </span><span class="pre"><b><span style="font-size:10.5pt;font-family:Consolas;color:black">"rbd"</span></b></span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#333333">.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My ceph keys have the default name and are in the default locations. I have cinder_backend_ceph: "yes". I don’t have a nova_backend setting but I have nova_backend_ceph: "yes"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I added nova_backend: "rbd" and redeployed and now I get a different error: rados.Rados.connect\nrados.ObjectNotFound<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://paste.ubuntu.com/p/2kdKCW7npw/">Ubuntu Pastebin</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
The contents of this e-mail message and <br>any attachments are intended solely for the <br>addressee(s) and may contain confidential <br>and/or legally privileged information. If you<br>are not the intended recipient of this message<br>or if this message has been addressed to you <br>in error, please immediately alert the sender<br>by reply e-mail and then delete this message <br>and any attachments. If you are not the <br>intended recipient, you are notified that <br>any use, dissemination, distribution, copying,<br>or storage of this message or any attachment <br>is strictly prohibited.</body>
</html>