<div dir="ltr"><div class="gmail_default" style="font-size:small;color:#444444">Hello Laurent,</div><div class="gmail_default" style="font-size:small;color:#444444"><br></div><div class="gmail_default" style="font-size:small;color:#444444">It turns out I only have one keyring in the container.</div><div class="gmail_default" style="font-size:small;color:#444444"><br></div><div class="gmail_default" style="font-size:small;color:#444444">root@compute1:/home/stack# docker exec -it nova_compute bash<br>(nova-compute)[nova@compute1 /]$ cd /etc/ceph/<br>(nova-compute)[nova@compute1 ceph]$ ls<br>ceph.client.nova.keyring ceph.conf rbdmap<br></div><div class="gmail_default" style="font-size:small;color:#444444"><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div><font face="arial, sans-serif">Regards</font></div><div><font face="arial, sans-serif"><br></font></div><div dir="ltr"><font face="arial, sans-serif">Tony Karera</font></div><div dir="ltr"><br></div><div><br></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont <<a href="mailto:laurentfdumont@gmail.com">laurentfdumont@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container (<span style="color:rgb(68,68,68)">etc/ceph/ceph.client.cinder.</span><span style="color:rgb(68,68,68)">keyring,/etc/ceph/ceph.</span><span style="color:rgb(68,68,68)">keyring,/etc/ceph/keyring,/</span><span style="color:rgb(68,68,68)">etc/ceph/keyring.bin)</span></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Hello Sean,</div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br></div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">Below are the output on the compute node and deployment</div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br></div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">root@compute1:/etc/kolla/nova-compute# ls<br>ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf<br></div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br></div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">(kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/<br>ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf<br></div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br></div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)">And I can confirm that the content is the same.</div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br></div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br></div><div class="gmail_default" style="font-size:small;color:rgb(68,68,68)"><br></div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div><font face="arial, sans-serif">Regards</font></div><div><font face="arial, sans-serif"><br></font></div><div dir="ltr"><font face="arial, sans-serif">Tony Karera</font></div><div dir="ltr"><br></div><div><br></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote:<br>
> I would investigate that compute error first. Creating volumes means the<br>
> controllers are doing the action. Starting a VM on a compute means you also<br>
> need Ceph to works on the compute to mount the rdb target.<br>
<br>
nova as part of its startup process in aintiallying the resouce tracker will<br>
try to connect to ceph if you are using the rbd image backend to report how much stroage<br>
is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as<br>
then that will block the agent from starting up fully and will cause it to be missing the hypervior list.<br>
<br>
the error seams to indicate that the cinder keyring is not in the nova container<br>
that likely means you have not put it in /etc/kolla/config/nova<br>
i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/<br>
on the compute node to ensure the cinder keyring is actully copied and has the correct content<br>
<br>
i have <br>
stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/<br>
ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf<br>
<br>
<br>
[client.cinder]<br>
key = *********************************<br>
caps mgr = "profile rbd pool=volumes, profile rbd pool=vms"<br>
caps mon = "profile rbd"<br>
caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images"<br>
stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring<br>
[client.nova]<br>
key = *********************************<br>
caps mgr = "profile rbd pool=volumes, profile rbd pool=vms"<br>
caps mon = "profile rbd"<br>
caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images"<br>
<br>
blanked out the key wiht *************** after the fact but you should have something similar<br>
<br>
<br>
in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool <br>
so i neede to modify my ceph.conf to make that work with kolla <br>
<br>
stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf<br>
# minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f<br>
[global]<br>
fsid = *********************<br>
mon_host = [*********************]<br>
<br>
[client.glance]<br>
rbd default data pool = images-data<br>
<br>
[client.cinder]<br>
rbd default data pool = volumes-data<br>
<br>
[client.nova]<br>
rbd default data pool = vms-data<br>
<br>
using 2 keyrings/user allows me to set different default data pools for cinder and nova.<br>
<br>
> <br>
> Working in Wallaby with the error doesn't mean it would 100% work in<br>
> Victoria.<br>
> <br>
> On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>> wrote:<br>
> <br>
> > Hey Guys, Any other idea ?<br>
> > <br>
> > Regards<br>
> > <br>
> > Tony Karera<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>> wrote:<br>
> > <br>
> > > Just to add on that,<br>
> > > <br>
> > > compute service is listed, I can create Volumes, I have the same cinder<br>
> > > keyring in the /etc/kolla/config/nova directory as I have in the<br>
> > > /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring<br>
> > > Regards<br>
> > > <br>
> > > Tony Karera<br>
> > > <br>
> > > <br>
> > > <br>
> > > <br>
> > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <<a href="mailto:tonykarera@gmail.com" target="_blank">tonykarera@gmail.com</a>> wrote:<br>
> > > <br>
> > > > Hello Guys,<br>
> > > > <br>
> > > > Thanks a lot.<br>
> > > > <br>
> > > > I had actually checked the nova -compute.log on the compute node and<br>
> > > > they were showing the error I will post at the end about the cinder keyring<br>
> > > > but I know its correct because its the same I was using on wallaby, I even<br>
> > > > tried to use another ceph cluster with ofcouse different keyrings but its<br>
> > > > the same issue.<br>
> > > > <br>
> > > > Below is the error<br>
> > > > <br>
> > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to<br>
> > > > find a keyring on<br>
> > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:<br>
> > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1<br>
> > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at<br>
> > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,<br>
> > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable<br>
> > > > to find a keyring on<br>
> > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:<br>
> > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1<br>
> > > > AuthRegistry(0x7fbcdc060698) no keyring found at<br>
> > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,<br>
> > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable<br>
> > > > to find a keyring on<br>
> > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:<br>
> > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1<br>
> > > > AuthRegistry(0x7fbce2f4e020) no keyring found at<br>
> > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,<br>
> > > > disabling cephx\n[errno 2] RADOS object not found (error connecting to the<br>
> > > > cluster)\n'<br>
> > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager<br>
> > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of<br>
> > > > the above exception, another exception occurred:<br>
> > > > Regards<br>
> > > > <br>
> > > > Tony Karera<br>
> > > > <br>
> > > > <br>
> > > > <br>
> > > > <br>
> > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>> wrote:<br>
> > > > <br>
> > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote:<br>
> > > > > > It could also be a compute cell discovery issue maybe?<br>
> > > > > no they shoudl still show up in the hypervior list api<br>
> > > > > > <br>
> > > > > > Do you see anything under "openstack compute service list"?<br>
> > > > > if they show up in the service list but not they hyperiors api it<br>
> > > > > means that the comptue service started and registered its service entry<br>
> > > > > but<br>
> > > > > something broke it before it could create a compute node recored in the<br>
> > > > > db.<br>
> > > > > <br>
> > > > > with ceph the case i have hit this most often is when the keyright used<br>
> > > > > by nova to<br>
> > > > > get the avaiable capastiy of the ceph cluster is wrong whihc prevent<br>
> > > > > the resoucetack and compute manager<br>
> > > > > form actully creating the compute node record.<br>
> > > > > <br>
> > > > > <br>
> > > > > it can happen for other reason too but best place to start is check if<br>
> > > > > there is an error in the nova compute agent log and go from there.<br>
> > > > > > <br>
> > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>><br>
> > > > > wrote:<br>
> > > > > > <br>
> > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote:<br>
> > > > > > > > Hello Team,<br>
> > > > > > > > <br>
> > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu<br>
> > > > > 20.04<br>
> > > > > > > and<br>
> > > > > > > > ceph as the backend storage for Nova, Cinder and Glance.<br>
> > > > > > > > <br>
> > > > > > > > It finished with no error but it has failed to register any on the<br>
> > > > > > > Compute<br>
> > > > > > > > Nodes under Hypervisors.<br>
> > > > > > > > <br>
> > > > > > > > kolla-openstack) stack@deployment:~$ openstack hypervisor list<br>
> > > > > > > > <br>
> > > > > > > > (kolla-openstack) stack@deployment:~$<br>
> > > > > > > > <br>
> > > > > > > > <br>
> > > > > > > > Any idea on how to resolve this ?<br>
> > > > > > > that usually means that somehthing prevented the comptue agent form<br>
> > > > > > > strating properly<br>
> > > > > > > <br>
> > > > > > > for example incorrect ceph keyrings there are several other case<br>
> > > > > but you<br>
> > > > > > > mentioned you are<br>
> > > > > > > using ceph.<br>
> > > > > > > <br>
> > > > > > > if this is hte case you should see error in the compute agent log.<br>
> > > > > > > <br>
> > > > > > > > <br>
> > > > > > > > Regards<br>
> > > > > > > > <br>
> > > > > > > > Tony Karera<br>
> > > > > > > <br>
> > > > > > > <br>
> > > > > > > <br>
> > > > > > > <br>
> > > > > <br>
> > > > > <br>
> > > > > <br>
<br>
<br>
</blockquote></div>
</blockquote></div>
</blockquote></div>