Hello Team, I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance. It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors. Any idea on how to resolve this ? Regards Tony Karera
Hello Team, I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance. It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors. Any idea on how to resolve this ? Regards Tony Karera
Hello Team, I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance. It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors. kolla-openstack) stack@deployment:~$ openstack hypervisor list (kolla-openstack) stack@deployment:~$ Any idea on how to resolve this ? Regards Tony Karera
Hello Team,
I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance.
It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors.
kolla-openstack) stack@deployment:~$ openstack hypervisor list
(kolla-openstack) stack@deployment:~$
Any idea on how to resolve this ?
On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: that usually means that somehthing prevented the comptue agent form strating properly for example incorrect ceph keyrings there are several other case but you mentioned you are using ceph. if this is hte case you should see error in the compute agent log.
Regards
Tony Karera
It could also be a compute cell discovery issue maybe? Do you see anything under "openstack compute service list"? On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <smooney@redhat.com> wrote:
Hello Team,
I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance.
It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors.
kolla-openstack) stack@deployment:~$ openstack hypervisor list
(kolla-openstack) stack@deployment:~$
Any idea on how to resolve this ?
On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: that usually means that somehthing prevented the comptue agent form strating properly
for example incorrect ceph keyrings there are several other case but you mentioned you are using ceph.
if this is hte case you should see error in the compute agent log.
Regards
Tony Karera
On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote:
It could also be a compute cell discovery issue maybe? no they shoudl still show up in the hypervior list api
Do you see anything under "openstack compute service list"? if they show up in the service list but not they hyperiors api it means that the comptue service started and registered its service entry but something broke it before it could create a compute node recored in the db.
with ceph the case i have hit this most often is when the keyright used by nova to get the avaiable capastiy of the ceph cluster is wrong whihc prevent the resoucetack and compute manager form actully creating the compute node record. it can happen for other reason too but best place to start is check if there is an error in the nova compute agent log and go from there.
On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <smooney@redhat.com> wrote:
Hello Team,
I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance.
It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors.
kolla-openstack) stack@deployment:~$ openstack hypervisor list
(kolla-openstack) stack@deployment:~$
Any idea on how to resolve this ?
On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: that usually means that somehthing prevented the comptue agent form strating properly
for example incorrect ceph keyrings there are several other case but you mentioned you are using ceph.
if this is hte case you should see error in the compute agent log.
Regards
Tony Karera
Hello Guys, Thanks a lot. I had actually checked the nova -compute.log on the compute node and they were showing the error I will post at the end about the cinder keyring but I know its correct because its the same I was using on wallaby, I even tried to use another ceph cluster with ofcouse different keyrings but its the same issue. Below is the error r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc05a8b8) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc060698) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbce2f4e020) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n[errno 2] RADOS object not found (error connecting to the cluster)\n' 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of the above exception, another exception occurred: Regards Tony Karera On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <smooney@redhat.com> wrote:
On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote:
It could also be a compute cell discovery issue maybe? no they shoudl still show up in the hypervior list api
Do you see anything under "openstack compute service list"? if they show up in the service list but not they hyperiors api it means that the comptue service started and registered its service entry but something broke it before it could create a compute node recored in the db.
with ceph the case i have hit this most often is when the keyright used by nova to get the avaiable capastiy of the ceph cluster is wrong whihc prevent the resoucetack and compute manager form actully creating the compute node record.
it can happen for other reason too but best place to start is check if there is an error in the nova compute agent log and go from there.
On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <smooney@redhat.com> wrote:
On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote:
Hello Team,
I have deployed Openstack Victoria using Kolla-ansible on Ubuntu
20.04
ceph as the backend storage for Nova, Cinder and Glance.
It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors.
kolla-openstack) stack@deployment:~$ openstack hypervisor list
(kolla-openstack) stack@deployment:~$
Any idea on how to resolve this ?
and that usually means that somehthing prevented the comptue agent form strating properly
for example incorrect ceph keyrings there are several other case but you mentioned you are using ceph.
if this is hte case you should see error in the compute agent log.
Regards
Tony Karera
Just to add on that, compute service is listed, I can create Volumes, I have the same cinder keyring in the /etc/kolla/config/nova directory as I have in the /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring Regards Tony Karera On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <tonykarera@gmail.com> wrote:
Hello Guys,
Thanks a lot.
I had actually checked the nova -compute.log on the compute node and they were showing the error I will post at the end about the cinder keyring but I know its correct because its the same I was using on wallaby, I even tried to use another ceph cluster with ofcouse different keyrings but its the same issue.
Below is the error
r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc05a8b8) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc060698) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbce2f4e020) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n[errno 2] RADOS object not found (error connecting to the cluster)\n' 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of the above exception, another exception occurred: Regards
Tony Karera
On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <smooney@redhat.com> wrote:
On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote:
It could also be a compute cell discovery issue maybe? no they shoudl still show up in the hypervior list api
Do you see anything under "openstack compute service list"? if they show up in the service list but not they hyperiors api it means that the comptue service started and registered its service entry but something broke it before it could create a compute node recored in the db.
with ceph the case i have hit this most often is when the keyright used by nova to get the avaiable capastiy of the ceph cluster is wrong whihc prevent the resoucetack and compute manager form actully creating the compute node record.
it can happen for other reason too but best place to start is check if there is an error in the nova compute agent log and go from there.
On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <smooney@redhat.com>
wrote:
On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote:
Hello Team,
I have deployed Openstack Victoria using Kolla-ansible on Ubuntu
20.04
ceph as the backend storage for Nova, Cinder and Glance.
It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors.
kolla-openstack) stack@deployment:~$ openstack hypervisor list
(kolla-openstack) stack@deployment:~$
Any idea on how to resolve this ?
and that usually means that somehthing prevented the comptue agent form strating properly
for example incorrect ceph keyrings there are several other case but you mentioned you are using ceph.
if this is hte case you should see error in the compute agent log.
Regards
Tony Karera
Hey Guys, Any other idea ? Regards Tony Karera On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <tonykarera@gmail.com> wrote:
Just to add on that,
compute service is listed, I can create Volumes, I have the same cinder keyring in the /etc/kolla/config/nova directory as I have in the /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <tonykarera@gmail.com> wrote:
Hello Guys,
Thanks a lot.
I had actually checked the nova -compute.log on the compute node and they were showing the error I will post at the end about the cinder keyring but I know its correct because its the same I was using on wallaby, I even tried to use another ceph cluster with ofcouse different keyrings but its the same issue.
Below is the error
r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc05a8b8) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc060698) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbce2f4e020) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n[errno 2] RADOS object not found (error connecting to the cluster)\n' 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of the above exception, another exception occurred: Regards
Tony Karera
On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <smooney@redhat.com> wrote:
On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote:
It could also be a compute cell discovery issue maybe? no they shoudl still show up in the hypervior list api
Do you see anything under "openstack compute service list"? if they show up in the service list but not they hyperiors api it means that the comptue service started and registered its service entry but something broke it before it could create a compute node recored in the db.
with ceph the case i have hit this most often is when the keyright used by nova to get the avaiable capastiy of the ceph cluster is wrong whihc prevent the resoucetack and compute manager form actully creating the compute node record.
it can happen for other reason too but best place to start is check if there is an error in the nova compute agent log and go from there.
On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <smooney@redhat.com>
wrote:
On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote:
Hello Team,
I have deployed Openstack Victoria using Kolla-ansible on Ubuntu
20.04
ceph as the backend storage for Nova, Cinder and Glance.
It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors.
kolla-openstack) stack@deployment:~$ openstack hypervisor list
(kolla-openstack) stack@deployment:~$
Any idea on how to resolve this ?
and that usually means that somehthing prevented the comptue agent form strating properly
for example incorrect ceph keyrings there are several other case but you mentioned you are using ceph.
if this is hte case you should see error in the compute agent log.
Regards
Tony Karera
I would investigate that compute error first. Creating volumes means the controllers are doing the action. Starting a VM on a compute means you also need Ceph to works on the compute to mount the rdb target. Working in Wallaby with the error doesn't mean it would 100% work in Victoria. On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com> wrote:
Hey Guys, Any other idea ?
Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <tonykarera@gmail.com> wrote:
Just to add on that,
compute service is listed, I can create Volumes, I have the same cinder keyring in the /etc/kolla/config/nova directory as I have in the /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <tonykarera@gmail.com> wrote:
Hello Guys,
Thanks a lot.
I had actually checked the nova -compute.log on the compute node and they were showing the error I will post at the end about the cinder keyring but I know its correct because its the same I was using on wallaby, I even tried to use another ceph cluster with ofcouse different keyrings but its the same issue.
Below is the error
r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc05a8b8) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc060698) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbce2f4e020) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n[errno 2] RADOS object not found (error connecting to the cluster)\n' 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of the above exception, another exception occurred: Regards
Tony Karera
On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <smooney@redhat.com> wrote:
On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote:
It could also be a compute cell discovery issue maybe? no they shoudl still show up in the hypervior list api
Do you see anything under "openstack compute service list"? if they show up in the service list but not they hyperiors api it means that the comptue service started and registered its service entry but something broke it before it could create a compute node recored in the db.
with ceph the case i have hit this most often is when the keyright used by nova to get the avaiable capastiy of the ceph cluster is wrong whihc prevent the resoucetack and compute manager form actully creating the compute node record.
it can happen for other reason too but best place to start is check if there is an error in the nova compute agent log and go from there.
On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <smooney@redhat.com>
wrote:
On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > Hello Team, > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu
20.04
and > ceph as the backend storage for Nova, Cinder and Glance. > > It finished with no error but it has failed to register any on the Compute > Nodes under Hypervisors. > > kolla-openstack) stack@deployment:~$ openstack hypervisor list > > (kolla-openstack) stack@deployment:~$ > > > Any idea on how to resolve this ? that usually means that somehthing prevented the comptue agent form strating properly
for example incorrect ceph keyrings there are several other case but you mentioned you are using ceph.
if this is hte case you should see error in the compute agent log.
> > Regards > > Tony Karera
On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote:
I would investigate that compute error first. Creating volumes means the controllers are doing the action. Starting a VM on a compute means you also need Ceph to works on the compute to mount the rdb target.
nova as part of its startup process in aintiallying the resouce tracker will try to connect to ceph if you are using the rbd image backend to report how much stroage is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as then that will block the agent from starting up fully and will cause it to be missing the hypervior list. the error seams to indicate that the cinder keyring is not in the nova container that likely means you have not put it in /etc/kolla/config/nova i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/ on the compute node to ensure the cinder keyring is actully copied and has the correct content i have stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf [client.cinder] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring [client.nova] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" blanked out the key wiht *************** after the fact but you should have something similar in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool so i neede to modify my ceph.conf to make that work with kolla stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f [global] fsid = ********************* mon_host = [*********************] [client.glance] rbd default data pool = images-data [client.cinder] rbd default data pool = volumes-data [client.nova] rbd default data pool = vms-data using 2 keyrings/user allows me to set different default data pools for cinder and nova.
Working in Wallaby with the error doesn't mean it would 100% work in Victoria.
On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com> wrote:
Hey Guys, Any other idea ?
Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <tonykarera@gmail.com> wrote:
Just to add on that,
compute service is listed, I can create Volumes, I have the same cinder keyring in the /etc/kolla/config/nova directory as I have in the /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <tonykarera@gmail.com> wrote:
Hello Guys,
Thanks a lot.
I had actually checked the nova -compute.log on the compute node and they were showing the error I will post at the end about the cinder keyring but I know its correct because its the same I was using on wallaby, I even tried to use another ceph cluster with ofcouse different keyrings but its the same issue.
Below is the error
r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc05a8b8) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc060698) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbce2f4e020) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n[errno 2] RADOS object not found (error connecting to the cluster)\n' 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of the above exception, another exception occurred: Regards
Tony Karera
On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <smooney@redhat.com> wrote:
On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote:
It could also be a compute cell discovery issue maybe? no they shoudl still show up in the hypervior list api
Do you see anything under "openstack compute service list"? if they show up in the service list but not they hyperiors api it means that the comptue service started and registered its service entry but something broke it before it could create a compute node recored in the db.
with ceph the case i have hit this most often is when the keyright used by nova to get the avaiable capastiy of the ceph cluster is wrong whihc prevent the resoucetack and compute manager form actully creating the compute node record.
it can happen for other reason too but best place to start is check if there is an error in the nova compute agent log and go from there.
On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <smooney@redhat.com>
wrote:
> On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > Hello Team, > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu
20.04
> and > > ceph as the backend storage for Nova, Cinder and Glance. > > > > It finished with no error but it has failed to register any on the > Compute > > Nodes under Hypervisors. > > > > kolla-openstack) stack@deployment:~$ openstack hypervisor list > > > > (kolla-openstack) stack@deployment:~$ > > > > > > Any idea on how to resolve this ? > that usually means that somehthing prevented the comptue agent form > strating properly > > for example incorrect ceph keyrings there are several other case but you > mentioned you are > using ceph. > > if this is hte case you should see error in the compute agent log. > > > > > Regards > > > > Tony Karera > > > >
Hello Sean, Below are the output on the compute node and deployment root@compute1:/etc/kolla/nova-compute# ls ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf (kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf And I can confirm that the content is the same. Regards Tony Karera On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> wrote:
On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote:
I would investigate that compute error first. Creating volumes means the controllers are doing the action. Starting a VM on a compute means you also need Ceph to works on the compute to mount the rdb target.
nova as part of its startup process in aintiallying the resouce tracker will try to connect to ceph if you are using the rbd image backend to report how much stroage is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as then that will block the agent from starting up fully and will cause it to be missing the hypervior list.
the error seams to indicate that the cinder keyring is not in the nova container that likely means you have not put it in /etc/kolla/config/nova i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/ on the compute node to ensure the cinder keyring is actully copied and has the correct content
i have stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
[client.cinder] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring [client.nova] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images"
blanked out the key wiht *************** after the fact but you should have something similar
in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool so i neede to modify my ceph.conf to make that work with kolla
stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f [global] fsid = ********************* mon_host = [*********************]
[client.glance] rbd default data pool = images-data
[client.cinder] rbd default data pool = volumes-data
[client.nova] rbd default data pool = vms-data
using 2 keyrings/user allows me to set different default data pools for cinder and nova.
Working in Wallaby with the error doesn't mean it would 100% work in Victoria.
On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com>
wrote:
Hey Guys, Any other idea ?
Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <tonykarera@gmail.com>
Just to add on that,
compute service is listed, I can create Volumes, I have the same
cinder
keyring in the /etc/kolla/config/nova directory as I have in the /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <tonykarera@gmail.com> wrote:
Hello Guys,
Thanks a lot.
I had actually checked the nova -compute.log on the compute node and they were showing the error I will post at the end about the cinder keyring but I know its correct because its the same I was using on wallaby, I even tried to use another ceph cluster with ofcouse different keyrings but its the same issue.
Below is the error
r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to find a keyring on
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
(2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc05a8b8) no keyring found at
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
(2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc060698) no keyring found at
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
(2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbce2f4e020) no keyring found at
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
disabling cephx\n[errno 2] RADOS object not found (error connecting to the cluster)\n' 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of the above exception, another exception occurred: Regards
Tony Karera
On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <smooney@redhat.com> wrote:
On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > It could also be a compute cell discovery issue maybe? no they shoudl still show up in the hypervior list api > > Do you see anything under "openstack compute service list"? if they show up in the service list but not they hyperiors api it means that the comptue service started and registered its service entry but something broke it before it could create a compute node recored in the db.
with ceph the case i have hit this most often is when the keyright used by nova to get the avaiable capastiy of the ceph cluster is wrong whihc
the resoucetack and compute manager form actully creating the compute node record.
it can happen for other reason too but best place to start is check if there is an error in the nova compute agent log and go from
> > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < smooney@redhat.com> wrote: > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > Hello Team, > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 > > and > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > It finished with no error but it has failed to register any on the > > Compute > > > Nodes under Hypervisors. > > > > > > kolla-openstack) stack@deployment:~$ openstack hypervisor
wrote: prevent there. list
> > > > > > (kolla-openstack) stack@deployment:~$ > > > > > > > > > Any idea on how to resolve this ? > > that usually means that somehthing prevented the comptue agent form > > strating properly > > > > for example incorrect ceph keyrings there are several other case but you > > mentioned you are > > using ceph. > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > Regards > > > > > > Tony Karera > > > > > > > >
I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container ( etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Sean,
Below are the output on the compute node and deployment
root@compute1:/etc/kolla/nova-compute# ls ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
(kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf
And I can confirm that the content is the same.
Regards
Tony Karera
On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> wrote:
On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote:
I would investigate that compute error first. Creating volumes means the controllers are doing the action. Starting a VM on a compute means you also need Ceph to works on the compute to mount the rdb target.
nova as part of its startup process in aintiallying the resouce tracker will try to connect to ceph if you are using the rbd image backend to report how much stroage is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as then that will block the agent from starting up fully and will cause it to be missing the hypervior list.
the error seams to indicate that the cinder keyring is not in the nova container that likely means you have not put it in /etc/kolla/config/nova i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/ on the compute node to ensure the cinder keyring is actully copied and has the correct content
i have stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
[client.cinder] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring [client.nova] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images"
blanked out the key wiht *************** after the fact but you should have something similar
in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool so i neede to modify my ceph.conf to make that work with kolla
stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f [global] fsid = ********************* mon_host = [*********************]
[client.glance] rbd default data pool = images-data
[client.cinder] rbd default data pool = volumes-data
[client.nova] rbd default data pool = vms-data
using 2 keyrings/user allows me to set different default data pools for cinder and nova.
Working in Wallaby with the error doesn't mean it would 100% work in Victoria.
On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com>
wrote:
Hey Guys, Any other idea ?
Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <tonykarera@gmail.com>
Just to add on that,
compute service is listed, I can create Volumes, I have the same
cinder
keyring in the /etc/kolla/config/nova directory as I have in the /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <tonykarera@gmail.com> wrote:
Hello Guys,
Thanks a lot.
I had actually checked the nova -compute.log on the compute node and they were showing the error I will post at the end about the cinder keyring but I know its correct because its the same I was using on wallaby, I even tried to use another ceph cluster with ofcouse different keyrings but its the same issue.
Below is the error
r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to find a keyring on
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
(2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc05a8b8) no keyring found at
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
(2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbcdc060698) no keyring found at
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable to find a keyring on
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
(2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 AuthRegistry(0x7fbce2f4e020) no keyring found at
/etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
disabling cephx\n[errno 2] RADOS object not found (error connecting to the cluster)\n' 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of the above exception, another exception occurred: Regards
Tony Karera
On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <smooney@redhat.com> wrote:
> On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > It could also be a compute cell discovery issue maybe? > no they shoudl still show up in the hypervior list api > > > > Do you see anything under "openstack compute service list"? > if they show up in the service list but not they hyperiors api it > means that the comptue service started and registered its service entry > but > something broke it before it could create a compute node recored in the > db. > > with ceph the case i have hit this most often is when the keyright used > by nova to > get the avaiable capastiy of the ceph cluster is wrong whihc
> the resoucetack and compute manager > form actully creating the compute node record. > > > it can happen for other reason too but best place to start is check if > there is an error in the nova compute agent log and go from
wrote: prevent there.
> > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < smooney@redhat.com> > wrote: > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > Hello Team, > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu > 20.04 > > > and > > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > > > It finished with no error but it has failed to register any on the > > > Compute > > > > Nodes under Hypervisors. > > > > > > > > kolla-openstack) stack@deployment:~$ openstack hypervisor list > > > > > > > > (kolla-openstack) stack@deployment:~$ > > > > > > > > > > > > Any idea on how to resolve this ? > > > that usually means that somehthing prevented the comptue agent form > > > strating properly > > > > > > for example incorrect ceph keyrings there are several other case > but you > > > mentioned you are > > > using ceph. > > > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > > > > Regards > > > > > > > > Tony Karera > > > > > > > > > > > > > > >
Hello Laurent, It turns out I only have one keyring in the container. root@compute1:/home/stack# docker exec -it nova_compute bash (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ (nova-compute)[nova@compute1 ceph]$ ls ceph.client.nova.keyring ceph.conf rbdmap Regards Tony Karera On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container ( etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin)
On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Sean,
Below are the output on the compute node and deployment
root@compute1:/etc/kolla/nova-compute# ls ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
(kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf
And I can confirm that the content is the same.
Regards
Tony Karera
On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> wrote:
I would investigate that compute error first. Creating volumes means
On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: the
controllers are doing the action. Starting a VM on a compute means you also need Ceph to works on the compute to mount the rdb target.
nova as part of its startup process in aintiallying the resouce tracker will try to connect to ceph if you are using the rbd image backend to report how much stroage is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as then that will block the agent from starting up fully and will cause it to be missing the hypervior list.
the error seams to indicate that the cinder keyring is not in the nova container that likely means you have not put it in /etc/kolla/config/nova i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/ on the compute node to ensure the cinder keyring is actully copied and has the correct content
i have stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
[client.cinder] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring [client.nova] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images"
blanked out the key wiht *************** after the fact but you should have something similar
in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool so i neede to modify my ceph.conf to make that work with kolla
stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f [global] fsid = ********************* mon_host = [*********************]
[client.glance] rbd default data pool = images-data
[client.cinder] rbd default data pool = volumes-data
[client.nova] rbd default data pool = vms-data
using 2 keyrings/user allows me to set different default data pools for cinder and nova.
Working in Wallaby with the error doesn't mean it would 100% work in Victoria.
On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com>
wrote:
Hey Guys, Any other idea ?
Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <tonykarera@gmail.com>
Just to add on that,
compute service is listed, I can create Volumes, I have the same
cinder
keyring in the /etc/kolla/config/nova directory as I have in the /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <tonykarera@gmail.com> wrote:
> Hello Guys, > > Thanks a lot. > > I had actually checked the nova -compute.log on the compute node and > they were showing the error I will post at the end about the cinder keyring > but I know its correct because its the same I was using on wallaby, I even > tried to use another ceph cluster with ofcouse different keyrings but its > the same issue. > > Below is the error > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to > find a keyring on > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 > AuthRegistry(0x7fbcdc05a8b8) no keyring found at > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > to find a keyring on > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > AuthRegistry(0x7fbcdc060698) no keyring found at > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > to find a keyring on > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > AuthRegistry(0x7fbce2f4e020) no keyring found at > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > disabling cephx\n[errno 2] RADOS object not found (error connecting to the > cluster)\n' > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of > the above exception, another exception occurred: > Regards > > Tony Karera > > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <smooney@redhat.com> wrote: > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > > It could also be a compute cell discovery issue maybe? > > no they shoudl still show up in the hypervior list api > > > > > > Do you see anything under "openstack compute service list"? > > if they show up in the service list but not they hyperiors api it > > means that the comptue service started and registered its service entry > > but > > something broke it before it could create a compute node recored in the > > db. > > > > with ceph the case i have hit this most often is when the keyright used > > by nova to > > get the avaiable capastiy of the ceph cluster is wrong whihc
> > the resoucetack and compute manager > > form actully creating the compute node record. > > > > > > it can happen for other reason too but best place to start is check if > > there is an error in the nova compute agent log and go from
wrote: prevent there.
> > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < smooney@redhat.com> > > wrote: > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > > Hello Team, > > > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu > > 20.04 > > > > and > > > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > > > > > It finished with no error but it has failed to register any on the > > > > Compute > > > > > Nodes under Hypervisors. > > > > > > > > > > kolla-openstack) stack@deployment:~$ openstack hypervisor list > > > > > > > > > > (kolla-openstack) stack@deployment:~$ > > > > > > > > > > > > > > > Any idea on how to resolve this ? > > > > that usually means that somehthing prevented the comptue agent form > > > > strating properly > > > > > > > > for example incorrect ceph keyrings there are several other case > > but you > > > > mentioned you are > > > > using ceph. > > > > > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > > > > > > > Regards > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > >
Hello Team, I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying. I don't understand why there is a certain issue now. Regards Tony Karera On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Laurent,
It turns out I only have one keyring in the container.
root@compute1:/home/stack# docker exec -it nova_compute bash (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ (nova-compute)[nova@compute1 ceph]$ ls ceph.client.nova.keyring ceph.conf rbdmap
Regards
Tony Karera
On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container ( etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin)
On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Sean,
Below are the output on the compute node and deployment
root@compute1:/etc/kolla/nova-compute# ls ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
(kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf
And I can confirm that the content is the same.
Regards
Tony Karera
On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> wrote:
I would investigate that compute error first. Creating volumes means
On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: the
controllers are doing the action. Starting a VM on a compute means you also need Ceph to works on the compute to mount the rdb target.
nova as part of its startup process in aintiallying the resouce tracker will try to connect to ceph if you are using the rbd image backend to report how much stroage is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as then that will block the agent from starting up fully and will cause it to be missing the hypervior list.
the error seams to indicate that the cinder keyring is not in the nova container that likely means you have not put it in /etc/kolla/config/nova i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/ on the compute node to ensure the cinder keyring is actully copied and has the correct content
i have stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
[client.cinder] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring [client.nova] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images"
blanked out the key wiht *************** after the fact but you should have something similar
in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool so i neede to modify my ceph.conf to make that work with kolla
stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f [global] fsid = ********************* mon_host = [*********************]
[client.glance] rbd default data pool = images-data
[client.cinder] rbd default data pool = volumes-data
[client.nova] rbd default data pool = vms-data
using 2 keyrings/user allows me to set different default data pools for cinder and nova.
Working in Wallaby with the error doesn't mean it would 100% work in Victoria.
On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com>
wrote:
Hey Guys, Any other idea ?
Regards
Tony Karera
On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <tonykarera@gmail.com>
> Just to add on that, > > compute service is listed, I can create Volumes, I have the same
cinder
> keyring in the /etc/kolla/config/nova directory as I have in the > /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring > Regards > > Tony Karera > > > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <tonykarera@gmail.com> wrote: > > > Hello Guys, > > > > Thanks a lot. > > > > I had actually checked the nova -compute.log on the compute node and > > they were showing the error I will post at the end about the cinder keyring > > but I know its correct because its the same I was using on wallaby, I even > > tried to use another ceph cluster with ofcouse different keyrings but its > > the same issue. > > > > Below is the error > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to > > find a keyring on > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > to find a keyring on > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > AuthRegistry(0x7fbcdc060698) no keyring found at > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > to find a keyring on > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > AuthRegistry(0x7fbce2f4e020) no keyring found at > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > disabling cephx\n[errno 2] RADOS object not found (error connecting to the > > cluster)\n' > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of > > the above exception, another exception occurred: > > Regards > > > > Tony Karera > > > > > > > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <smooney@redhat.com> wrote: > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > > > It could also be a compute cell discovery issue maybe? > > > no they shoudl still show up in the hypervior list api > > > > > > > > Do you see anything under "openstack compute service list"? > > > if they show up in the service list but not they hyperiors api it > > > means that the comptue service started and registered its service entry > > > but > > > something broke it before it could create a compute node recored in the > > > db. > > > > > > with ceph the case i have hit this most often is when the keyright used > > > by nova to > > > get the avaiable capastiy of the ceph cluster is wrong whihc
> > > the resoucetack and compute manager > > > form actully creating the compute node record. > > > > > > > > > it can happen for other reason too but best place to start is check if > > > there is an error in the nova compute agent log and go from
wrote: prevent there.
> > > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < smooney@redhat.com> > > > wrote: > > > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > > > Hello Team, > > > > > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu > > > 20.04 > > > > > and > > > > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > > > > > > > It finished with no error but it has failed to register any on the > > > > > Compute > > > > > > Nodes under Hypervisors. > > > > > > > > > > > > kolla-openstack) stack@deployment:~$ openstack hypervisor list > > > > > > > > > > > > (kolla-openstack) stack@deployment:~$ > > > > > > > > > > > > > > > > > > Any idea on how to resolve this ? > > > > > that usually means that somehthing prevented the comptue agent form > > > > > strating properly > > > > > > > > > > for example incorrect ceph keyrings there are several other case > > > but you > > > > > mentioned you are > > > > > using ceph. > > > > > > > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
I would really appreciate any support on this On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com> wrote:
Hello Team,
I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying.
I don't understand why there is a certain issue now. Regards
Tony Karera
On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Laurent,
It turns out I only have one keyring in the container.
root@compute1:/home/stack# docker exec -it nova_compute bash (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ (nova-compute)[nova@compute1 ceph]$ ls ceph.client.nova.keyring ceph.conf rbdmap
Regards
Tony Karera
On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container ( etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin)
On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Sean,
Below are the output on the compute node and deployment
root@compute1:/etc/kolla/nova-compute# ls ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
(kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf
And I can confirm that the content is the same.
Regards
Tony Karera
On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> wrote:
I would investigate that compute error first. Creating volumes means
On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: the
controllers are doing the action. Starting a VM on a compute means you also need Ceph to works on the compute to mount the rdb target.
nova as part of its startup process in aintiallying the resouce tracker will try to connect to ceph if you are using the rbd image backend to report how much stroage is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as then that will block the agent from starting up fully and will cause it to be missing the hypervior list.
the error seams to indicate that the cinder keyring is not in the nova container that likely means you have not put it in /etc/kolla/config/nova i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/ on the compute node to ensure the cinder keyring is actully copied and has the correct content
i have stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
[client.cinder] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring [client.nova] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images"
blanked out the key wiht *************** after the fact but you should have something similar
in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool so i neede to modify my ceph.conf to make that work with kolla
stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f [global] fsid = ********************* mon_host = [*********************]
[client.glance] rbd default data pool = images-data
[client.cinder] rbd default data pool = volumes-data
[client.nova] rbd default data pool = vms-data
using 2 keyrings/user allows me to set different default data pools for cinder and nova.
Working in Wallaby with the error doesn't mean it would 100% work in Victoria.
On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com>
wrote:
> Hey Guys, Any other idea ? > > Regards > > Tony Karera > > > > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <tonykarera@gmail.com>
> > > Just to add on that, > > > > compute service is listed, I can create Volumes, I have the same cinder > > keyring in the /etc/kolla/config/nova directory as I have in the > > /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring > > Regards > > > > Tony Karera > > > > > > > > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < tonykarera@gmail.com> wrote: > > > > > Hello Guys, > > > > > > Thanks a lot. > > > > > > I had actually checked the nova -compute.log on the compute node and > > > they were showing the error I will post at the end about the cinder keyring > > > but I know its correct because its the same I was using on wallaby, I even > > > tried to use another ceph cluster with ofcouse different keyrings but its > > > the same issue. > > > > > > Below is the error > > > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to > > > find a keyring on > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > > to find a keyring on > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > > AuthRegistry(0x7fbcdc060698) no keyring found at > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > > to find a keyring on > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > > AuthRegistry(0x7fbce2f4e020) no keyring found at > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > disabling cephx\n[errno 2] RADOS object not found (error connecting to the > > > cluster)\n' > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of > > > the above exception, another exception occurred: > > > Regards > > > > > > Tony Karera > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < smooney@redhat.com> wrote: > > > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > > > > It could also be a compute cell discovery issue maybe? > > > > no they shoudl still show up in the hypervior list api > > > > > > > > > > Do you see anything under "openstack compute service list"? > > > > if they show up in the service list but not they hyperiors api it > > > > means that the comptue service started and registered its service entry > > > > but > > > > something broke it before it could create a compute node recored in the > > > > db. > > > > > > > > with ceph the case i have hit this most often is when the keyright used > > > > by nova to > > > > get the avaiable capastiy of the ceph cluster is wrong whihc
> > > > the resoucetack and compute manager > > > > form actully creating the compute node record. > > > > > > > > > > > > it can happen for other reason too but best place to start is check if > > > > there is an error in the nova compute agent log and go from
wrote: prevent there.
> > > > > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < smooney@redhat.com> > > > > wrote: > > > > > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > > > > Hello Team, > > > > > > > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu > > > > 20.04 > > > > > > and > > > > > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > > > > > > > > > It finished with no error but it has failed to register any on the > > > > > > Compute > > > > > > > Nodes under Hypervisors. > > > > > > > > > > > > > > kolla-openstack) stack@deployment:~$ openstack hypervisor list > > > > > > > > > > > > > > (kolla-openstack) stack@deployment:~$ > > > > > > > > > > > > > > > > > > > > > Any idea on how to resolve this ? > > > > > > that usually means that somehthing prevented the comptue agent form > > > > > > strating properly > > > > > > > > > > > > for example incorrect ceph keyrings there are several other case > > > > but you > > > > > > mentioned you are > > > > > > using ceph. > > > > > > > > > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Keep in mind that this is asynchronous communication with a community of volunteers. Lots of folks are already off today or will be soon. Sometimes it could be a couple of days before you get a response. I hope you're able to solve your issue. Jimmy On Sep 24 2021, at 4:10 pm, Karera Tony <tonykarera@gmail.com> wrote:
I would really appreciate any support on this
On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com (mailto:tonykarera@gmail.com)> wrote:
Hello Team,
I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying.
I don't understand why there is a certain issue now. Regards
Tony Karera
On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com (mailto:tonykarera@gmail.com)> wrote:
Hello Laurent,
It turns out I only have one keyring in the container.
root@compute1:/home/stack# docker exec -it nova_compute bash (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ (nova-compute)[nova@compute1 ceph]$ ls ceph.client.nova.keyring ceph.conf rbdmap
Regards
Tony Karera
On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont <laurentfdumont@gmail.com (mailto:laurentfdumont@gmail.com)> wrote:
I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container (etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin)
On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com (mailto:tonykarera@gmail.com)> wrote:
Hello Sean,
Below are the output on the compute node and deployment
root@compute1:/etc/kolla/nova-compute# ls ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
(kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf
And I can confirm that the content is the same.
Regards
Tony Karera
On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com (mailto:smooney@redhat.com)> wrote:
On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: > I would investigate that compute error first. Creating volumes means the > controllers are doing the action. Starting a VM on a compute means you also > need Ceph to works on the compute to mount the rdb target.
nova as part of its startup process in aintiallying the resouce tracker will try to connect to ceph if you are using the rbd image backend to report how much stroage is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as then that will block the agent from starting up fully and will cause it to be missing the hypervior list.
the error seams to indicate that the cinder keyring is not in the nova container that likely means you have not put it in /etc/kolla/config/nova i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/ on the compute node to ensure the cinder keyring is actully copied and has the correct content
i have stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
[client.cinder] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring [client.nova] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images"
blanked out the key wiht *************** after the fact but you should have something similar
in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool so i neede to modify my ceph.conf to make that work with kolla
stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f [global] fsid = ********************* mon_host = [*********************]
[client.glance] rbd default data pool = images-data
[client.cinder] rbd default data pool = volumes-data
[client.nova] rbd default data pool = vms-data
using 2 keyrings/user allows me to set different default data pools for cinder and nova. > > Working in Wallaby with the error doesn't mean it would 100% work in > Victoria. > > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com (mailto:tonykarera@gmail.com)> wrote: > > > Hey Guys, Any other idea ? > > > > Regards > > > > Tony Karera > > > > > > > > > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <tonykarera@gmail.com (mailto:tonykarera@gmail.com)> wrote: > > > > > Just to add on that, > > > > > > compute service is listed, I can create Volumes, I have the same cinder > > > keyring in the /etc/kolla/config/nova directory as I have in the > > > /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring > > > Regards > > > > > > Tony Karera > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <tonykarera@gmail.com (mailto:tonykarera@gmail.com)> wrote: > > > > > > > Hello Guys, > > > > > > > > Thanks a lot. > > > > > > > > I had actually checked the nova -compute.log on the compute node and > > > > they were showing the error I will post at the end about the cinder keyring > > > > but I know its correct because its the same I was using on wallaby, I even > > > > tried to use another ceph cluster with ofcouse different keyrings but its > > > > the same issue. > > > > > > > > Below is the error > > > > > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to > > > > find a keyring on > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > > > to find a keyring on > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > > > AuthRegistry(0x7fbcdc060698) no keyring found at > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > > > to find a keyring on > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > disabling cephx\n[errno 2] RADOS object not found (error connecting to the > > > > cluster)\n' > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of > > > > the above exception, another exception occurred: > > > > Regards > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <smooney@redhat.com (mailto:smooney@redhat.com)> wrote: > > > > > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > > > > > It could also be a compute cell discovery issue maybe? > > > > > no they shoudl still show up in the hypervior list api > > > > > > > > > > > > Do you see anything under "openstack compute service list"? > > > > > if they show up in the service list but not they hyperiors api it > > > > > means that the comptue service started and registered its service entry > > > > > but > > > > > something broke it before it could create a compute node recored in the > > > > > db. > > > > > > > > > > with ceph the case i have hit this most often is when the keyright used > > > > > by nova to > > > > > get the avaiable capastiy of the ceph cluster is wrong whihc prevent > > > > > the resoucetack and compute manager > > > > > form actully creating the compute node record. > > > > > > > > > > > > > > > it can happen for other reason too but best place to start is check if > > > > > there is an error in the nova compute agent log and go from there. > > > > > > > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <smooney@redhat.com (mailto:smooney@redhat.com)> > > > > > wrote: > > > > > > > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > > > > > Hello Team, > > > > > > > > > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu > > > > > 20.04 > > > > > > > and > > > > > > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > > > > > > > > > > > It finished with no error but it has failed to register any on the > > > > > > > Compute > > > > > > > > Nodes under Hypervisors. > > > > > > > > > > > > > > > > kolla-openstack) stack@deployment:~$ openstack hypervisor list > > > > > > > > > > > > > > > > (kolla-openstack) stack@deployment:~$ > > > > > > > > > > > > > > > > > > > > > > > > Any idea on how to resolve this ? > > > > > > > that usually means that somehthing prevented the comptue agent form > > > > > > > strating properly > > > > > > > > > > > > > > for example incorrect ceph keyrings there are several other case > > > > > but you > > > > > > > mentioned you are > > > > > > > using ceph. > > > > > > > > > > > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
I know that there is some Kolla folks around but keep in mind that this is a volunteer based list :) I think you might get a bit more one to one help on IRC in their kolla channel. On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera@gmail.com> wrote:
I would really appreciate any support on this
On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com> wrote:
Hello Team,
I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying.
I don't understand why there is a certain issue now. Regards
Tony Karera
On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Laurent,
It turns out I only have one keyring in the container.
root@compute1:/home/stack# docker exec -it nova_compute bash (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ (nova-compute)[nova@compute1 ceph]$ ls ceph.client.nova.keyring ceph.conf rbdmap
Regards
Tony Karera
On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container ( etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin)
On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Sean,
Below are the output on the compute node and deployment
root@compute1:/etc/kolla/nova-compute# ls ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
(kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf
And I can confirm that the content is the same.
Regards
Tony Karera
On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> wrote:
On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: > I would investigate that compute error first. Creating volumes means the > controllers are doing the action. Starting a VM on a compute means you also > need Ceph to works on the compute to mount the rdb target.
nova as part of its startup process in aintiallying the resouce tracker will try to connect to ceph if you are using the rbd image backend to report how much stroage is avaiable. if the keyring does not work on the vms pool as the user nova is connecting as then that will block the agent from starting up fully and will cause it to be missing the hypervior list.
the error seams to indicate that the cinder keyring is not in the nova container that likely means you have not put it in /etc/kolla/config/nova i woudl check /etc/kolla/config/nova on the deployment host and sudo ls /etc/kolla/nova-compute/ on the compute node to ensure the cinder keyring is actully copied and has the correct content
i have stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
[client.cinder] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images" stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.client.nova.keyring [client.nova] key = ********************************* caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" caps mon = "profile rbd" caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images"
blanked out the key wiht *************** after the fact but you should have something similar
in my case i decied to use a seperate key for nova rbd backend because i was also using EC poosl with a seperate data and metadata pool so i neede to modify my ceph.conf to make that work with kolla
stack@cloud:/opt/repos/devstack$ sudo cat /etc/kolla/nova-compute/ceph.conf # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f [global] fsid = ********************* mon_host = [*********************]
[client.glance] rbd default data pool = images-data
[client.cinder] rbd default data pool = volumes-data
[client.nova] rbd default data pool = vms-data
using 2 keyrings/user allows me to set different default data pools for cinder and nova.
> > Working in Wallaby with the error doesn't mean it would 100% work in > Victoria. > > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com> wrote: > > > Hey Guys, Any other idea ? > > > > Regards > > > > Tony Karera > > > > > > > > > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <tonykarera@gmail.com> wrote: > > > > > Just to add on that, > > > > > > compute service is listed, I can create Volumes, I have the same cinder > > > keyring in the /etc/kolla/config/nova directory as I have in the > > > /etc/kolla/config/cinder/cinder-volume directory along with the nova keyring > > > Regards > > > > > > Tony Karera > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < tonykarera@gmail.com> wrote: > > > > > > > Hello Guys, > > > > > > > > Thanks a lot. > > > > > > > > I had actually checked the nova -compute.log on the compute node and > > > > they were showing the error I will post at the end about the cinder keyring > > > > but I know its correct because its the same I was using on wallaby, I even > > > > tried to use another ceph cluster with ofcouse different keyrings but its > > > > the same issue. > > > > > > > > Below is the error > > > > > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 auth: unable to > > > > find a keyring on > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > > > to find a keyring on > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > > > AuthRegistry(0x7fbcdc060698) no keyring found at > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 auth: unable > > > > to find a keyring on > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > disabling cephx\n[errno 2] RADOS object not found (error connecting to the > > > > cluster)\n' > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During handling of > > > > the above exception, another exception occurred: > > > > Regards > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < smooney@redhat.com> wrote: > > > > > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > > > > > It could also be a compute cell discovery issue maybe? > > > > > no they shoudl still show up in the hypervior list api > > > > > > > > > > > > Do you see anything under "openstack compute service list"? > > > > > if they show up in the service list but not they hyperiors api it > > > > > means that the comptue service started and registered its service entry > > > > > but > > > > > something broke it before it could create a compute node recored in the > > > > > db. > > > > > > > > > > with ceph the case i have hit this most often is when the keyright used > > > > > by nova to > > > > > get the avaiable capastiy of the ceph cluster is wrong whihc prevent > > > > > the resoucetack and compute manager > > > > > form actully creating the compute node record. > > > > > > > > > > > > > > > it can happen for other reason too but best place to start is check if > > > > > there is an error in the nova compute agent log and go from there. > > > > > > > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < smooney@redhat.com> > > > > > wrote: > > > > > > > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > > > > > Hello Team, > > > > > > > > > > > > > > > > I have deployed Openstack Victoria using Kolla-ansible on Ubuntu > > > > > 20.04 > > > > > > > and > > > > > > > > ceph as the backend storage for Nova, Cinder and Glance. > > > > > > > > > > > > > > > > It finished with no error but it has failed to register any on the > > > > > > > Compute > > > > > > > > Nodes under Hypervisors. > > > > > > > > > > > > > > > > kolla-openstack) stack@deployment:~$ openstack hypervisor list > > > > > > > > > > > > > > > > (kolla-openstack) stack@deployment:~$ > > > > > > > > > > > > > > > > > > > > > > > > Any idea on how to resolve this ? > > > > > > > that usually means that somehthing prevented the comptue agent form > > > > > > > strating properly > > > > > > > > > > > > > > for example incorrect ceph keyrings there are several other case > > > > > but you > > > > > > > mentioned you are > > > > > > > using ceph. > > > > > > > > > > > > > > if this is hte case you should see error in the compute agent log. > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Hello Team, I even tried to manually put the ceph.client.cinder.keyring in the nova_compute container but the issue persisted. I also tried reinstalling Openstack on another Environment but I still have the same issue. Anyone with any idea on how to proceed ? Regards Tony Karera On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
I know that there is some Kolla folks around but keep in mind that this is a volunteer based list :)
I think you might get a bit more one to one help on IRC in their kolla channel.
On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera@gmail.com> wrote:
I would really appreciate any support on this
On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com> wrote:
Hello Team,
I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying.
I don't understand why there is a certain issue now. Regards
Tony Karera
On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Laurent,
It turns out I only have one keyring in the container.
root@compute1:/home/stack# docker exec -it nova_compute bash (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ (nova-compute)[nova@compute1 ceph]$ ls ceph.client.nova.keyring ceph.conf rbdmap
Regards
Tony Karera
On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < laurentfdumont@gmail.com> wrote:
I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container ( etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin)
On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Sean,
Below are the output on the compute node and deployment
root@compute1:/etc/kolla/nova-compute# ls ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf config.json nova.conf
(kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf
And I can confirm that the content is the same.
Regards
Tony Karera
On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> wrote:
> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: > > I would investigate that compute error first. Creating volumes > means the > > controllers are doing the action. Starting a VM on a compute means > you also > > need Ceph to works on the compute to mount the rdb target. > > nova as part of its startup process in aintiallying the resouce > tracker will > try to connect to ceph if you are using the rbd image backend to > report how much stroage > is avaiable. if the keyring does not work on the vms pool as the > user nova is connecting as > then that will block the agent from starting up fully and will cause > it to be missing the hypervior list. > > the error seams to indicate that the cinder keyring is not in the > nova container > that likely means you have not put it in /etc/kolla/config/nova > i woudl check /etc/kolla/config/nova on the deployment host and sudo > ls /etc/kolla/nova-compute/ > on the compute node to ensure the cinder keyring is actully copied > and has the correct content > > i have > stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ > ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf > config.json nova.conf > > > [client.cinder] > key = ********************************* > caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" > caps mon = "profile rbd" > caps osd = "profile rbd pool=volumes, profile rbd pool=vms, > profile rbd pool=images" > stack@cloud:/opt/repos/devstack$ sudo cat > /etc/kolla/nova-compute/ceph.client.nova.keyring > [client.nova] > key = ********************************* > caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" > caps mon = "profile rbd" > caps osd = "profile rbd pool=volumes, profile rbd pool=vms, > profile rbd pool=images" > > blanked out the key wiht *************** after the fact but you > should have something similar > > > in my case i decied to use a seperate key for nova rbd backend > because i was also using EC poosl with a seperate data and metadata pool > so i neede to modify my ceph.conf to make that work with kolla > > stack@cloud:/opt/repos/devstack$ sudo cat > /etc/kolla/nova-compute/ceph.conf > # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f > [global] > fsid = ********************* > mon_host = [*********************] > > [client.glance] > rbd default data pool = images-data > > [client.cinder] > rbd default data pool = volumes-data > > [client.nova] > rbd default data pool = vms-data > > using 2 keyrings/user allows me to set different default data pools > for cinder and nova. > > > > > Working in Wallaby with the error doesn't mean it would 100% work > in > > Victoria. > > > > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com> > wrote: > > > > > Hey Guys, Any other idea ? > > > > > > Regards > > > > > > Tony Karera > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < > tonykarera@gmail.com> wrote: > > > > > > > Just to add on that, > > > > > > > > compute service is listed, I can create Volumes, I have the > same cinder > > > > keyring in the /etc/kolla/config/nova directory as I have in > the > > > > /etc/kolla/config/cinder/cinder-volume directory along with > the nova keyring > > > > Regards > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < > tonykarera@gmail.com> wrote: > > > > > > > > > Hello Guys, > > > > > > > > > > Thanks a lot. > > > > > > > > > > I had actually checked the nova -compute.log on the compute > node and > > > > > they were showing the error I will post at the end about the > cinder keyring > > > > > but I know its correct because its the same I was using on > wallaby, I even > > > > > tried to use another ceph cluster with ofcouse different > keyrings but its > > > > > the same issue. > > > > > > > > > > Below is the error > > > > > > > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 > auth: unable to > > > > > find a keyring on > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 > 7fbce2f4f700 -1 > > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 > -1 auth: unable > > > > > to find a keyring on > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 > 7fbce2f4f700 -1 > > > > > AuthRegistry(0x7fbcdc060698) no keyring found at > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 > -1 auth: unable > > > > > to find a keyring on > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 > 7fbce2f4f700 -1 > > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at > > > > > > /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > > > > disabling cephx\n[errno 2] RADOS object not found (error > connecting to the > > > > > cluster)\n' > > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager > > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During > handling of > > > > > the above exception, another exception occurred: > > > > > Regards > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < > smooney@redhat.com> wrote: > > > > > > > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: > > > > > > > It could also be a compute cell discovery issue maybe? > > > > > > no they shoudl still show up in the hypervior list api > > > > > > > > > > > > > > Do you see anything under "openstack compute service > list"? > > > > > > if they show up in the service list but not they hyperiors > api it > > > > > > means that the comptue service started and registered its > service entry > > > > > > but > > > > > > something broke it before it could create a compute node > recored in the > > > > > > db. > > > > > > > > > > > > with ceph the case i have hit this most often is when the > keyright used > > > > > > by nova to > > > > > > get the avaiable capastiy of the ceph cluster is wrong > whihc prevent > > > > > > the resoucetack and compute manager > > > > > > form actully creating the compute node record. > > > > > > > > > > > > > > > > > > it can happen for other reason too but best place to start > is check if > > > > > > there is an error in the nova compute agent log and go > from there. > > > > > > > > > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < > smooney@redhat.com> > > > > > > wrote: > > > > > > > > > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: > > > > > > > > > Hello Team, > > > > > > > > > > > > > > > > > > I have deployed Openstack Victoria using > Kolla-ansible on Ubuntu > > > > > > 20.04 > > > > > > > > and > > > > > > > > > ceph as the backend storage for Nova, Cinder and > Glance. > > > > > > > > > > > > > > > > > > It finished with no error but it has failed to > register any on the > > > > > > > > Compute > > > > > > > > > Nodes under Hypervisors. > > > > > > > > > > > > > > > > > > kolla-openstack) stack@deployment:~$ openstack > hypervisor list > > > > > > > > > > > > > > > > > > (kolla-openstack) stack@deployment:~$ > > > > > > > > > > > > > > > > > > > > > > > > > > > Any idea on how to resolve this ? > > > > > > > > that usually means that somehthing prevented the > comptue agent form > > > > > > > > strating properly > > > > > > > > > > > > > > > > for example incorrect ceph keyrings there are several > other case > > > > > > but you > > > > > > > > mentioned you are > > > > > > > > using ceph. > > > > > > > > > > > > > > > > if this is hte case you should see error in the > compute agent log. > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > Tony Karera > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Hi Tony, As Laurent mentioned - it would be best if you would reach out on #openstack-kolla IRC channel - and we’ll try to do our best to help you. Here’s an IRC guide from Contributors guide - if you’re not familiar: https://docs.openstack.org/contributors/common/irc.html Regards, Michal Nasiadka W dniu pon., 27.09.2021 o 07:52 Karera Tony <tonykarera@gmail.com> napisał(a):
Hello Team,
I even tried to manually put the ceph.client.cinder.keyring in the nova_compute container but the issue persisted.
I also tried reinstalling Openstack on another Environment but I still have the same issue.
Anyone with any idea on how to proceed ? Regards
Tony Karera
On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
I know that there is some Kolla folks around but keep in mind that this is a volunteer based list :)
I think you might get a bit more one to one help on IRC in their kolla channel.
On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera@gmail.com> wrote:
I would really appreciate any support on this
On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com> wrote:
Hello Team,
I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying.
I don't understand why there is a certain issue now. Regards
Tony Karera
On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Laurent,
It turns out I only have one keyring in the container.
root@compute1:/home/stack# docker exec -it nova_compute bash (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ (nova-compute)[nova@compute1 ceph]$ ls ceph.client.nova.keyring ceph.conf rbdmap
Regards
Tony Karera
On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < laurentfdumont@gmail.com> wrote:
I do believe Kolla runs a container version of each service on computes. Are you looking inside the nova-compute container ( etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin)
On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> wrote:
> Hello Sean, > > Below are the output on the compute node and deployment > > root@compute1:/etc/kolla/nova-compute# ls > ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf > config.json nova.conf > > (kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ > ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf > > And I can confirm that the content is the same. > > > > Regards > > Tony Karera > > > > > On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> > wrote: > >> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >> > I would investigate that compute error first. Creating volumes >> means the >> > controllers are doing the action. Starting a VM on a compute >> means you also >> > need Ceph to works on the compute to mount the rdb target. >> >> nova as part of its startup process in aintiallying the resouce >> tracker will >> try to connect to ceph if you are using the rbd image backend to >> report how much stroage >> is avaiable. if the keyring does not work on the vms pool as the >> user nova is connecting as >> then that will block the agent from starting up fully and will >> cause it to be missing the hypervior list. >> >> the error seams to indicate that the cinder keyring is not in the >> nova container >> that likely means you have not put it in /etc/kolla/config/nova >> i woudl check /etc/kolla/config/nova on the deployment host and >> sudo ls /etc/kolla/nova-compute/ >> on the compute node to ensure the cinder keyring is actully copied >> and has the correct content >> >> i have >> stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >> config.json nova.conf >> >> >> [client.cinder] >> key = ********************************* >> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >> caps mon = "profile rbd" >> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >> profile rbd pool=images" >> stack@cloud:/opt/repos/devstack$ sudo cat >> /etc/kolla/nova-compute/ceph.client.nova.keyring >> [client.nova] >> key = ********************************* >> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >> caps mon = "profile rbd" >> caps osd = "profile rbd pool=volumes, profile rbd pool=vms, >> profile rbd pool=images" >> >> blanked out the key wiht *************** after the fact but you >> should have something similar >> >> >> in my case i decied to use a seperate key for nova rbd backend >> because i was also using EC poosl with a seperate data and metadata pool >> so i neede to modify my ceph.conf to make that work with kolla >> >> stack@cloud:/opt/repos/devstack$ sudo cat >> /etc/kolla/nova-compute/ceph.conf >> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >> [global] >> fsid = ********************* >> mon_host = [*********************] >> >> [client.glance] >> rbd default data pool = images-data >> >> [client.cinder] >> rbd default data pool = volumes-data >> >> [client.nova] >> rbd default data pool = vms-data >> >> using 2 keyrings/user allows me to set different default data pools >> for cinder and nova. >> >> > >> > Working in Wallaby with the error doesn't mean it would 100% work >> in >> > Victoria. >> > >> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera@gmail.com> >> wrote: >> > >> > > Hey Guys, Any other idea ? >> > > >> > > Regards >> > > >> > > Tony Karera >> > > >> > > >> > > >> > > >> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >> tonykarera@gmail.com> wrote: >> > > >> > > > Just to add on that, >> > > > >> > > > compute service is listed, I can create Volumes, I have the >> same cinder >> > > > keyring in the /etc/kolla/config/nova directory as I have in >> the >> > > > /etc/kolla/config/cinder/cinder-volume directory along with >> the nova keyring >> > > > Regards >> > > > >> > > > Tony Karera >> > > > >> > > > >> > > > >> > > > >> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >> tonykarera@gmail.com> wrote: >> > > > >> > > > > Hello Guys, >> > > > > >> > > > > Thanks a lot. >> > > > > >> > > > > I had actually checked the nova -compute.log on the >> compute node and >> > > > > they were showing the error I will post at the end about >> the cinder keyring >> > > > > but I know its correct because its the same I was using on >> wallaby, I even >> > > > > tried to use another ceph cluster with ofcouse different >> keyrings but its >> > > > > the same issue. >> > > > > >> > > > > Below is the error >> > > > > >> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >> auth: unable to >> > > > > find a keyring on >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000 >> 7fbce2f4f700 -1 >> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >> -1 auth: unable >> > > > > to find a keyring on >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >> 7fbce2f4f700 -1 >> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >> -1 auth: unable >> > > > > to find a keyring on >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000 >> 7fbce2f4f700 -1 >> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >> > > > > >> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >> > > > > disabling cephx\n[errno 2] RADOS object not found (error >> connecting to the >> > > > > cluster)\n' >> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During >> handling of >> > > > > the above exception, another exception occurred: >> > > > > Regards >> > > > > >> > > > > Tony Karera >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >> smooney@redhat.com> wrote: >> > > > > >> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >> > > > > > > It could also be a compute cell discovery issue maybe? >> > > > > > no they shoudl still show up in the hypervior list api >> > > > > > > >> > > > > > > Do you see anything under "openstack compute service >> list"? >> > > > > > if they show up in the service list but not they >> hyperiors api it >> > > > > > means that the comptue service started and registered its >> service entry >> > > > > > but >> > > > > > something broke it before it could create a compute node >> recored in the >> > > > > > db. >> > > > > > >> > > > > > with ceph the case i have hit this most often is when the >> keyright used >> > > > > > by nova to >> > > > > > get the avaiable capastiy of the ceph cluster is wrong >> whihc prevent >> > > > > > the resoucetack and compute manager >> > > > > > form actully creating the compute node record. >> > > > > > >> > > > > > >> > > > > > it can happen for other reason too but best place to >> start is check if >> > > > > > there is an error in the nova compute agent log and go >> from there. >> > > > > > > >> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >> smooney@redhat.com> >> > > > > > wrote: >> > > > > > > >> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >> > > > > > > > > Hello Team, >> > > > > > > > > >> > > > > > > > > I have deployed Openstack Victoria using >> Kolla-ansible on Ubuntu >> > > > > > 20.04 >> > > > > > > > and >> > > > > > > > > ceph as the backend storage for Nova, Cinder and >> Glance. >> > > > > > > > > >> > > > > > > > > It finished with no error but it has failed to >> register any on the >> > > > > > > > Compute >> > > > > > > > > Nodes under Hypervisors. >> > > > > > > > > >> > > > > > > > > kolla-openstack) stack@deployment:~$ openstack >> hypervisor list >> > > > > > > > > >> > > > > > > > > (kolla-openstack) stack@deployment:~$ >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > Any idea on how to resolve this ? >> > > > > > > > that usually means that somehthing prevented the >> comptue agent form >> > > > > > > > strating properly >> > > > > > > > >> > > > > > > > for example incorrect ceph keyrings there are several >> other case >> > > > > > but you >> > > > > > > > mentioned you are >> > > > > > > > using ceph. >> > > > > > > > >> > > > > > > > if this is hte case you should see error in the >> compute agent log. >> > > > > > > > >> > > > > > > > > >> > > > > > > > > Regards >> > > > > > > > > >> > > > > > > > > Tony Karera >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> >> >> -- Michał Nasiadka mnasiadka@gmail.com
I tried it but no one was replying to the chats Regards Tony Karera On Mon, Sep 27, 2021 at 7:57 AM Michał Nasiadka <mnasiadka@gmail.com> wrote:
Hi Tony,
As Laurent mentioned - it would be best if you would reach out on #openstack-kolla IRC channel - and we’ll try to do our best to help you.
Here’s an IRC guide from Contributors guide - if you’re not familiar: https://docs.openstack.org/contributors/common/irc.html
Regards, Michal Nasiadka
W dniu pon., 27.09.2021 o 07:52 Karera Tony <tonykarera@gmail.com> napisał(a):
Hello Team,
I even tried to manually put the ceph.client.cinder.keyring in the nova_compute container but the issue persisted.
I also tried reinstalling Openstack on another Environment but I still have the same issue.
Anyone with any idea on how to proceed ? Regards
Tony Karera
On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
I know that there is some Kolla folks around but keep in mind that this is a volunteer based list :)
I think you might get a bit more one to one help on IRC in their kolla channel.
On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera@gmail.com> wrote:
I would really appreciate any support on this
On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com> wrote:
Hello Team,
I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying.
I don't understand why there is a certain issue now. Regards
Tony Karera
On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Laurent,
It turns out I only have one keyring in the container.
root@compute1:/home/stack# docker exec -it nova_compute bash (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ (nova-compute)[nova@compute1 ceph]$ ls ceph.client.nova.keyring ceph.conf rbdmap
Regards
Tony Karera
On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < laurentfdumont@gmail.com> wrote:
> I do believe Kolla runs a container version of each service on > computes. Are you looking inside the nova-compute container ( > etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. > keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) > > On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> > wrote: > >> Hello Sean, >> >> Below are the output on the compute node and deployment >> >> root@compute1:/etc/kolla/nova-compute# ls >> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >> config.json nova.conf >> >> (kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ >> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >> >> And I can confirm that the content is the same. >> >> >> >> Regards >> >> Tony Karera >> >> >> >> >> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> >> wrote: >> >>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>> > I would investigate that compute error first. Creating volumes >>> means the >>> > controllers are doing the action. Starting a VM on a compute >>> means you also >>> > need Ceph to works on the compute to mount the rdb target. >>> >>> nova as part of its startup process in aintiallying the resouce >>> tracker will >>> try to connect to ceph if you are using the rbd image backend to >>> report how much stroage >>> is avaiable. if the keyring does not work on the vms pool as the >>> user nova is connecting as >>> then that will block the agent from starting up fully and will >>> cause it to be missing the hypervior list. >>> >>> the error seams to indicate that the cinder keyring is not in the >>> nova container >>> that likely means you have not put it in /etc/kolla/config/nova >>> i woudl check /etc/kolla/config/nova on the deployment host and >>> sudo ls /etc/kolla/nova-compute/ >>> on the compute node to ensure the cinder keyring is actully copied >>> and has the correct content >>> >>> i have >>> stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>> config.json nova.conf >>> >>> >>> [client.cinder] >>> key = ********************************* >>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>> caps mon = "profile rbd" >>> caps osd = "profile rbd pool=volumes, profile rbd >>> pool=vms, profile rbd pool=images" >>> stack@cloud:/opt/repos/devstack$ sudo cat >>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>> [client.nova] >>> key = ********************************* >>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>> caps mon = "profile rbd" >>> caps osd = "profile rbd pool=volumes, profile rbd >>> pool=vms, profile rbd pool=images" >>> >>> blanked out the key wiht *************** after the fact but you >>> should have something similar >>> >>> >>> in my case i decied to use a seperate key for nova rbd backend >>> because i was also using EC poosl with a seperate data and metadata pool >>> so i neede to modify my ceph.conf to make that work with kolla >>> >>> stack@cloud:/opt/repos/devstack$ sudo cat >>> /etc/kolla/nova-compute/ceph.conf >>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>> [global] >>> fsid = ********************* >>> mon_host = [*********************] >>> >>> [client.glance] >>> rbd default data pool = images-data >>> >>> [client.cinder] >>> rbd default data pool = volumes-data >>> >>> [client.nova] >>> rbd default data pool = vms-data >>> >>> using 2 keyrings/user allows me to set different default data >>> pools for cinder and nova. >>> >>> > >>> > Working in Wallaby with the error doesn't mean it would 100% >>> work in >>> > Victoria. >>> > >>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony < >>> tonykarera@gmail.com> wrote: >>> > >>> > > Hey Guys, Any other idea ? >>> > > >>> > > Regards >>> > > >>> > > Tony Karera >>> > > >>> > > >>> > > >>> > > >>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>> tonykarera@gmail.com> wrote: >>> > > >>> > > > Just to add on that, >>> > > > >>> > > > compute service is listed, I can create Volumes, I have the >>> same cinder >>> > > > keyring in the /etc/kolla/config/nova directory as I have >>> in the >>> > > > /etc/kolla/config/cinder/cinder-volume directory along with >>> the nova keyring >>> > > > Regards >>> > > > >>> > > > Tony Karera >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>> tonykarera@gmail.com> wrote: >>> > > > >>> > > > > Hello Guys, >>> > > > > >>> > > > > Thanks a lot. >>> > > > > >>> > > > > I had actually checked the nova -compute.log on the >>> compute node and >>> > > > > they were showing the error I will post at the end about >>> the cinder keyring >>> > > > > but I know its correct because its the same I was using on >>> wallaby, I even >>> > > > > tried to use another ceph cluster with ofcouse different >>> keyrings but its >>> > > > > the same issue. >>> > > > > >>> > > > > Below is the error >>> > > > > >>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>> auth: unable to >>> > > > > find a keyring on >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> > > > > (2) No such file or >>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >>> -1 auth: unable >>> > > > > to find a keyring on >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> > > > > (2) No such file or >>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >>> -1 auth: unable >>> > > > > to find a keyring on >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> > > > > (2) No such file or >>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>> connecting to the >>> > > > > cluster)\n' >>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>> During handling of >>> > > > > the above exception, another exception occurred: >>> > > > > Regards >>> > > > > >>> > > > > Tony Karera >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>> smooney@redhat.com> wrote: >>> > > > > >>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>> > > > > > > It could also be a compute cell discovery issue maybe? >>> > > > > > no they shoudl still show up in the hypervior list api >>> > > > > > > >>> > > > > > > Do you see anything under "openstack compute service >>> list"? >>> > > > > > if they show up in the service list but not they >>> hyperiors api it >>> > > > > > means that the comptue service started and registered >>> its service entry >>> > > > > > but >>> > > > > > something broke it before it could create a compute node >>> recored in the >>> > > > > > db. >>> > > > > > >>> > > > > > with ceph the case i have hit this most often is when >>> the keyright used >>> > > > > > by nova to >>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>> whihc prevent >>> > > > > > the resoucetack and compute manager >>> > > > > > form actully creating the compute node record. >>> > > > > > >>> > > > > > >>> > > > > > it can happen for other reason too but best place to >>> start is check if >>> > > > > > there is an error in the nova compute agent log and go >>> from there. >>> > > > > > > >>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>> smooney@redhat.com> >>> > > > > > wrote: >>> > > > > > > >>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>> > > > > > > > > Hello Team, >>> > > > > > > > > >>> > > > > > > > > I have deployed Openstack Victoria using >>> Kolla-ansible on Ubuntu >>> > > > > > 20.04 >>> > > > > > > > and >>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>> Glance. >>> > > > > > > > > >>> > > > > > > > > It finished with no error but it has failed to >>> register any on the >>> > > > > > > > Compute >>> > > > > > > > > Nodes under Hypervisors. >>> > > > > > > > > >>> > > > > > > > > kolla-openstack) stack@deployment:~$ openstack >>> hypervisor list >>> > > > > > > > > >>> > > > > > > > > (kolla-openstack) stack@deployment:~$ >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > Any idea on how to resolve this ? >>> > > > > > > > that usually means that somehthing prevented the >>> comptue agent form >>> > > > > > > > strating properly >>> > > > > > > > >>> > > > > > > > for example incorrect ceph keyrings there are >>> several other case >>> > > > > > but you >>> > > > > > > > mentioned you are >>> > > > > > > > using ceph. >>> > > > > > > > >>> > > > > > > > if this is hte case you should see error in the >>> compute agent log. >>> > > > > > > > >>> > > > > > > > > >>> > > > > > > > > Regards >>> > > > > > > > > >>> > > > > > > > > Tony Karera >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> >>> >>> -- Michał Nasiadka mnasiadka@gmail.com
Hello Laurent, Thanks for the advice. I got a solution from the IRC chat group. Apparently, The ceph_nova_user defaults to the ceph_cinder_username. So even if you used nova at the ceph side, You have to uncomment the ceph_nova_user: "nova" in global.yml since it will default to cinder if not uncommented. It was merged recently Regards Tony Karera On Mon, Sep 27, 2021 at 2:45 PM Karera Tony <tonykarera@gmail.com> wrote:
I tried it but no one was replying to the chats
Regards
Tony Karera
On Mon, Sep 27, 2021 at 7:57 AM Michał Nasiadka <mnasiadka@gmail.com> wrote:
Hi Tony,
As Laurent mentioned - it would be best if you would reach out on #openstack-kolla IRC channel - and we’ll try to do our best to help you.
Here’s an IRC guide from Contributors guide - if you’re not familiar: https://docs.openstack.org/contributors/common/irc.html
Regards, Michal Nasiadka
W dniu pon., 27.09.2021 o 07:52 Karera Tony <tonykarera@gmail.com> napisał(a):
Hello Team,
I even tried to manually put the ceph.client.cinder.keyring in the nova_compute container but the issue persisted.
I also tried reinstalling Openstack on another Environment but I still have the same issue.
Anyone with any idea on how to proceed ? Regards
Tony Karera
On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
I know that there is some Kolla folks around but keep in mind that this is a volunteer based list :)
I think you might get a bit more one to one help on IRC in their kolla channel.
On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera@gmail.com> wrote:
I would really appreciate any support on this
On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com> wrote:
Hello Team,
I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying.
I don't understand why there is a certain issue now. Regards
Tony Karera
On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> wrote:
> Hello Laurent, > > It turns out I only have one keyring in the container. > > root@compute1:/home/stack# docker exec -it nova_compute bash > (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ > (nova-compute)[nova@compute1 ceph]$ ls > ceph.client.nova.keyring ceph.conf rbdmap > > Regards > > Tony Karera > > > > > On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < > laurentfdumont@gmail.com> wrote: > >> I do believe Kolla runs a container version of each service on >> computes. Are you looking inside the nova-compute container ( >> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >> >> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> >> wrote: >> >>> Hello Sean, >>> >>> Below are the output on the compute node and deployment >>> >>> root@compute1:/etc/kolla/nova-compute# ls >>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>> config.json nova.conf >>> >>> (kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ >>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>> >>> And I can confirm that the content is the same. >>> >>> >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> >>> wrote: >>> >>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>> > I would investigate that compute error first. Creating volumes >>>> means the >>>> > controllers are doing the action. Starting a VM on a compute >>>> means you also >>>> > need Ceph to works on the compute to mount the rdb target. >>>> >>>> nova as part of its startup process in aintiallying the resouce >>>> tracker will >>>> try to connect to ceph if you are using the rbd image backend to >>>> report how much stroage >>>> is avaiable. if the keyring does not work on the vms pool as the >>>> user nova is connecting as >>>> then that will block the agent from starting up fully and will >>>> cause it to be missing the hypervior list. >>>> >>>> the error seams to indicate that the cinder keyring is not in the >>>> nova container >>>> that likely means you have not put it in /etc/kolla/config/nova >>>> i woudl check /etc/kolla/config/nova on the deployment host and >>>> sudo ls /etc/kolla/nova-compute/ >>>> on the compute node to ensure the cinder keyring is actully >>>> copied and has the correct content >>>> >>>> i have >>>> stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>> config.json nova.conf >>>> >>>> >>>> [client.cinder] >>>> key = ********************************* >>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>> pool=vms" >>>> caps mon = "profile rbd" >>>> caps osd = "profile rbd pool=volumes, profile rbd >>>> pool=vms, profile rbd pool=images" >>>> stack@cloud:/opt/repos/devstack$ sudo cat >>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>> [client.nova] >>>> key = ********************************* >>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>> pool=vms" >>>> caps mon = "profile rbd" >>>> caps osd = "profile rbd pool=volumes, profile rbd >>>> pool=vms, profile rbd pool=images" >>>> >>>> blanked out the key wiht *************** after the fact but you >>>> should have something similar >>>> >>>> >>>> in my case i decied to use a seperate key for nova rbd backend >>>> because i was also using EC poosl with a seperate data and metadata pool >>>> so i neede to modify my ceph.conf to make that work with kolla >>>> >>>> stack@cloud:/opt/repos/devstack$ sudo cat >>>> /etc/kolla/nova-compute/ceph.conf >>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>> [global] >>>> fsid = ********************* >>>> mon_host = [*********************] >>>> >>>> [client.glance] >>>> rbd default data pool = images-data >>>> >>>> [client.cinder] >>>> rbd default data pool = volumes-data >>>> >>>> [client.nova] >>>> rbd default data pool = vms-data >>>> >>>> using 2 keyrings/user allows me to set different default data >>>> pools for cinder and nova. >>>> >>>> > >>>> > Working in Wallaby with the error doesn't mean it would 100% >>>> work in >>>> > Victoria. >>>> > >>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony < >>>> tonykarera@gmail.com> wrote: >>>> > >>>> > > Hey Guys, Any other idea ? >>>> > > >>>> > > Regards >>>> > > >>>> > > Tony Karera >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>>> tonykarera@gmail.com> wrote: >>>> > > >>>> > > > Just to add on that, >>>> > > > >>>> > > > compute service is listed, I can create Volumes, I have the >>>> same cinder >>>> > > > keyring in the /etc/kolla/config/nova directory as I have >>>> in the >>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with >>>> the nova keyring >>>> > > > Regards >>>> > > > >>>> > > > Tony Karera >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>> tonykarera@gmail.com> wrote: >>>> > > > >>>> > > > > Hello Guys, >>>> > > > > >>>> > > > > Thanks a lot. >>>> > > > > >>>> > > > > I had actually checked the nova -compute.log on the >>>> compute node and >>>> > > > > they were showing the error I will post at the end about >>>> the cinder keyring >>>> > > > > but I know its correct because its the same I was using >>>> on wallaby, I even >>>> > > > > tried to use another ceph cluster with ofcouse different >>>> keyrings but its >>>> > > > > the same issue. >>>> > > > > >>>> > > > > Below is the error >>>> > > > > >>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>> auth: unable to >>>> > > > > find a keyring on >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>> > > > > (2) No such file or >>>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>> 7fbce2f4f700 -1 auth: unable >>>> > > > > to find a keyring on >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>> > > > > (2) No such file or >>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>> 7fbce2f4f700 -1 auth: unable >>>> > > > > to find a keyring on >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>> > > > > (2) No such file or >>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>> connecting to the >>>> > > > > cluster)\n' >>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>> During handling of >>>> > > > > the above exception, another exception occurred: >>>> > > > > Regards >>>> > > > > >>>> > > > > Tony Karera >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>> smooney@redhat.com> wrote: >>>> > > > > >>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>>> > > > > > > It could also be a compute cell discovery issue maybe? >>>> > > > > > no they shoudl still show up in the hypervior list api >>>> > > > > > > >>>> > > > > > > Do you see anything under "openstack compute service >>>> list"? >>>> > > > > > if they show up in the service list but not they >>>> hyperiors api it >>>> > > > > > means that the comptue service started and registered >>>> its service entry >>>> > > > > > but >>>> > > > > > something broke it before it could create a compute >>>> node recored in the >>>> > > > > > db. >>>> > > > > > >>>> > > > > > with ceph the case i have hit this most often is when >>>> the keyright used >>>> > > > > > by nova to >>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>>> whihc prevent >>>> > > > > > the resoucetack and compute manager >>>> > > > > > form actully creating the compute node record. >>>> > > > > > >>>> > > > > > >>>> > > > > > it can happen for other reason too but best place to >>>> start is check if >>>> > > > > > there is an error in the nova compute agent log and go >>>> from there. >>>> > > > > > > >>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>> smooney@redhat.com> >>>> > > > > > wrote: >>>> > > > > > > >>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony >>>> wrote: >>>> > > > > > > > > Hello Team, >>>> > > > > > > > > >>>> > > > > > > > > I have deployed Openstack Victoria using >>>> Kolla-ansible on Ubuntu >>>> > > > > > 20.04 >>>> > > > > > > > and >>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>> Glance. >>>> > > > > > > > > >>>> > > > > > > > > It finished with no error but it has failed to >>>> register any on the >>>> > > > > > > > Compute >>>> > > > > > > > > Nodes under Hypervisors. >>>> > > > > > > > > >>>> > > > > > > > > kolla-openstack) stack@deployment:~$ openstack >>>> hypervisor list >>>> > > > > > > > > >>>> > > > > > > > > (kolla-openstack) stack@deployment:~$ >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > Any idea on how to resolve this ? >>>> > > > > > > > that usually means that somehthing prevented the >>>> comptue agent form >>>> > > > > > > > strating properly >>>> > > > > > > > >>>> > > > > > > > for example incorrect ceph keyrings there are >>>> several other case >>>> > > > > > but you >>>> > > > > > > > mentioned you are >>>> > > > > > > > using ceph. >>>> > > > > > > > >>>> > > > > > > > if this is hte case you should see error in the >>>> compute agent log. >>>> > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > Regards >>>> > > > > > > > > >>>> > > > > > > > > Tony Karera >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> >>>> >>>> -- Michał Nasiadka mnasiadka@gmail.com
Perfect! Glad you were able to find the cause. On Mon, Sep 27, 2021 at 9:58 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Laurent,
Thanks for the advice.
I got a solution from the IRC chat group. Apparently, The ceph_nova_user defaults to the ceph_cinder_username. So even if you used nova at the ceph side, You have to uncomment the ceph_nova_user: "nova" in global.yml since it will default to cinder if not uncommented.
It was merged recently
Regards
Tony Karera
On Mon, Sep 27, 2021 at 2:45 PM Karera Tony <tonykarera@gmail.com> wrote:
I tried it but no one was replying to the chats
Regards
Tony Karera
On Mon, Sep 27, 2021 at 7:57 AM Michał Nasiadka <mnasiadka@gmail.com> wrote:
Hi Tony,
As Laurent mentioned - it would be best if you would reach out on #openstack-kolla IRC channel - and we’ll try to do our best to help you.
Here’s an IRC guide from Contributors guide - if you’re not familiar: https://docs.openstack.org/contributors/common/irc.html
Regards, Michal Nasiadka
W dniu pon., 27.09.2021 o 07:52 Karera Tony <tonykarera@gmail.com> napisał(a):
Hello Team,
I even tried to manually put the ceph.client.cinder.keyring in the nova_compute container but the issue persisted.
I also tried reinstalling Openstack on another Environment but I still have the same issue.
Anyone with any idea on how to proceed ? Regards
Tony Karera
On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont < laurentfdumont@gmail.com> wrote:
I know that there is some Kolla folks around but keep in mind that this is a volunteer based list :)
I think you might get a bit more one to one help on IRC in their kolla channel.
On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera@gmail.com> wrote:
I would really appreciate any support on this
On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com> wrote:
> Hello Team, > > I don't know if there has been any change in the packages but the > way I am deploying is the same way I have been deploying. > > I don't understand why there is a certain issue now. > Regards > > Tony Karera > > > > > On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> > wrote: > >> Hello Laurent, >> >> It turns out I only have one keyring in the container. >> >> root@compute1:/home/stack# docker exec -it nova_compute bash >> (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ >> (nova-compute)[nova@compute1 ceph]$ ls >> ceph.client.nova.keyring ceph.conf rbdmap >> >> Regards >> >> Tony Karera >> >> >> >> >> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < >> laurentfdumont@gmail.com> wrote: >> >>> I do believe Kolla runs a container version of each service on >>> computes. Are you looking inside the nova-compute container ( >>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >>> >>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> >>> wrote: >>> >>>> Hello Sean, >>>> >>>> Below are the output on the compute node and deployment >>>> >>>> root@compute1:/etc/kolla/nova-compute# ls >>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>> config.json nova.conf >>>> >>>> (kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ >>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>> >>>> And I can confirm that the content is the same. >>>> >>>> >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> >>>> wrote: >>>> >>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>>> > I would investigate that compute error first. Creating volumes >>>>> means the >>>>> > controllers are doing the action. Starting a VM on a compute >>>>> means you also >>>>> > need Ceph to works on the compute to mount the rdb target. >>>>> >>>>> nova as part of its startup process in aintiallying the resouce >>>>> tracker will >>>>> try to connect to ceph if you are using the rbd image backend to >>>>> report how much stroage >>>>> is avaiable. if the keyring does not work on the vms pool as >>>>> the user nova is connecting as >>>>> then that will block the agent from starting up fully and will >>>>> cause it to be missing the hypervior list. >>>>> >>>>> the error seams to indicate that the cinder keyring is not in >>>>> the nova container >>>>> that likely means you have not put it in /etc/kolla/config/nova >>>>> i woudl check /etc/kolla/config/nova on the deployment host and >>>>> sudo ls /etc/kolla/nova-compute/ >>>>> on the compute node to ensure the cinder keyring is actully >>>>> copied and has the correct content >>>>> >>>>> i have >>>>> stack@cloud:/opt/repos/devstack$ sudo ls >>>>> /etc/kolla/nova-compute/ >>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>> config.json nova.conf >>>>> >>>>> >>>>> [client.cinder] >>>>> key = ********************************* >>>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>>> pool=vms" >>>>> caps mon = "profile rbd" >>>>> caps osd = "profile rbd pool=volumes, profile rbd >>>>> pool=vms, profile rbd pool=images" >>>>> stack@cloud:/opt/repos/devstack$ sudo cat >>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>>> [client.nova] >>>>> key = ********************************* >>>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>>> pool=vms" >>>>> caps mon = "profile rbd" >>>>> caps osd = "profile rbd pool=volumes, profile rbd >>>>> pool=vms, profile rbd pool=images" >>>>> >>>>> blanked out the key wiht *************** after the fact but you >>>>> should have something similar >>>>> >>>>> >>>>> in my case i decied to use a seperate key for nova rbd backend >>>>> because i was also using EC poosl with a seperate data and metadata pool >>>>> so i neede to modify my ceph.conf to make that work with kolla >>>>> >>>>> stack@cloud:/opt/repos/devstack$ sudo cat >>>>> /etc/kolla/nova-compute/ceph.conf >>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>>> [global] >>>>> fsid = ********************* >>>>> mon_host = [*********************] >>>>> >>>>> [client.glance] >>>>> rbd default data pool = images-data >>>>> >>>>> [client.cinder] >>>>> rbd default data pool = volumes-data >>>>> >>>>> [client.nova] >>>>> rbd default data pool = vms-data >>>>> >>>>> using 2 keyrings/user allows me to set different default data >>>>> pools for cinder and nova. >>>>> >>>>> > >>>>> > Working in Wallaby with the error doesn't mean it would 100% >>>>> work in >>>>> > Victoria. >>>>> > >>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony < >>>>> tonykarera@gmail.com> wrote: >>>>> > >>>>> > > Hey Guys, Any other idea ? >>>>> > > >>>>> > > Regards >>>>> > > >>>>> > > Tony Karera >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>>>> tonykarera@gmail.com> wrote: >>>>> > > >>>>> > > > Just to add on that, >>>>> > > > >>>>> > > > compute service is listed, I can create Volumes, I have >>>>> the same cinder >>>>> > > > keyring in the /etc/kolla/config/nova directory as I have >>>>> in the >>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along >>>>> with the nova keyring >>>>> > > > Regards >>>>> > > > >>>>> > > > Tony Karera >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>>> tonykarera@gmail.com> wrote: >>>>> > > > >>>>> > > > > Hello Guys, >>>>> > > > > >>>>> > > > > Thanks a lot. >>>>> > > > > >>>>> > > > > I had actually checked the nova -compute.log on the >>>>> compute node and >>>>> > > > > they were showing the error I will post at the end about >>>>> the cinder keyring >>>>> > > > > but I know its correct because its the same I was using >>>>> on wallaby, I even >>>>> > > > > tried to use another ceph cluster with ofcouse different >>>>> keyrings but its >>>>> > > > > the same issue. >>>>> > > > > >>>>> > > > > Below is the error >>>>> > > > > >>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>> auth: unable to >>>>> > > > > find a keyring on >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>> > > > > (2) No such file or >>>>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>>> 7fbce2f4f700 -1 auth: unable >>>>> > > > > to find a keyring on >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>> > > > > (2) No such file or >>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>>> 7fbce2f4f700 -1 auth: unable >>>>> > > > > to find a keyring on >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>> > > > > (2) No such file or >>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>>> connecting to the >>>>> > > > > cluster)\n' >>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>> During handling of >>>>> > > > > the above exception, another exception occurred: >>>>> > > > > Regards >>>>> > > > > >>>>> > > > > Tony Karera >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>>> smooney@redhat.com> wrote: >>>>> > > > > >>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont >>>>> wrote: >>>>> > > > > > > It could also be a compute cell discovery issue >>>>> maybe? >>>>> > > > > > no they shoudl still show up in the hypervior list api >>>>> > > > > > > >>>>> > > > > > > Do you see anything under "openstack compute service >>>>> list"? >>>>> > > > > > if they show up in the service list but not they >>>>> hyperiors api it >>>>> > > > > > means that the comptue service started and registered >>>>> its service entry >>>>> > > > > > but >>>>> > > > > > something broke it before it could create a compute >>>>> node recored in the >>>>> > > > > > db. >>>>> > > > > > >>>>> > > > > > with ceph the case i have hit this most often is when >>>>> the keyright used >>>>> > > > > > by nova to >>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>>>> whihc prevent >>>>> > > > > > the resoucetack and compute manager >>>>> > > > > > form actully creating the compute node record. >>>>> > > > > > >>>>> > > > > > >>>>> > > > > > it can happen for other reason too but best place to >>>>> start is check if >>>>> > > > > > there is an error in the nova compute agent log and go >>>>> from there. >>>>> > > > > > > >>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>>> smooney@redhat.com> >>>>> > > > > > wrote: >>>>> > > > > > > >>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony >>>>> wrote: >>>>> > > > > > > > > Hello Team, >>>>> > > > > > > > > >>>>> > > > > > > > > I have deployed Openstack Victoria using >>>>> Kolla-ansible on Ubuntu >>>>> > > > > > 20.04 >>>>> > > > > > > > and >>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>>> Glance. >>>>> > > > > > > > > >>>>> > > > > > > > > It finished with no error but it has failed to >>>>> register any on the >>>>> > > > > > > > Compute >>>>> > > > > > > > > Nodes under Hypervisors. >>>>> > > > > > > > > >>>>> > > > > > > > > kolla-openstack) stack@deployment:~$ openstack >>>>> hypervisor list >>>>> > > > > > > > > >>>>> > > > > > > > > (kolla-openstack) stack@deployment:~$ >>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > Any idea on how to resolve this ? >>>>> > > > > > > > that usually means that somehthing prevented the >>>>> comptue agent form >>>>> > > > > > > > strating properly >>>>> > > > > > > > >>>>> > > > > > > > for example incorrect ceph keyrings there are >>>>> several other case >>>>> > > > > > but you >>>>> > > > > > > > mentioned you are >>>>> > > > > > > > using ceph. >>>>> > > > > > > > >>>>> > > > > > > > if this is hte case you should see error in the >>>>> compute agent log. >>>>> > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > Regards >>>>> > > > > > > > > >>>>> > > > > > > > > Tony Karera >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> >>>>> >>>>> -- Michał Nasiadka mnasiadka@gmail.com
participants (5)
-
Jimmy McArthur
-
Karera Tony
-
Laurent Dumont
-
Michał Nasiadka
-
Sean Mooney