I tried it but no one was replying to the chats Regards Tony Karera On Mon, Sep 27, 2021 at 7:57 AM Michał Nasiadka <mnasiadka@gmail.com> wrote:
Hi Tony,
As Laurent mentioned - it would be best if you would reach out on #openstack-kolla IRC channel - and we’ll try to do our best to help you.
Here’s an IRC guide from Contributors guide - if you’re not familiar: https://docs.openstack.org/contributors/common/irc.html
Regards, Michal Nasiadka
W dniu pon., 27.09.2021 o 07:52 Karera Tony <tonykarera@gmail.com> napisał(a):
Hello Team,
I even tried to manually put the ceph.client.cinder.keyring in the nova_compute container but the issue persisted.
I also tried reinstalling Openstack on another Environment but I still have the same issue.
Anyone with any idea on how to proceed ? Regards
Tony Karera
On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
I know that there is some Kolla folks around but keep in mind that this is a volunteer based list :)
I think you might get a bit more one to one help on IRC in their kolla channel.
On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera@gmail.com> wrote:
I would really appreciate any support on this
On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com> wrote:
Hello Team,
I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying.
I don't understand why there is a certain issue now. Regards
Tony Karera
On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Laurent,
It turns out I only have one keyring in the container.
root@compute1:/home/stack# docker exec -it nova_compute bash (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ (nova-compute)[nova@compute1 ceph]$ ls ceph.client.nova.keyring ceph.conf rbdmap
Regards
Tony Karera
On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < laurentfdumont@gmail.com> wrote:
> I do believe Kolla runs a container version of each service on > computes. Are you looking inside the nova-compute container ( > etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. > keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) > > On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> > wrote: > >> Hello Sean, >> >> Below are the output on the compute node and deployment >> >> root@compute1:/etc/kolla/nova-compute# ls >> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >> config.json nova.conf >> >> (kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ >> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >> >> And I can confirm that the content is the same. >> >> >> >> Regards >> >> Tony Karera >> >> >> >> >> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> >> wrote: >> >>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>> > I would investigate that compute error first. Creating volumes >>> means the >>> > controllers are doing the action. Starting a VM on a compute >>> means you also >>> > need Ceph to works on the compute to mount the rdb target. >>> >>> nova as part of its startup process in aintiallying the resouce >>> tracker will >>> try to connect to ceph if you are using the rbd image backend to >>> report how much stroage >>> is avaiable. if the keyring does not work on the vms pool as the >>> user nova is connecting as >>> then that will block the agent from starting up fully and will >>> cause it to be missing the hypervior list. >>> >>> the error seams to indicate that the cinder keyring is not in the >>> nova container >>> that likely means you have not put it in /etc/kolla/config/nova >>> i woudl check /etc/kolla/config/nova on the deployment host and >>> sudo ls /etc/kolla/nova-compute/ >>> on the compute node to ensure the cinder keyring is actully copied >>> and has the correct content >>> >>> i have >>> stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>> config.json nova.conf >>> >>> >>> [client.cinder] >>> key = ********************************* >>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>> caps mon = "profile rbd" >>> caps osd = "profile rbd pool=volumes, profile rbd >>> pool=vms, profile rbd pool=images" >>> stack@cloud:/opt/repos/devstack$ sudo cat >>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>> [client.nova] >>> key = ********************************* >>> caps mgr = "profile rbd pool=volumes, profile rbd pool=vms" >>> caps mon = "profile rbd" >>> caps osd = "profile rbd pool=volumes, profile rbd >>> pool=vms, profile rbd pool=images" >>> >>> blanked out the key wiht *************** after the fact but you >>> should have something similar >>> >>> >>> in my case i decied to use a seperate key for nova rbd backend >>> because i was also using EC poosl with a seperate data and metadata pool >>> so i neede to modify my ceph.conf to make that work with kolla >>> >>> stack@cloud:/opt/repos/devstack$ sudo cat >>> /etc/kolla/nova-compute/ceph.conf >>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>> [global] >>> fsid = ********************* >>> mon_host = [*********************] >>> >>> [client.glance] >>> rbd default data pool = images-data >>> >>> [client.cinder] >>> rbd default data pool = volumes-data >>> >>> [client.nova] >>> rbd default data pool = vms-data >>> >>> using 2 keyrings/user allows me to set different default data >>> pools for cinder and nova. >>> >>> > >>> > Working in Wallaby with the error doesn't mean it would 100% >>> work in >>> > Victoria. >>> > >>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony < >>> tonykarera@gmail.com> wrote: >>> > >>> > > Hey Guys, Any other idea ? >>> > > >>> > > Regards >>> > > >>> > > Tony Karera >>> > > >>> > > >>> > > >>> > > >>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>> tonykarera@gmail.com> wrote: >>> > > >>> > > > Just to add on that, >>> > > > >>> > > > compute service is listed, I can create Volumes, I have the >>> same cinder >>> > > > keyring in the /etc/kolla/config/nova directory as I have >>> in the >>> > > > /etc/kolla/config/cinder/cinder-volume directory along with >>> the nova keyring >>> > > > Regards >>> > > > >>> > > > Tony Karera >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>> tonykarera@gmail.com> wrote: >>> > > > >>> > > > > Hello Guys, >>> > > > > >>> > > > > Thanks a lot. >>> > > > > >>> > > > > I had actually checked the nova -compute.log on the >>> compute node and >>> > > > > they were showing the error I will post at the end about >>> the cinder keyring >>> > > > > but I know its correct because its the same I was using on >>> wallaby, I even >>> > > > > tried to use another ceph cluster with ofcouse different >>> keyrings but its >>> > > > > the same issue. >>> > > > > >>> > > > > Below is the error >>> > > > > >>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>> auth: unable to >>> > > > > find a keyring on >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> > > > > (2) No such file or >>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >>> -1 auth: unable >>> > > > > to find a keyring on >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> > > > > (2) No such file or >>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 >>> -1 auth: unable >>> > > > > to find a keyring on >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> > > > > (2) No such file or >>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>> > > > > >>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>> connecting to the >>> > > > > cluster)\n' >>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>> During handling of >>> > > > > the above exception, another exception occurred: >>> > > > > Regards >>> > > > > >>> > > > > Tony Karera >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>> smooney@redhat.com> wrote: >>> > > > > >>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>> > > > > > > It could also be a compute cell discovery issue maybe? >>> > > > > > no they shoudl still show up in the hypervior list api >>> > > > > > > >>> > > > > > > Do you see anything under "openstack compute service >>> list"? >>> > > > > > if they show up in the service list but not they >>> hyperiors api it >>> > > > > > means that the comptue service started and registered >>> its service entry >>> > > > > > but >>> > > > > > something broke it before it could create a compute node >>> recored in the >>> > > > > > db. >>> > > > > > >>> > > > > > with ceph the case i have hit this most often is when >>> the keyright used >>> > > > > > by nova to >>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>> whihc prevent >>> > > > > > the resoucetack and compute manager >>> > > > > > form actully creating the compute node record. >>> > > > > > >>> > > > > > >>> > > > > > it can happen for other reason too but best place to >>> start is check if >>> > > > > > there is an error in the nova compute agent log and go >>> from there. >>> > > > > > > >>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>> smooney@redhat.com> >>> > > > > > wrote: >>> > > > > > > >>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote: >>> > > > > > > > > Hello Team, >>> > > > > > > > > >>> > > > > > > > > I have deployed Openstack Victoria using >>> Kolla-ansible on Ubuntu >>> > > > > > 20.04 >>> > > > > > > > and >>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>> Glance. >>> > > > > > > > > >>> > > > > > > > > It finished with no error but it has failed to >>> register any on the >>> > > > > > > > Compute >>> > > > > > > > > Nodes under Hypervisors. >>> > > > > > > > > >>> > > > > > > > > kolla-openstack) stack@deployment:~$ openstack >>> hypervisor list >>> > > > > > > > > >>> > > > > > > > > (kolla-openstack) stack@deployment:~$ >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > Any idea on how to resolve this ? >>> > > > > > > > that usually means that somehthing prevented the >>> comptue agent form >>> > > > > > > > strating properly >>> > > > > > > > >>> > > > > > > > for example incorrect ceph keyrings there are >>> several other case >>> > > > > > but you >>> > > > > > > > mentioned you are >>> > > > > > > > using ceph. >>> > > > > > > > >>> > > > > > > > if this is hte case you should see error in the >>> compute agent log. >>> > > > > > > > >>> > > > > > > > > >>> > > > > > > > > Regards >>> > > > > > > > > >>> > > > > > > > > Tony Karera >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> >>> >>> -- Michał Nasiadka mnasiadka@gmail.com