Perfect! Glad you were able to find the cause. On Mon, Sep 27, 2021 at 9:58 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Laurent,
Thanks for the advice.
I got a solution from the IRC chat group. Apparently, The ceph_nova_user defaults to the ceph_cinder_username. So even if you used nova at the ceph side, You have to uncomment the ceph_nova_user: "nova" in global.yml since it will default to cinder if not uncommented.
It was merged recently
Regards
Tony Karera
On Mon, Sep 27, 2021 at 2:45 PM Karera Tony <tonykarera@gmail.com> wrote:
I tried it but no one was replying to the chats
Regards
Tony Karera
On Mon, Sep 27, 2021 at 7:57 AM Michał Nasiadka <mnasiadka@gmail.com> wrote:
Hi Tony,
As Laurent mentioned - it would be best if you would reach out on #openstack-kolla IRC channel - and we’ll try to do our best to help you.
Here’s an IRC guide from Contributors guide - if you’re not familiar: https://docs.openstack.org/contributors/common/irc.html
Regards, Michal Nasiadka
W dniu pon., 27.09.2021 o 07:52 Karera Tony <tonykarera@gmail.com> napisał(a):
Hello Team,
I even tried to manually put the ceph.client.cinder.keyring in the nova_compute container but the issue persisted.
I also tried reinstalling Openstack on another Environment but I still have the same issue.
Anyone with any idea on how to proceed ? Regards
Tony Karera
On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont < laurentfdumont@gmail.com> wrote:
I know that there is some Kolla folks around but keep in mind that this is a volunteer based list :)
I think you might get a bit more one to one help on IRC in their kolla channel.
On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera@gmail.com> wrote:
I would really appreciate any support on this
On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com> wrote:
> Hello Team, > > I don't know if there has been any change in the packages but the > way I am deploying is the same way I have been deploying. > > I don't understand why there is a certain issue now. > Regards > > Tony Karera > > > > > On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> > wrote: > >> Hello Laurent, >> >> It turns out I only have one keyring in the container. >> >> root@compute1:/home/stack# docker exec -it nova_compute bash >> (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ >> (nova-compute)[nova@compute1 ceph]$ ls >> ceph.client.nova.keyring ceph.conf rbdmap >> >> Regards >> >> Tony Karera >> >> >> >> >> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < >> laurentfdumont@gmail.com> wrote: >> >>> I do believe Kolla runs a container version of each service on >>> computes. Are you looking inside the nova-compute container ( >>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >>> >>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> >>> wrote: >>> >>>> Hello Sean, >>>> >>>> Below are the output on the compute node and deployment >>>> >>>> root@compute1:/etc/kolla/nova-compute# ls >>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>> config.json nova.conf >>>> >>>> (kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ >>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>> >>>> And I can confirm that the content is the same. >>>> >>>> >>>> >>>> Regards >>>> >>>> Tony Karera >>>> >>>> >>>> >>>> >>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> >>>> wrote: >>>> >>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>>> > I would investigate that compute error first. Creating volumes >>>>> means the >>>>> > controllers are doing the action. Starting a VM on a compute >>>>> means you also >>>>> > need Ceph to works on the compute to mount the rdb target. >>>>> >>>>> nova as part of its startup process in aintiallying the resouce >>>>> tracker will >>>>> try to connect to ceph if you are using the rbd image backend to >>>>> report how much stroage >>>>> is avaiable. if the keyring does not work on the vms pool as >>>>> the user nova is connecting as >>>>> then that will block the agent from starting up fully and will >>>>> cause it to be missing the hypervior list. >>>>> >>>>> the error seams to indicate that the cinder keyring is not in >>>>> the nova container >>>>> that likely means you have not put it in /etc/kolla/config/nova >>>>> i woudl check /etc/kolla/config/nova on the deployment host and >>>>> sudo ls /etc/kolla/nova-compute/ >>>>> on the compute node to ensure the cinder keyring is actully >>>>> copied and has the correct content >>>>> >>>>> i have >>>>> stack@cloud:/opt/repos/devstack$ sudo ls >>>>> /etc/kolla/nova-compute/ >>>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>>> config.json nova.conf >>>>> >>>>> >>>>> [client.cinder] >>>>> key = ********************************* >>>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>>> pool=vms" >>>>> caps mon = "profile rbd" >>>>> caps osd = "profile rbd pool=volumes, profile rbd >>>>> pool=vms, profile rbd pool=images" >>>>> stack@cloud:/opt/repos/devstack$ sudo cat >>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>>> [client.nova] >>>>> key = ********************************* >>>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>>> pool=vms" >>>>> caps mon = "profile rbd" >>>>> caps osd = "profile rbd pool=volumes, profile rbd >>>>> pool=vms, profile rbd pool=images" >>>>> >>>>> blanked out the key wiht *************** after the fact but you >>>>> should have something similar >>>>> >>>>> >>>>> in my case i decied to use a seperate key for nova rbd backend >>>>> because i was also using EC poosl with a seperate data and metadata pool >>>>> so i neede to modify my ceph.conf to make that work with kolla >>>>> >>>>> stack@cloud:/opt/repos/devstack$ sudo cat >>>>> /etc/kolla/nova-compute/ceph.conf >>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>>> [global] >>>>> fsid = ********************* >>>>> mon_host = [*********************] >>>>> >>>>> [client.glance] >>>>> rbd default data pool = images-data >>>>> >>>>> [client.cinder] >>>>> rbd default data pool = volumes-data >>>>> >>>>> [client.nova] >>>>> rbd default data pool = vms-data >>>>> >>>>> using 2 keyrings/user allows me to set different default data >>>>> pools for cinder and nova. >>>>> >>>>> > >>>>> > Working in Wallaby with the error doesn't mean it would 100% >>>>> work in >>>>> > Victoria. >>>>> > >>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony < >>>>> tonykarera@gmail.com> wrote: >>>>> > >>>>> > > Hey Guys, Any other idea ? >>>>> > > >>>>> > > Regards >>>>> > > >>>>> > > Tony Karera >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>>>> tonykarera@gmail.com> wrote: >>>>> > > >>>>> > > > Just to add on that, >>>>> > > > >>>>> > > > compute service is listed, I can create Volumes, I have >>>>> the same cinder >>>>> > > > keyring in the /etc/kolla/config/nova directory as I have >>>>> in the >>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along >>>>> with the nova keyring >>>>> > > > Regards >>>>> > > > >>>>> > > > Tony Karera >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>>> tonykarera@gmail.com> wrote: >>>>> > > > >>>>> > > > > Hello Guys, >>>>> > > > > >>>>> > > > > Thanks a lot. >>>>> > > > > >>>>> > > > > I had actually checked the nova -compute.log on the >>>>> compute node and >>>>> > > > > they were showing the error I will post at the end about >>>>> the cinder keyring >>>>> > > > > but I know its correct because its the same I was using >>>>> on wallaby, I even >>>>> > > > > tried to use another ceph cluster with ofcouse different >>>>> keyrings but its >>>>> > > > > the same issue. >>>>> > > > > >>>>> > > > > Below is the error >>>>> > > > > >>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>> auth: unable to >>>>> > > > > find a keyring on >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>> > > > > (2) No such file or >>>>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>>> 7fbce2f4f700 -1 auth: unable >>>>> > > > > to find a keyring on >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>> > > > > (2) No such file or >>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>>> 7fbce2f4f700 -1 auth: unable >>>>> > > > > to find a keyring on >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>>> > > > > (2) No such file or >>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>>> > > > > >>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>>> connecting to the >>>>> > > > > cluster)\n' >>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>>> During handling of >>>>> > > > > the above exception, another exception occurred: >>>>> > > > > Regards >>>>> > > > > >>>>> > > > > Tony Karera >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>>> smooney@redhat.com> wrote: >>>>> > > > > >>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont >>>>> wrote: >>>>> > > > > > > It could also be a compute cell discovery issue >>>>> maybe? >>>>> > > > > > no they shoudl still show up in the hypervior list api >>>>> > > > > > > >>>>> > > > > > > Do you see anything under "openstack compute service >>>>> list"? >>>>> > > > > > if they show up in the service list but not they >>>>> hyperiors api it >>>>> > > > > > means that the comptue service started and registered >>>>> its service entry >>>>> > > > > > but >>>>> > > > > > something broke it before it could create a compute >>>>> node recored in the >>>>> > > > > > db. >>>>> > > > > > >>>>> > > > > > with ceph the case i have hit this most often is when >>>>> the keyright used >>>>> > > > > > by nova to >>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>>>> whihc prevent >>>>> > > > > > the resoucetack and compute manager >>>>> > > > > > form actully creating the compute node record. >>>>> > > > > > >>>>> > > > > > >>>>> > > > > > it can happen for other reason too but best place to >>>>> start is check if >>>>> > > > > > there is an error in the nova compute agent log and go >>>>> from there. >>>>> > > > > > > >>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>>> smooney@redhat.com> >>>>> > > > > > wrote: >>>>> > > > > > > >>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony >>>>> wrote: >>>>> > > > > > > > > Hello Team, >>>>> > > > > > > > > >>>>> > > > > > > > > I have deployed Openstack Victoria using >>>>> Kolla-ansible on Ubuntu >>>>> > > > > > 20.04 >>>>> > > > > > > > and >>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>>> Glance. >>>>> > > > > > > > > >>>>> > > > > > > > > It finished with no error but it has failed to >>>>> register any on the >>>>> > > > > > > > Compute >>>>> > > > > > > > > Nodes under Hypervisors. >>>>> > > > > > > > > >>>>> > > > > > > > > kolla-openstack) stack@deployment:~$ openstack >>>>> hypervisor list >>>>> > > > > > > > > >>>>> > > > > > > > > (kolla-openstack) stack@deployment:~$ >>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > Any idea on how to resolve this ? >>>>> > > > > > > > that usually means that somehthing prevented the >>>>> comptue agent form >>>>> > > > > > > > strating properly >>>>> > > > > > > > >>>>> > > > > > > > for example incorrect ceph keyrings there are >>>>> several other case >>>>> > > > > > but you >>>>> > > > > > > > mentioned you are >>>>> > > > > > > > using ceph. >>>>> > > > > > > > >>>>> > > > > > > > if this is hte case you should see error in the >>>>> compute agent log. >>>>> > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > Regards >>>>> > > > > > > > > >>>>> > > > > > > > > Tony Karera >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> >>>>> >>>>> -- Michał Nasiadka mnasiadka@gmail.com