Hello Laurent, Thanks for the advice. I got a solution from the IRC chat group. Apparently, The ceph_nova_user defaults to the ceph_cinder_username. So even if you used nova at the ceph side, You have to uncomment the ceph_nova_user: "nova" in global.yml since it will default to cinder if not uncommented. It was merged recently Regards Tony Karera On Mon, Sep 27, 2021 at 2:45 PM Karera Tony <tonykarera@gmail.com> wrote:
I tried it but no one was replying to the chats
Regards
Tony Karera
On Mon, Sep 27, 2021 at 7:57 AM Michał Nasiadka <mnasiadka@gmail.com> wrote:
Hi Tony,
As Laurent mentioned - it would be best if you would reach out on #openstack-kolla IRC channel - and we’ll try to do our best to help you.
Here’s an IRC guide from Contributors guide - if you’re not familiar: https://docs.openstack.org/contributors/common/irc.html
Regards, Michal Nasiadka
W dniu pon., 27.09.2021 o 07:52 Karera Tony <tonykarera@gmail.com> napisał(a):
Hello Team,
I even tried to manually put the ceph.client.cinder.keyring in the nova_compute container but the issue persisted.
I also tried reinstalling Openstack on another Environment but I still have the same issue.
Anyone with any idea on how to proceed ? Regards
Tony Karera
On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
I know that there is some Kolla folks around but keep in mind that this is a volunteer based list :)
I think you might get a bit more one to one help on IRC in their kolla channel.
On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera@gmail.com> wrote:
I would really appreciate any support on this
On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera@gmail.com> wrote:
Hello Team,
I don't know if there has been any change in the packages but the way I am deploying is the same way I have been deploying.
I don't understand why there is a certain issue now. Regards
Tony Karera
On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera@gmail.com> wrote:
> Hello Laurent, > > It turns out I only have one keyring in the container. > > root@compute1:/home/stack# docker exec -it nova_compute bash > (nova-compute)[nova@compute1 /]$ cd /etc/ceph/ > (nova-compute)[nova@compute1 ceph]$ ls > ceph.client.nova.keyring ceph.conf rbdmap > > Regards > > Tony Karera > > > > > On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont < > laurentfdumont@gmail.com> wrote: > >> I do believe Kolla runs a container version of each service on >> computes. Are you looking inside the nova-compute container ( >> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph. >> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin) >> >> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera@gmail.com> >> wrote: >> >>> Hello Sean, >>> >>> Below are the output on the compute node and deployment >>> >>> root@compute1:/etc/kolla/nova-compute# ls >>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>> config.json nova.conf >>> >>> (kolla-openstack) stack@deployment:~$ ls /etc/kolla/config/nova/ >>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>> >>> And I can confirm that the content is the same. >>> >>> >>> >>> Regards >>> >>> Tony Karera >>> >>> >>> >>> >>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney@redhat.com> >>> wrote: >>> >>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote: >>>> > I would investigate that compute error first. Creating volumes >>>> means the >>>> > controllers are doing the action. Starting a VM on a compute >>>> means you also >>>> > need Ceph to works on the compute to mount the rdb target. >>>> >>>> nova as part of its startup process in aintiallying the resouce >>>> tracker will >>>> try to connect to ceph if you are using the rbd image backend to >>>> report how much stroage >>>> is avaiable. if the keyring does not work on the vms pool as the >>>> user nova is connecting as >>>> then that will block the agent from starting up fully and will >>>> cause it to be missing the hypervior list. >>>> >>>> the error seams to indicate that the cinder keyring is not in the >>>> nova container >>>> that likely means you have not put it in /etc/kolla/config/nova >>>> i woudl check /etc/kolla/config/nova on the deployment host and >>>> sudo ls /etc/kolla/nova-compute/ >>>> on the compute node to ensure the cinder keyring is actully >>>> copied and has the correct content >>>> >>>> i have >>>> stack@cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/ >>>> ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf >>>> config.json nova.conf >>>> >>>> >>>> [client.cinder] >>>> key = ********************************* >>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>> pool=vms" >>>> caps mon = "profile rbd" >>>> caps osd = "profile rbd pool=volumes, profile rbd >>>> pool=vms, profile rbd pool=images" >>>> stack@cloud:/opt/repos/devstack$ sudo cat >>>> /etc/kolla/nova-compute/ceph.client.nova.keyring >>>> [client.nova] >>>> key = ********************************* >>>> caps mgr = "profile rbd pool=volumes, profile rbd >>>> pool=vms" >>>> caps mon = "profile rbd" >>>> caps osd = "profile rbd pool=volumes, profile rbd >>>> pool=vms, profile rbd pool=images" >>>> >>>> blanked out the key wiht *************** after the fact but you >>>> should have something similar >>>> >>>> >>>> in my case i decied to use a seperate key for nova rbd backend >>>> because i was also using EC poosl with a seperate data and metadata pool >>>> so i neede to modify my ceph.conf to make that work with kolla >>>> >>>> stack@cloud:/opt/repos/devstack$ sudo cat >>>> /etc/kolla/nova-compute/ceph.conf >>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f >>>> [global] >>>> fsid = ********************* >>>> mon_host = [*********************] >>>> >>>> [client.glance] >>>> rbd default data pool = images-data >>>> >>>> [client.cinder] >>>> rbd default data pool = volumes-data >>>> >>>> [client.nova] >>>> rbd default data pool = vms-data >>>> >>>> using 2 keyrings/user allows me to set different default data >>>> pools for cinder and nova. >>>> >>>> > >>>> > Working in Wallaby with the error doesn't mean it would 100% >>>> work in >>>> > Victoria. >>>> > >>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony < >>>> tonykarera@gmail.com> wrote: >>>> > >>>> > > Hey Guys, Any other idea ? >>>> > > >>>> > > Regards >>>> > > >>>> > > Tony Karera >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony < >>>> tonykarera@gmail.com> wrote: >>>> > > >>>> > > > Just to add on that, >>>> > > > >>>> > > > compute service is listed, I can create Volumes, I have the >>>> same cinder >>>> > > > keyring in the /etc/kolla/config/nova directory as I have >>>> in the >>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with >>>> the nova keyring >>>> > > > Regards >>>> > > > >>>> > > > Tony Karera >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony < >>>> tonykarera@gmail.com> wrote: >>>> > > > >>>> > > > > Hello Guys, >>>> > > > > >>>> > > > > Thanks a lot. >>>> > > > > >>>> > > > > I had actually checked the nova -compute.log on the >>>> compute node and >>>> > > > > they were showing the error I will post at the end about >>>> the cinder keyring >>>> > > > > but I know its correct because its the same I was using >>>> on wallaby, I even >>>> > > > > tried to use another ceph cluster with ofcouse different >>>> keyrings but its >>>> > > > > the same issue. >>>> > > > > >>>> > > > > Below is the error >>>> > > > > >>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>> auth: unable to >>>> > > > > find a keyring on >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>> > > > > (2) No such file or >>>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1 >>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>> 7fbce2f4f700 -1 auth: unable >>>> > > > > to find a keyring on >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>> > > > > (2) No such file or >>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 >>>> 7fbce2f4f700 -1 auth: unable >>>> > > > > to find a keyring on >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>>> > > > > (2) No such file or >>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1 >>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at >>>> > > > > >>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error >>>> connecting to the >>>> > > > > cluster)\n' >>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager >>>> During handling of >>>> > > > > the above exception, another exception occurred: >>>> > > > > Regards >>>> > > > > >>>> > > > > Tony Karera >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney < >>>> smooney@redhat.com> wrote: >>>> > > > > >>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote: >>>> > > > > > > It could also be a compute cell discovery issue maybe? >>>> > > > > > no they shoudl still show up in the hypervior list api >>>> > > > > > > >>>> > > > > > > Do you see anything under "openstack compute service >>>> list"? >>>> > > > > > if they show up in the service list but not they >>>> hyperiors api it >>>> > > > > > means that the comptue service started and registered >>>> its service entry >>>> > > > > > but >>>> > > > > > something broke it before it could create a compute >>>> node recored in the >>>> > > > > > db. >>>> > > > > > >>>> > > > > > with ceph the case i have hit this most often is when >>>> the keyright used >>>> > > > > > by nova to >>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong >>>> whihc prevent >>>> > > > > > the resoucetack and compute manager >>>> > > > > > form actully creating the compute node record. >>>> > > > > > >>>> > > > > > >>>> > > > > > it can happen for other reason too but best place to >>>> start is check if >>>> > > > > > there is an error in the nova compute agent log and go >>>> from there. >>>> > > > > > > >>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney < >>>> smooney@redhat.com> >>>> > > > > > wrote: >>>> > > > > > > >>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony >>>> wrote: >>>> > > > > > > > > Hello Team, >>>> > > > > > > > > >>>> > > > > > > > > I have deployed Openstack Victoria using >>>> Kolla-ansible on Ubuntu >>>> > > > > > 20.04 >>>> > > > > > > > and >>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and >>>> Glance. >>>> > > > > > > > > >>>> > > > > > > > > It finished with no error but it has failed to >>>> register any on the >>>> > > > > > > > Compute >>>> > > > > > > > > Nodes under Hypervisors. >>>> > > > > > > > > >>>> > > > > > > > > kolla-openstack) stack@deployment:~$ openstack >>>> hypervisor list >>>> > > > > > > > > >>>> > > > > > > > > (kolla-openstack) stack@deployment:~$ >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > Any idea on how to resolve this ? >>>> > > > > > > > that usually means that somehthing prevented the >>>> comptue agent form >>>> > > > > > > > strating properly >>>> > > > > > > > >>>> > > > > > > > for example incorrect ceph keyrings there are >>>> several other case >>>> > > > > > but you >>>> > > > > > > > mentioned you are >>>> > > > > > > > using ceph. >>>> > > > > > > > >>>> > > > > > > > if this is hte case you should see error in the >>>> compute agent log. >>>> > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > Regards >>>> > > > > > > > > >>>> > > > > > > > > Tony Karera >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> >>>> >>>> -- Michał Nasiadka mnasiadka@gmail.com