Open Stack

Hi Tony,

As Laurent mentioned - it would be best if you would reach out on
#openstack-kolla IRC channel - and we’ll try to do our best to help you.

Here’s an IRC guide from Contributors guide - if you’re not familiar:
https://docs.openstack.org/contributors/common/irc.html

Regards,
Michal Nasiadka

W dniu pon., 27.09.2021 o 07:52 Karera Tony <tonykarera at gmail.com>
napisał(a):

> Hello Team,
>
> I even tried to manually put the ceph.client.cinder.keyring in the
> nova_compute container but the issue persisted.
>
> I also tried reinstalling Openstack on another Environment but I still
> have the same issue.
>
> Anyone with any idea on how to proceed ?
> Regards
>
> Tony Karera
>
>
>
>
> On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont <laurentfdumont at gmail.com>
> wrote:
>
>> I know that there is some Kolla folks around but keep in mind that this
>> is a volunteer based list :)
>>
>> I think you might get a bit more one to one help on IRC in their kolla
>> channel.
>>
>> On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera at gmail.com> wrote:
>>
>>> I would really appreciate any support on this
>>>
>>> On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera at gmail.com> wrote:
>>>
>>>> Hello Team,
>>>>
>>>> I don't know if there has been any change in the packages but the way I
>>>> am deploying is the same way I have been deploying.
>>>>
>>>> I don't understand why there is a certain issue now.
>>>> Regards
>>>>
>>>> Tony Karera
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera at gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Laurent,
>>>>>
>>>>> It turns out I only have one keyring in the container.
>>>>>
>>>>> root at compute1:/home/stack# docker exec -it nova_compute bash
>>>>> (nova-compute)[nova at compute1 /]$ cd /etc/ceph/
>>>>> (nova-compute)[nova at compute1 ceph]$ ls
>>>>> ceph.client.nova.keyring  ceph.conf  rbdmap
>>>>>
>>>>> Regards
>>>>>
>>>>> Tony Karera
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont <
>>>>> laurentfdumont at gmail.com> wrote:
>>>>>
>>>>>> I do believe Kolla runs a container version of each service on
>>>>>> computes. Are you looking inside the nova-compute container (
>>>>>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.
>>>>>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin)
>>>>>>
>>>>>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello Sean,
>>>>>>>
>>>>>>> Below are the output on the compute node and deployment
>>>>>>>
>>>>>>> root at compute1:/etc/kolla/nova-compute# ls
>>>>>>> ceph.client.cinder.keyring  ceph.client.nova.keyring  ceph.conf
>>>>>>>  config.json  nova.conf
>>>>>>>
>>>>>>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/
>>>>>>> ceph.client.cinder.keyring  ceph.client.nova.keyring  ceph.conf
>>>>>>>
>>>>>>> And I can confirm that the content is the same.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Tony Karera
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote:
>>>>>>>> > I would investigate that compute error first. Creating volumes
>>>>>>>> means the
>>>>>>>> > controllers are doing the action. Starting a VM on a compute
>>>>>>>> means you also
>>>>>>>> > need Ceph to works on the compute to mount the rdb target.
>>>>>>>>
>>>>>>>> nova as part of its startup process in aintiallying the resouce
>>>>>>>> tracker will
>>>>>>>> try to connect to ceph if you are using the rbd image backend to
>>>>>>>> report how much stroage
>>>>>>>> is avaiable. if the keyring does  not work on the vms pool as the
>>>>>>>> user nova is connecting as
>>>>>>>> then that will block the agent from starting up fully and will
>>>>>>>> cause it to be missing the hypervior list.
>>>>>>>>
>>>>>>>> the error seams to indicate that the cinder keyring is not in the
>>>>>>>> nova container
>>>>>>>> that likely means you have not put it in /etc/kolla/config/nova
>>>>>>>> i woudl check /etc/kolla/config/nova on the deployment host and
>>>>>>>> sudo ls /etc/kolla/nova-compute/
>>>>>>>> on the compute node to ensure the cinder keyring is actully copied
>>>>>>>> and has the correct content
>>>>>>>>
>>>>>>>> i have
>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/
>>>>>>>> ceph.client.cinder.keyring  ceph.client.nova.keyring  ceph.conf
>>>>>>>> config.json  nova.conf
>>>>>>>>
>>>>>>>>
>>>>>>>> [client.cinder]
>>>>>>>>         key = *********************************
>>>>>>>>         caps mgr = "profile rbd pool=volumes, profile rbd pool=vms"
>>>>>>>>         caps mon = "profile rbd"
>>>>>>>>         caps osd = "profile rbd pool=volumes, profile rbd pool=vms,
>>>>>>>> profile rbd pool=images"
>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat
>>>>>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring
>>>>>>>> [client.nova]
>>>>>>>>         key = *********************************
>>>>>>>>         caps mgr = "profile rbd pool=volumes, profile rbd pool=vms"
>>>>>>>>         caps mon = "profile rbd"
>>>>>>>>         caps osd = "profile rbd pool=volumes, profile rbd pool=vms,
>>>>>>>> profile rbd pool=images"
>>>>>>>>
>>>>>>>> blanked out the key wiht *************** after the fact but you
>>>>>>>> should have something similar
>>>>>>>>
>>>>>>>>
>>>>>>>> in my case i decied to use a seperate key for nova rbd backend
>>>>>>>> because i was also using EC poosl with a seperate data and metadata pool
>>>>>>>> so i neede to modify my ceph.conf to make that work with kolla
>>>>>>>>
>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat
>>>>>>>> /etc/kolla/nova-compute/ceph.conf
>>>>>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f
>>>>>>>> [global]
>>>>>>>> fsid = *********************
>>>>>>>> mon_host = [*********************]
>>>>>>>>
>>>>>>>> [client.glance]
>>>>>>>> rbd default data pool = images-data
>>>>>>>>
>>>>>>>> [client.cinder]
>>>>>>>> rbd default data pool = volumes-data
>>>>>>>>
>>>>>>>> [client.nova]
>>>>>>>> rbd default data pool = vms-data
>>>>>>>>
>>>>>>>> using 2 keyrings/user allows me to set different default data pools
>>>>>>>> for cinder and nova.
>>>>>>>>
>>>>>>>> >
>>>>>>>> > Working in Wallaby with the error doesn't mean it would 100% work
>>>>>>>> in
>>>>>>>> > Victoria.
>>>>>>>> >
>>>>>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <tonykarera at gmail.com>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > > Hey Guys, Any other idea ?
>>>>>>>> > >
>>>>>>>> > > Regards
>>>>>>>> > >
>>>>>>>> > > Tony Karera
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <
>>>>>>>> tonykarera at gmail.com> wrote:
>>>>>>>> > >
>>>>>>>> > > > Just to add on that,
>>>>>>>> > > >
>>>>>>>> > > > compute service is listed, I can create Volumes, I have the
>>>>>>>> same cinder
>>>>>>>> > > > keyring in the /etc/kolla/config/nova directory as I have in
>>>>>>>> the
>>>>>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with
>>>>>>>> the nova keyring
>>>>>>>> > > > Regards
>>>>>>>> > > >
>>>>>>>> > > > Tony Karera
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > >
>>>>>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <
>>>>>>>> tonykarera at gmail.com> wrote:
>>>>>>>> > > >
>>>>>>>> > > > > Hello Guys,
>>>>>>>> > > > >
>>>>>>>> > > > > Thanks a lot.
>>>>>>>> > > > >
>>>>>>>> > > > > I had actually checked the nova -compute.log  on the
>>>>>>>> compute node and
>>>>>>>> > > > > they were showing the error I will post at the end about
>>>>>>>> the cinder keyring
>>>>>>>> > > > > but I know its correct because its the same I was using on
>>>>>>>> wallaby, I even
>>>>>>>> > > > > tried to use another ceph cluster with ofcouse different
>>>>>>>> keyrings but its
>>>>>>>> > > > > the same issue.
>>>>>>>> > > > >
>>>>>>>> > > > > Below is the error
>>>>>>>> > > > >
>>>>>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1
>>>>>>>> auth: unable to
>>>>>>>> > > > > find a keyring on
>>>>>>>> > > > >
>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
>>>>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.574+0000
>>>>>>>> 7fbce2f4f700 -1
>>>>>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at
>>>>>>>> > > > >
>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700
>>>>>>>> -1 auth: unable
>>>>>>>> > > > > to find a keyring on
>>>>>>>> > > > >
>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
>>>>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000
>>>>>>>> 7fbce2f4f700 -1
>>>>>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at
>>>>>>>> > > > >
>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700
>>>>>>>> -1 auth: unable
>>>>>>>> > > > > to find a keyring on
>>>>>>>> > > > >
>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
>>>>>>>> > > > > (2) No such file or directory\n2021-09-22T15:04:31.582+0000
>>>>>>>> 7fbce2f4f700 -1
>>>>>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at
>>>>>>>> > > > >
>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
>>>>>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error
>>>>>>>> connecting to the
>>>>>>>> > > > > cluster)\n'
>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager
>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager During
>>>>>>>> handling of
>>>>>>>> > > > > the above exception, another exception occurred:
>>>>>>>> > > > > Regards
>>>>>>>> > > > >
>>>>>>>> > > > > Tony Karera
>>>>>>>> > > > >
>>>>>>>> > > > >
>>>>>>>> > > > >
>>>>>>>> > > > >
>>>>>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <
>>>>>>>> smooney at redhat.com> wrote:
>>>>>>>> > > > >
>>>>>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote:
>>>>>>>> > > > > > > It could also be a compute cell discovery issue maybe?
>>>>>>>> > > > > > no they shoudl still show up in the hypervior list api
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > Do you see anything under "openstack compute service
>>>>>>>> list"?
>>>>>>>> > > > > > if they show up in the service list but not they
>>>>>>>> hyperiors api it
>>>>>>>> > > > > > means that the comptue service started and registered its
>>>>>>>> service entry
>>>>>>>> > > > > > but
>>>>>>>> > > > > > something broke it before it could create a compute node
>>>>>>>> recored in the
>>>>>>>> > > > > > db.
>>>>>>>> > > > > >
>>>>>>>> > > > > > with ceph the case i have hit this most often is when the
>>>>>>>> keyright used
>>>>>>>> > > > > > by nova to
>>>>>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong
>>>>>>>> whihc prevent
>>>>>>>> > > > > > the resoucetack and compute manager
>>>>>>>> > > > > > form actully creating the compute node record.
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > > it can happen for other reason too but best place to
>>>>>>>> start is check if
>>>>>>>> > > > > > there is an error in the nova compute agent log and go
>>>>>>>> from there.
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <
>>>>>>>> smooney at redhat.com>
>>>>>>>> > > > > > wrote:
>>>>>>>> > > > > > >
>>>>>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote:
>>>>>>>> > > > > > > > > Hello Team,
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > I have deployed Openstack Victoria using
>>>>>>>> Kolla-ansible on Ubuntu
>>>>>>>> > > > > > 20.04
>>>>>>>> > > > > > > > and
>>>>>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and
>>>>>>>> Glance.
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > It finished with no error but it has failed to
>>>>>>>> register any on the
>>>>>>>> > > > > > > > Compute
>>>>>>>> > > > > > > > > Nodes under Hypervisors.
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack
>>>>>>>> hypervisor list
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > (kolla-openstack) stack at deployment:~$
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > Any idea on how to resolve this ?
>>>>>>>> > > > > > > > that usually means that somehthing prevented the
>>>>>>>> comptue agent form
>>>>>>>> > > > > > > > strating properly
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > for example incorrect ceph keyrings there are several
>>>>>>>> other case
>>>>>>>> > > > > > but you
>>>>>>>> > > > > > > > mentioned you are
>>>>>>>> > > > > > > > using ceph.
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > if this is hte case you should see error in the
>>>>>>>> compute agent log.
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > Regards
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > Tony Karera
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>>
>>>>>>>>
>>>>>>>> --
Michał Nasiadka
mnasiadka at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210927/00d2d149/attachment-0001.htm>