Open Stack

Perfect! Glad you were able to find the cause.

On Mon, Sep 27, 2021 at 9:58 AM Karera Tony <tonykarera at gmail.com> wrote:

> Hello Laurent,
>
> Thanks for the advice.
>
> I got a solution from the IRC chat group. Apparently, The ceph_nova_user
> defaults to the ceph_cinder_username. So even if you used nova at the ceph
> side, You have to uncomment the ceph_nova_user: "nova"  in global.yml since
> it will default to cinder if not uncommented.
>
> It was merged recently
>
> Regards
>
> Tony Karera
>
>
>
>
> On Mon, Sep 27, 2021 at 2:45 PM Karera Tony <tonykarera at gmail.com> wrote:
>
>> I tried it but no one was replying to the chats
>>
>> Regards
>>
>> Tony Karera
>>
>>
>>
>>
>> On Mon, Sep 27, 2021 at 7:57 AM Michał Nasiadka <mnasiadka at gmail.com>
>> wrote:
>>
>>> Hi Tony,
>>>
>>> As Laurent mentioned - it would be best if you would reach out on
>>> #openstack-kolla IRC channel - and we’ll try to do our best to help you.
>>>
>>> Here’s an IRC guide from Contributors guide - if you’re not familiar:
>>> https://docs.openstack.org/contributors/common/irc.html
>>>
>>> Regards,
>>> Michal Nasiadka
>>>
>>> W dniu pon., 27.09.2021 o 07:52 Karera Tony <tonykarera at gmail.com>
>>> napisał(a):
>>>
>>>> Hello Team,
>>>>
>>>> I even tried to manually put the ceph.client.cinder.keyring in the
>>>> nova_compute container but the issue persisted.
>>>>
>>>> I also tried reinstalling Openstack on another Environment but I still
>>>> have the same issue.
>>>>
>>>> Anyone with any idea on how to proceed ?
>>>> Regards
>>>>
>>>> Tony Karera
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont <
>>>> laurentfdumont at gmail.com> wrote:
>>>>
>>>>> I know that there is some Kolla folks around but keep in mind that
>>>>> this is a volunteer based list :)
>>>>>
>>>>> I think you might get a bit more one to one help on IRC in their kolla
>>>>> channel.
>>>>>
>>>>> On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I would really appreciate any support on this
>>>>>>
>>>>>> On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera at gmail.com> wrote:
>>>>>>
>>>>>>> Hello Team,
>>>>>>>
>>>>>>> I don't know if there has been any change in the packages but the
>>>>>>> way I am deploying is the same way I have been deploying.
>>>>>>>
>>>>>>> I don't understand why there is a certain issue now.
>>>>>>> Regards
>>>>>>>
>>>>>>> Tony Karera
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello Laurent,
>>>>>>>>
>>>>>>>> It turns out I only have one keyring in the container.
>>>>>>>>
>>>>>>>> root at compute1:/home/stack# docker exec -it nova_compute bash
>>>>>>>> (nova-compute)[nova at compute1 /]$ cd /etc/ceph/
>>>>>>>> (nova-compute)[nova at compute1 ceph]$ ls
>>>>>>>> ceph.client.nova.keyring  ceph.conf  rbdmap
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Tony Karera
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont <
>>>>>>>> laurentfdumont at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I do believe Kolla runs a container version of each service on
>>>>>>>>> computes. Are you looking inside the nova-compute container (
>>>>>>>>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.
>>>>>>>>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin)
>>>>>>>>>
>>>>>>>>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hello Sean,
>>>>>>>>>>
>>>>>>>>>> Below are the output on the compute node and deployment
>>>>>>>>>>
>>>>>>>>>> root at compute1:/etc/kolla/nova-compute# ls
>>>>>>>>>> ceph.client.cinder.keyring  ceph.client.nova.keyring  ceph.conf
>>>>>>>>>>  config.json  nova.conf
>>>>>>>>>>
>>>>>>>>>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/
>>>>>>>>>> ceph.client.cinder.keyring  ceph.client.nova.keyring  ceph.conf
>>>>>>>>>>
>>>>>>>>>> And I can confirm that the content is the same.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>> Tony Karera
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney at redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote:
>>>>>>>>>>> > I would investigate that compute error first. Creating volumes
>>>>>>>>>>> means the
>>>>>>>>>>> > controllers are doing the action. Starting a VM on a compute
>>>>>>>>>>> means you also
>>>>>>>>>>> > need Ceph to works on the compute to mount the rdb target.
>>>>>>>>>>>
>>>>>>>>>>> nova as part of its startup process in aintiallying the resouce
>>>>>>>>>>> tracker will
>>>>>>>>>>> try to connect to ceph if you are using the rbd image backend to
>>>>>>>>>>> report how much stroage
>>>>>>>>>>> is avaiable. if the keyring does  not work on the vms pool as
>>>>>>>>>>> the user nova is connecting as
>>>>>>>>>>> then that will block the agent from starting up fully and will
>>>>>>>>>>> cause it to be missing the hypervior list.
>>>>>>>>>>>
>>>>>>>>>>> the error seams to indicate that the cinder keyring is not in
>>>>>>>>>>> the nova container
>>>>>>>>>>> that likely means you have not put it in /etc/kolla/config/nova
>>>>>>>>>>> i woudl check /etc/kolla/config/nova on the deployment host and
>>>>>>>>>>> sudo ls /etc/kolla/nova-compute/
>>>>>>>>>>> on the compute node to ensure the cinder keyring is actully
>>>>>>>>>>> copied and has the correct content
>>>>>>>>>>>
>>>>>>>>>>> i have
>>>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo ls
>>>>>>>>>>> /etc/kolla/nova-compute/
>>>>>>>>>>> ceph.client.cinder.keyring  ceph.client.nova.keyring  ceph.conf
>>>>>>>>>>> config.json  nova.conf
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [client.cinder]
>>>>>>>>>>>         key = *********************************
>>>>>>>>>>>         caps mgr = "profile rbd pool=volumes, profile rbd
>>>>>>>>>>> pool=vms"
>>>>>>>>>>>         caps mon = "profile rbd"
>>>>>>>>>>>         caps osd = "profile rbd pool=volumes, profile rbd
>>>>>>>>>>> pool=vms, profile rbd pool=images"
>>>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat
>>>>>>>>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring
>>>>>>>>>>> [client.nova]
>>>>>>>>>>>         key = *********************************
>>>>>>>>>>>         caps mgr = "profile rbd pool=volumes, profile rbd
>>>>>>>>>>> pool=vms"
>>>>>>>>>>>         caps mon = "profile rbd"
>>>>>>>>>>>         caps osd = "profile rbd pool=volumes, profile rbd
>>>>>>>>>>> pool=vms, profile rbd pool=images"
>>>>>>>>>>>
>>>>>>>>>>> blanked out the key wiht *************** after the fact but you
>>>>>>>>>>> should have something similar
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> in my case i decied to use a seperate key for nova rbd backend
>>>>>>>>>>> because i was also using EC poosl with a seperate data and metadata pool
>>>>>>>>>>> so i neede to modify my ceph.conf to make that work with kolla
>>>>>>>>>>>
>>>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat
>>>>>>>>>>> /etc/kolla/nova-compute/ceph.conf
>>>>>>>>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f
>>>>>>>>>>> [global]
>>>>>>>>>>> fsid = *********************
>>>>>>>>>>> mon_host = [*********************]
>>>>>>>>>>>
>>>>>>>>>>> [client.glance]
>>>>>>>>>>> rbd default data pool = images-data
>>>>>>>>>>>
>>>>>>>>>>> [client.cinder]
>>>>>>>>>>> rbd default data pool = volumes-data
>>>>>>>>>>>
>>>>>>>>>>> [client.nova]
>>>>>>>>>>> rbd default data pool = vms-data
>>>>>>>>>>>
>>>>>>>>>>> using 2 keyrings/user allows me to set different default data
>>>>>>>>>>> pools for cinder and nova.
>>>>>>>>>>>
>>>>>>>>>>> >
>>>>>>>>>>> > Working in Wallaby with the error doesn't mean it would 100%
>>>>>>>>>>> work in
>>>>>>>>>>> > Victoria.
>>>>>>>>>>> >
>>>>>>>>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <
>>>>>>>>>>> tonykarera at gmail.com> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > > Hey Guys, Any other idea ?
>>>>>>>>>>> > >
>>>>>>>>>>> > > Regards
>>>>>>>>>>> > >
>>>>>>>>>>> > > Tony Karera
>>>>>>>>>>> > >
>>>>>>>>>>> > >
>>>>>>>>>>> > >
>>>>>>>>>>> > >
>>>>>>>>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <
>>>>>>>>>>> tonykarera at gmail.com> wrote:
>>>>>>>>>>> > >
>>>>>>>>>>> > > > Just to add on that,
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > compute service is listed, I can create Volumes, I have
>>>>>>>>>>> the same cinder
>>>>>>>>>>> > > > keyring in the /etc/kolla/config/nova directory as I have
>>>>>>>>>>> in  the
>>>>>>>>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along
>>>>>>>>>>> with the nova keyring
>>>>>>>>>>> > > > Regards
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > Tony Karera
>>>>>>>>>>> > > >
>>>>>>>>>>> > > >
>>>>>>>>>>> > > >
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <
>>>>>>>>>>> tonykarera at gmail.com> wrote:
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > > Hello Guys,
>>>>>>>>>>> > > > >
>>>>>>>>>>> > > > > Thanks a lot.
>>>>>>>>>>> > > > >
>>>>>>>>>>> > > > > I had actually checked the nova -compute.log  on the
>>>>>>>>>>> compute node and
>>>>>>>>>>> > > > > they were showing the error I will post at the end about
>>>>>>>>>>> the cinder keyring
>>>>>>>>>>> > > > > but I know its correct because its the same I was using
>>>>>>>>>>> on wallaby, I even
>>>>>>>>>>> > > > > tried to use another ceph cluster with ofcouse different
>>>>>>>>>>> keyrings but its
>>>>>>>>>>> > > > > the same issue.
>>>>>>>>>>> > > > >
>>>>>>>>>>> > > > > Below is the error
>>>>>>>>>>> > > > >
>>>>>>>>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1
>>>>>>>>>>> auth: unable to
>>>>>>>>>>> > > > > find a keyring on
>>>>>>>>>>> > > > >
>>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
>>>>>>>>>>> > > > > (2) No such file or
>>>>>>>>>>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1
>>>>>>>>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at
>>>>>>>>>>> > > > >
>>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
>>>>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000
>>>>>>>>>>> 7fbce2f4f700 -1 auth: unable
>>>>>>>>>>> > > > > to find a keyring on
>>>>>>>>>>> > > > >
>>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
>>>>>>>>>>> > > > > (2) No such file or
>>>>>>>>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1
>>>>>>>>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at
>>>>>>>>>>> > > > >
>>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
>>>>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000
>>>>>>>>>>> 7fbce2f4f700 -1 auth: unable
>>>>>>>>>>> > > > > to find a keyring on
>>>>>>>>>>> > > > >
>>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
>>>>>>>>>>> > > > > (2) No such file or
>>>>>>>>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1
>>>>>>>>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at
>>>>>>>>>>> > > > >
>>>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
>>>>>>>>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error
>>>>>>>>>>> connecting to the
>>>>>>>>>>> > > > > cluster)\n'
>>>>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager
>>>>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager
>>>>>>>>>>> During handling of
>>>>>>>>>>> > > > > the above exception, another exception occurred:
>>>>>>>>>>> > > > > Regards
>>>>>>>>>>> > > > >
>>>>>>>>>>> > > > > Tony Karera
>>>>>>>>>>> > > > >
>>>>>>>>>>> > > > >
>>>>>>>>>>> > > > >
>>>>>>>>>>> > > > >
>>>>>>>>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <
>>>>>>>>>>> smooney at redhat.com> wrote:
>>>>>>>>>>> > > > >
>>>>>>>>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont
>>>>>>>>>>> wrote:
>>>>>>>>>>> > > > > > > It could also be a compute cell discovery issue
>>>>>>>>>>> maybe?
>>>>>>>>>>> > > > > > no they shoudl still show up in the hypervior list api
>>>>>>>>>>> > > > > > >
>>>>>>>>>>> > > > > > > Do you see anything under "openstack compute service
>>>>>>>>>>> list"?
>>>>>>>>>>> > > > > > if they show up in the service list but not they
>>>>>>>>>>> hyperiors api it
>>>>>>>>>>> > > > > > means that the comptue service started and registered
>>>>>>>>>>> its service entry
>>>>>>>>>>> > > > > > but
>>>>>>>>>>> > > > > > something broke it before it could create a compute
>>>>>>>>>>> node recored in the
>>>>>>>>>>> > > > > > db.
>>>>>>>>>>> > > > > >
>>>>>>>>>>> > > > > > with ceph the case i have hit this most often is when
>>>>>>>>>>> the keyright used
>>>>>>>>>>> > > > > > by nova to
>>>>>>>>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong
>>>>>>>>>>> whihc prevent
>>>>>>>>>>> > > > > > the resoucetack and compute manager
>>>>>>>>>>> > > > > > form actully creating the compute node record.
>>>>>>>>>>> > > > > >
>>>>>>>>>>> > > > > >
>>>>>>>>>>> > > > > > it can happen for other reason too but best place to
>>>>>>>>>>> start is check if
>>>>>>>>>>> > > > > > there is an error in the nova compute agent log and go
>>>>>>>>>>> from there.
>>>>>>>>>>> > > > > > >
>>>>>>>>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <
>>>>>>>>>>> smooney at redhat.com>
>>>>>>>>>>> > > > > > wrote:
>>>>>>>>>>> > > > > > >
>>>>>>>>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony
>>>>>>>>>>> wrote:
>>>>>>>>>>> > > > > > > > > Hello Team,
>>>>>>>>>>> > > > > > > > >
>>>>>>>>>>> > > > > > > > > I have deployed Openstack Victoria using
>>>>>>>>>>> Kolla-ansible on Ubuntu
>>>>>>>>>>> > > > > > 20.04
>>>>>>>>>>> > > > > > > > and
>>>>>>>>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and
>>>>>>>>>>> Glance.
>>>>>>>>>>> > > > > > > > >
>>>>>>>>>>> > > > > > > > > It finished with no error but it has failed to
>>>>>>>>>>> register any on the
>>>>>>>>>>> > > > > > > > Compute
>>>>>>>>>>> > > > > > > > > Nodes under Hypervisors.
>>>>>>>>>>> > > > > > > > >
>>>>>>>>>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack
>>>>>>>>>>> hypervisor list
>>>>>>>>>>> > > > > > > > >
>>>>>>>>>>> > > > > > > > > (kolla-openstack) stack at deployment:~$
>>>>>>>>>>> > > > > > > > >
>>>>>>>>>>> > > > > > > > >
>>>>>>>>>>> > > > > > > > > Any idea on how to resolve this ?
>>>>>>>>>>> > > > > > > > that usually means that somehthing prevented the
>>>>>>>>>>> comptue agent form
>>>>>>>>>>> > > > > > > > strating properly
>>>>>>>>>>> > > > > > > >
>>>>>>>>>>> > > > > > > > for example incorrect ceph keyrings there are
>>>>>>>>>>> several other case
>>>>>>>>>>> > > > > > but you
>>>>>>>>>>> > > > > > > > mentioned you are
>>>>>>>>>>> > > > > > > > using ceph.
>>>>>>>>>>> > > > > > > >
>>>>>>>>>>> > > > > > > > if this is hte case you should see error in the
>>>>>>>>>>> compute agent log.
>>>>>>>>>>> > > > > > > >
>>>>>>>>>>> > > > > > > > >
>>>>>>>>>>> > > > > > > > > Regards
>>>>>>>>>>> > > > > > > > >
>>>>>>>>>>> > > > > > > > > Tony Karera
>>>>>>>>>>> > > > > > > >
>>>>>>>>>>> > > > > > > >
>>>>>>>>>>> > > > > > > >
>>>>>>>>>>> > > > > > > >
>>>>>>>>>>> > > > > >
>>>>>>>>>>> > > > > >
>>>>>>>>>>> > > > > >
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>> Michał Nasiadka
>>> mnasiadka at gmail.com
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210927/da7d15ac/attachment-0001.htm>