Openstack hypervisor list is empty

Karera Tony tonykarera at gmail.com
Mon Sep 27 12:45:26 UTC 2021


I tried it but no one was replying to the chats

Regards

Tony Karera




On Mon, Sep 27, 2021 at 7:57 AM Michał Nasiadka <mnasiadka at gmail.com> wrote:

> Hi Tony,
>
> As Laurent mentioned - it would be best if you would reach out on
> #openstack-kolla IRC channel - and we’ll try to do our best to help you.
>
> Here’s an IRC guide from Contributors guide - if you’re not familiar:
> https://docs.openstack.org/contributors/common/irc.html
>
> Regards,
> Michal Nasiadka
>
> W dniu pon., 27.09.2021 o 07:52 Karera Tony <tonykarera at gmail.com>
> napisał(a):
>
>> Hello Team,
>>
>> I even tried to manually put the ceph.client.cinder.keyring in the
>> nova_compute container but the issue persisted.
>>
>> I also tried reinstalling Openstack on another Environment but I still
>> have the same issue.
>>
>> Anyone with any idea on how to proceed ?
>> Regards
>>
>> Tony Karera
>>
>>
>>
>>
>> On Sat, Sep 25, 2021 at 4:08 AM Laurent Dumont <laurentfdumont at gmail.com>
>> wrote:
>>
>>> I know that there is some Kolla folks around but keep in mind that this
>>> is a volunteer based list :)
>>>
>>> I think you might get a bit more one to one help on IRC in their kolla
>>> channel.
>>>
>>> On Fri, Sep 24, 2021 at 5:10 PM Karera Tony <tonykarera at gmail.com>
>>> wrote:
>>>
>>>> I would really appreciate any support on this
>>>>
>>>> On Fri, 24 Sep 2021, 11:13 Karera Tony, <tonykarera at gmail.com> wrote:
>>>>
>>>>> Hello Team,
>>>>>
>>>>> I don't know if there has been any change in the packages but the way
>>>>> I am deploying is the same way I have been deploying.
>>>>>
>>>>> I don't understand why there is a certain issue now.
>>>>> Regards
>>>>>
>>>>> Tony Karera
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 24, 2021 at 7:30 AM Karera Tony <tonykarera at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello Laurent,
>>>>>>
>>>>>> It turns out I only have one keyring in the container.
>>>>>>
>>>>>> root at compute1:/home/stack# docker exec -it nova_compute bash
>>>>>> (nova-compute)[nova at compute1 /]$ cd /etc/ceph/
>>>>>> (nova-compute)[nova at compute1 ceph]$ ls
>>>>>> ceph.client.nova.keyring  ceph.conf  rbdmap
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Tony Karera
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 24, 2021 at 2:47 AM Laurent Dumont <
>>>>>> laurentfdumont at gmail.com> wrote:
>>>>>>
>>>>>>> I do believe Kolla runs a container version of each service on
>>>>>>> computes. Are you looking inside the nova-compute container (
>>>>>>> etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.
>>>>>>> keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin)
>>>>>>>
>>>>>>> On Thu, Sep 23, 2021 at 11:24 AM Karera Tony <tonykarera at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello Sean,
>>>>>>>>
>>>>>>>> Below are the output on the compute node and deployment
>>>>>>>>
>>>>>>>> root at compute1:/etc/kolla/nova-compute# ls
>>>>>>>> ceph.client.cinder.keyring  ceph.client.nova.keyring  ceph.conf
>>>>>>>>  config.json  nova.conf
>>>>>>>>
>>>>>>>> (kolla-openstack) stack at deployment:~$ ls /etc/kolla/config/nova/
>>>>>>>> ceph.client.cinder.keyring  ceph.client.nova.keyring  ceph.conf
>>>>>>>>
>>>>>>>> And I can confirm that the content is the same.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Tony Karera
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 23, 2021 at 3:20 PM Sean Mooney <smooney at redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Thu, 2021-09-23 at 08:59 -0400, Laurent Dumont wrote:
>>>>>>>>> > I would investigate that compute error first. Creating volumes
>>>>>>>>> means the
>>>>>>>>> > controllers are doing the action. Starting a VM on a compute
>>>>>>>>> means you also
>>>>>>>>> > need Ceph to works on the compute to mount the rdb target.
>>>>>>>>>
>>>>>>>>> nova as part of its startup process in aintiallying the resouce
>>>>>>>>> tracker will
>>>>>>>>> try to connect to ceph if you are using the rbd image backend to
>>>>>>>>> report how much stroage
>>>>>>>>> is avaiable. if the keyring does  not work on the vms pool as the
>>>>>>>>> user nova is connecting as
>>>>>>>>> then that will block the agent from starting up fully and will
>>>>>>>>> cause it to be missing the hypervior list.
>>>>>>>>>
>>>>>>>>> the error seams to indicate that the cinder keyring is not in the
>>>>>>>>> nova container
>>>>>>>>> that likely means you have not put it in /etc/kolla/config/nova
>>>>>>>>> i woudl check /etc/kolla/config/nova on the deployment host and
>>>>>>>>> sudo ls /etc/kolla/nova-compute/
>>>>>>>>> on the compute node to ensure the cinder keyring is actully copied
>>>>>>>>> and has the correct content
>>>>>>>>>
>>>>>>>>> i have
>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo ls /etc/kolla/nova-compute/
>>>>>>>>> ceph.client.cinder.keyring  ceph.client.nova.keyring  ceph.conf
>>>>>>>>> config.json  nova.conf
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [client.cinder]
>>>>>>>>>         key = *********************************
>>>>>>>>>         caps mgr = "profile rbd pool=volumes, profile rbd pool=vms"
>>>>>>>>>         caps mon = "profile rbd"
>>>>>>>>>         caps osd = "profile rbd pool=volumes, profile rbd
>>>>>>>>> pool=vms, profile rbd pool=images"
>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat
>>>>>>>>> /etc/kolla/nova-compute/ceph.client.nova.keyring
>>>>>>>>> [client.nova]
>>>>>>>>>         key = *********************************
>>>>>>>>>         caps mgr = "profile rbd pool=volumes, profile rbd pool=vms"
>>>>>>>>>         caps mon = "profile rbd"
>>>>>>>>>         caps osd = "profile rbd pool=volumes, profile rbd
>>>>>>>>> pool=vms, profile rbd pool=images"
>>>>>>>>>
>>>>>>>>> blanked out the key wiht *************** after the fact but you
>>>>>>>>> should have something similar
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> in my case i decied to use a seperate key for nova rbd backend
>>>>>>>>> because i was also using EC poosl with a seperate data and metadata pool
>>>>>>>>> so i neede to modify my ceph.conf to make that work with kolla
>>>>>>>>>
>>>>>>>>> stack at cloud:/opt/repos/devstack$ sudo cat
>>>>>>>>> /etc/kolla/nova-compute/ceph.conf
>>>>>>>>> # minimal ceph.conf for 15b00858-ba8c-11eb-811f-f9257f38002f
>>>>>>>>> [global]
>>>>>>>>> fsid = *********************
>>>>>>>>> mon_host = [*********************]
>>>>>>>>>
>>>>>>>>> [client.glance]
>>>>>>>>> rbd default data pool = images-data
>>>>>>>>>
>>>>>>>>> [client.cinder]
>>>>>>>>> rbd default data pool = volumes-data
>>>>>>>>>
>>>>>>>>> [client.nova]
>>>>>>>>> rbd default data pool = vms-data
>>>>>>>>>
>>>>>>>>> using 2 keyrings/user allows me to set different default data
>>>>>>>>> pools for cinder and nova.
>>>>>>>>>
>>>>>>>>> >
>>>>>>>>> > Working in Wallaby with the error doesn't mean it would 100%
>>>>>>>>> work in
>>>>>>>>> > Victoria.
>>>>>>>>> >
>>>>>>>>> > On Thu, Sep 23, 2021 at 5:02 AM Karera Tony <
>>>>>>>>> tonykarera at gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > > Hey Guys, Any other idea ?
>>>>>>>>> > >
>>>>>>>>> > > Regards
>>>>>>>>> > >
>>>>>>>>> > > Tony Karera
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > > On Wed, Sep 22, 2021 at 5:20 PM Karera Tony <
>>>>>>>>> tonykarera at gmail.com> wrote:
>>>>>>>>> > >
>>>>>>>>> > > > Just to add on that,
>>>>>>>>> > > >
>>>>>>>>> > > > compute service is listed, I can create Volumes, I have the
>>>>>>>>> same cinder
>>>>>>>>> > > > keyring in the /etc/kolla/config/nova directory as I have
>>>>>>>>> in  the
>>>>>>>>> > > > /etc/kolla/config/cinder/cinder-volume directory along with
>>>>>>>>> the nova keyring
>>>>>>>>> > > > Regards
>>>>>>>>> > > >
>>>>>>>>> > > > Tony Karera
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > On Wed, Sep 22, 2021 at 5:08 PM Karera Tony <
>>>>>>>>> tonykarera at gmail.com> wrote:
>>>>>>>>> > > >
>>>>>>>>> > > > > Hello Guys,
>>>>>>>>> > > > >
>>>>>>>>> > > > > Thanks a lot.
>>>>>>>>> > > > >
>>>>>>>>> > > > > I had actually checked the nova -compute.log  on the
>>>>>>>>> compute node and
>>>>>>>>> > > > > they were showing the error I will post at the end about
>>>>>>>>> the cinder keyring
>>>>>>>>> > > > > but I know its correct because its the same I was using on
>>>>>>>>> wallaby, I even
>>>>>>>>> > > > > tried to use another ceph cluster with ofcouse different
>>>>>>>>> keyrings but its
>>>>>>>>> > > > > the same issue.
>>>>>>>>> > > > >
>>>>>>>>> > > > > Below is the error
>>>>>>>>> > > > >
>>>>>>>>> > > > > r Stderr: '2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1
>>>>>>>>> auth: unable to
>>>>>>>>> > > > > find a keyring on
>>>>>>>>> > > > >
>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
>>>>>>>>> > > > > (2) No such file or
>>>>>>>>> directory\n2021-09-22T15:04:31.574+0000 7fbce2f4f700 -1
>>>>>>>>> > > > > AuthRegistry(0x7fbcdc05a8b8) no keyring found at
>>>>>>>>> > > > >
>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
>>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700
>>>>>>>>> -1 auth: unable
>>>>>>>>> > > > > to find a keyring on
>>>>>>>>> > > > >
>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
>>>>>>>>> > > > > (2) No such file or
>>>>>>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1
>>>>>>>>> > > > > AuthRegistry(0x7fbcdc060698) no keyring found at
>>>>>>>>> > > > >
>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
>>>>>>>>> > > > > disabling cephx\n2021-09-22T15:04:31.582+0000 7fbce2f4f700
>>>>>>>>> -1 auth: unable
>>>>>>>>> > > > > to find a keyring on
>>>>>>>>> > > > >
>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
>>>>>>>>> > > > > (2) No such file or
>>>>>>>>> directory\n2021-09-22T15:04:31.582+0000 7fbce2f4f700 -1
>>>>>>>>> > > > > AuthRegistry(0x7fbce2f4e020) no keyring found at
>>>>>>>>> > > > >
>>>>>>>>> /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,,
>>>>>>>>> > > > > disabling cephx\n[errno 2] RADOS object not found (error
>>>>>>>>> connecting to the
>>>>>>>>> > > > > cluster)\n'
>>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager
>>>>>>>>> > > > > 2021-09-22 15:04:31.592 8 ERROR nova.compute.manager
>>>>>>>>> During handling of
>>>>>>>>> > > > > the above exception, another exception occurred:
>>>>>>>>> > > > > Regards
>>>>>>>>> > > > >
>>>>>>>>> > > > > Tony Karera
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > > On Wed, Sep 22, 2021 at 4:50 PM Sean Mooney <
>>>>>>>>> smooney at redhat.com> wrote:
>>>>>>>>> > > > >
>>>>>>>>> > > > > > On Wed, 2021-09-22 at 10:46 -0400, Laurent Dumont wrote:
>>>>>>>>> > > > > > > It could also be a compute cell discovery issue maybe?
>>>>>>>>> > > > > > no they shoudl still show up in the hypervior list api
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > Do you see anything under "openstack compute service
>>>>>>>>> list"?
>>>>>>>>> > > > > > if they show up in the service list but not they
>>>>>>>>> hyperiors api it
>>>>>>>>> > > > > > means that the comptue service started and registered
>>>>>>>>> its service entry
>>>>>>>>> > > > > > but
>>>>>>>>> > > > > > something broke it before it could create a compute node
>>>>>>>>> recored in the
>>>>>>>>> > > > > > db.
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > with ceph the case i have hit this most often is when
>>>>>>>>> the keyright used
>>>>>>>>> > > > > > by nova to
>>>>>>>>> > > > > > get the avaiable capastiy of the ceph cluster is wrong
>>>>>>>>> whihc prevent
>>>>>>>>> > > > > > the resoucetack and compute manager
>>>>>>>>> > > > > > form actully creating the compute node record.
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > it can happen for other reason too but best place to
>>>>>>>>> start is check if
>>>>>>>>> > > > > > there is an error in the nova compute agent log and go
>>>>>>>>> from there.
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > On Wed, Sep 22, 2021 at 10:33 AM Sean Mooney <
>>>>>>>>> smooney at redhat.com>
>>>>>>>>> > > > > > wrote:
>>>>>>>>> > > > > > >
>>>>>>>>> > > > > > > > On Wed, 2021-09-22 at 15:39 +0200, Karera Tony wrote:
>>>>>>>>> > > > > > > > > Hello Team,
>>>>>>>>> > > > > > > > >
>>>>>>>>> > > > > > > > > I have deployed Openstack Victoria using
>>>>>>>>> Kolla-ansible on Ubuntu
>>>>>>>>> > > > > > 20.04
>>>>>>>>> > > > > > > > and
>>>>>>>>> > > > > > > > > ceph as the backend storage for Nova, Cinder and
>>>>>>>>> Glance.
>>>>>>>>> > > > > > > > >
>>>>>>>>> > > > > > > > > It finished with no error but it has failed to
>>>>>>>>> register any on the
>>>>>>>>> > > > > > > > Compute
>>>>>>>>> > > > > > > > > Nodes under Hypervisors.
>>>>>>>>> > > > > > > > >
>>>>>>>>> > > > > > > > > kolla-openstack) stack at deployment:~$ openstack
>>>>>>>>> hypervisor list
>>>>>>>>> > > > > > > > >
>>>>>>>>> > > > > > > > > (kolla-openstack) stack at deployment:~$
>>>>>>>>> > > > > > > > >
>>>>>>>>> > > > > > > > >
>>>>>>>>> > > > > > > > > Any idea on how to resolve this ?
>>>>>>>>> > > > > > > > that usually means that somehthing prevented the
>>>>>>>>> comptue agent form
>>>>>>>>> > > > > > > > strating properly
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > for example incorrect ceph keyrings there are
>>>>>>>>> several other case
>>>>>>>>> > > > > > but you
>>>>>>>>> > > > > > > > mentioned you are
>>>>>>>>> > > > > > > > using ceph.
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > if this is hte case you should see error in the
>>>>>>>>> compute agent log.
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > > >
>>>>>>>>> > > > > > > > > Regards
>>>>>>>>> > > > > > > > >
>>>>>>>>> > > > > > > > > Tony Karera
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
> Michał Nasiadka
> mnasiadka at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210927/c0510948/attachment-0001.htm>


More information about the openstack-discuss mailing list