Open Stack

Hi Abhishek,
I cannot understand why this error is coming though 'ERROR: [errno 2] RADOS
object not found (error calling conf_read_file)'

In controller 1 and Edge 1 nodes i have the config files of both sides
along with the openstack and admin keyring files.
And in other controllers and edge nodes i have config files and
openstack keyring of other site and admin keyring of the same site.

When i am trying to connect using cephadm shell --config ceph.conf /
dcn01.conf --keyring <admin keyring> --fsid <central/edge> i am able to
connect to the cluster and issue commands as well.

With regards,
Swogat Pradhan

On Wed, Feb 22, 2023 at 1:02 PM Abhishek Kekane <akekane at redhat.com> wrote:

> I can see RADOS not found error multiple times in podman_logs.
>
> 2023-02-22T07:01:01.639404765+00:00 stderr F + echo 'Running command:
> '\''/usr/bin/glance-api --config-file /etc/glance/glance-api.conf
> --config-file /etc/glance/glance-image-import.conf'\'''
> 2023-02-22T07:01:01.639420550+00:00 stdout F Running command:
> '/usr/bin/glance-api --config-file /etc/glance/glance-api.conf
> --config-file /etc/glance/glance-image-import.conf'
> 2023-02-22T07:01:01.639464013+00:00 stderr F + exec /usr/bin/glance-api
> --config-file /etc/glance/glance-api.conf --config-file
> /etc/glance/glance-image-import.conf
> 2023-02-22T07:01:03.662036516+00:00 stderr F ERROR: [errno 2] RADOS object
> not found (error calling conf_read_file)
>
> I think this might be the reason glance-api is stuck in a recursive loop.
>
>
> Thanks & Best Regards,
>
> Abhishek Kekane
>
>
> On Wed, Feb 22, 2023 at 12:57 PM Swogat Pradhan <swogatpradhan22 at gmail.com>
> wrote:
>
>> Attaching podman logs glance_api.
>>
>> With regards,
>> Swogat Pradhan
>>
>> On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan <
>> swogatpradhan22 at gmail.com> wrote:
>>
>>> Hi All,
>>> I have set up glance multi store in the central site and the same issue
>>> is present.
>>> Attaching the logs.
>>>
>>> With regards,
>>> Swogat Pradhan
>>>
>>> On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan <
>>> swogatpradhan22 at gmail.com> wrote:
>>>
>>>> Hi,
>>>> I am currently working on client approval to run the stack update.
>>>> I will share the logs shortly.
>>>>
>>>> With regards,
>>>> Swogat Pradhan
>>>>
>>>> On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto at redhat.com>
>>>> wrote:
>>>>
>>>>> I agree with Pranali and Takashi.
>>>>>
>>>>> I have also seen "No module named 'boto3'" in my logs while setting up
>>>>> DCN with glance successfully. So that is not the cause of your fatal
>>>>> error. You should look in your logs for something else as the root
>>>>> cause of this issue.
>>>>>
>>>>>   John
>>>>>
>>>>> On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam at redhat.com>
>>>>> wrote:
>>>>> >
>>>>> > I looked at tcib definitions but we have never added boto3 even in
>>>>> master.
>>>>> >
>>>>> > However, according to the glance_store code, I expect glance_store
>>>>> would be able to ignore the error
>>>>> > even if boto3 is not enabled unless you enable s3 backend somehow.
>>>>> >
>>>>> >
>>>>> https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224
>>>>> >
>>>>> > If you can find the log line "Failed to load driver ..." in
>>>>> glance-api.log then the cause would be different
>>>>> > and we have to check the full logs(both
>>>>> /var/log/containers/glance/glance-api.log and probably
>>>>> > /var/log/containers/sdtouts/glance_api.log) to find out what is
>>>>> actually causing the crash.
>>>>> >
>>>>> > On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore at redhat.com>
>>>>> wrote:
>>>>> >>
>>>>> >> This doesn't seem like a fatal error, so the issue might be
>>>>> something else.
>>>>> >> What podman logs glance_api shows?
>>>>> >>
>>>>> >> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <
>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>> >>>
>>>>> >>> Hi Abhishek,
>>>>> >>> Yes, this feels like the same bug, as per the comment from Lars
>>>>> Wiegman (lars-fuga) my glance api container is in reboot loop, when glance
>>>>> multi store is configured.
>>>>> >>>
>>>>> >>> So I have been searching for a patch for tripleo.
>>>>> >>>
>>>>> >>> With regards,
>>>>> >>> Swogat Pradhan
>>>>> >>>
>>>>> >>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <
>>>>> akekane at redhat.com> wrote:
>>>>> >>>>
>>>>> >>>> Hi Swogat,
>>>>> >>>>
>>>>> >>>> I think this looks relevant, kindly go through this and confirm.
>>>>> >>>> https://bugs.launchpad.net/kolla/+bug/1884259
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> Thanks & Best Regards,
>>>>> >>>>
>>>>> >>>> Abhishek Kekane
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <
>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>> >>>>>
>>>>> >>>>> Hi pranali,
>>>>> >>>>> Those configs('central', 'cinder_store') were added when i had
>>>>> started deploying the cloud, those parameters have been removed from the
>>>>> yaml now, but exist in the config file.
>>>>> >>>>> Right now i have reverted the config so i am not getting any
>>>>> erros in the glance api logs.
>>>>> >>>>> But when i configure the multistore i get the following errors
>>>>> on both sides (central and edge) in glance_api.log:
>>>>> >>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool
>>>>> model set to 'EventletThreadPoolModel'
>>>>> >>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading
>>>>> KWallet
>>>>> >>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading
>>>>> SecretService
>>>>> >>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading
>>>>> Windows
>>>>> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading
>>>>> chainer
>>>>> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS
>>>>> >>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token
>>>>> [-] AuthToken middleware is set with
>>>>> keystone_authtoken.service_token_roles_required set to False. This is
>>>>> backwards compatible but deprecated behaviour. Please set this to True.
>>>>> >>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could
>>>>> not load 'glance.store.s3.Store': No module named 'boto3':
>>>>> ModuleNotFoundError: No module named 'boto3'
>>>>> >>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could
>>>>> not load 's3': No module named 'boto3': ModuleNotFoundError: No module
>>>>> named 'boto3'
>>>>> >>>>>
>>>>> >>>>> I get no error whatsoever other than the above.
>>>>> >>>>>
>>>>> >>>>> With regards,
>>>>> >>>>> Swogat Pradhan
>>>>> >>>>>
>>>>> >>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <
>>>>> pdeore at redhat.com> wrote:
>>>>> >>>>>>
>>>>> >>>>>> Hey Swogat,
>>>>> >>>>>>
>>>>> >>>>>> By looking at the conf files you've shared, not sure from where
>>>>> the extra store sections 'central', 'cinder_store' are added since you have
>>>>> not added those in any of your yaml files.
>>>>> >>>>>> Could you please share glance-api logs from both the nodes to
>>>>> check more details?
>>>>> >>>>>> Probably the HA configuration is causing this kind of issue but
>>>>> need more information related to deployment & log to understand the root
>>>>> cause.
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> Thanks,
>>>>> >>>>>> ~Pranali
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <
>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> Hi Pranali,
>>>>> >>>>>>> Thank you for your response.
>>>>> >>>>>>> The glance.yaml i had shared is for the central site.
>>>>> >>>>>>> I am attaching the required files for your reference.
>>>>> >>>>>>>
>>>>> >>>>>>> With regards,
>>>>> >>>>>>> Swogat Pradhan
>>>>> >>>>>>>
>>>>> >>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <
>>>>> pdeore at redhat.com> wrote:
>>>>> >>>>>>>>
>>>>> >>>>>>>> Hi Swogat,
>>>>> >>>>>>>>
>>>>> >>>>>>>> The glance.yaml you've shared is for central site or edge
>>>>> site?
>>>>> >>>>>>>> Is it possible for you to share your glance-api.conf just to
>>>>> check how configuration values are set?
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> Thanks,
>>>>> >>>>>>>> Pranali
>>>>> >>>>>>>>
>>>>> >>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <
>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Additional INFO:
>>>>> >>>>>>>>> The glance_api container is in reboot loop after adding
>>>>> multi store config.
>>>>> >>>>>>>>> Which is why the image service has become unreachable.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <
>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Hi,
>>>>> >>>>>>>>>> Openstack wallaby | tripleo
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> I am trying to setup DCN where i want to setup a glance
>>>>> multistore config in both central and edge sites as per the documentation.
>>>>> >>>>>>>>>> But when ever i run stack update with the following config
>>>>> for multistore:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> [stack at hkg2director ~]$ cat workplace/glance.yaml
>>>>> >>>>>>>>>> parameter_defaults:
>>>>> >>>>>>>>>>   GlanceShowMultipleLocations: true
>>>>> >>>>>>>>>>   GlanceBackend: rbd
>>>>> >>>>>>>>>>   GlanceEnabledImportMethods: web-download,copy-image
>>>>> >>>>>>>>>>   GlanceBackendID: ceph
>>>>> >>>>>>>>>>   GlanceMultistoreConfig:
>>>>> >>>>>>>>>>     dcn01:
>>>>> >>>>>>>>>>       GlanceBackend: rbd
>>>>> >>>>>>>>>>       GlanceStoreDescription: 'dcn01 rbd glance store'
>>>>> >>>>>>>>>>       CephClusterName: dcn01
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> The image service becomes inaccessible. Giving the
>>>>> following error:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> (overcloud) [stack at hkg2director ~]$ openstack image list
>>>>> >>>>>>>>>> Failed to contact the endpoint at
>>>>> https://bolt.bdxworld.com:13292 for discovery. Fallback to using that
>>>>> endpoint as the base url.
>>>>> >>>>>>>>>> Failed to contact the endpoint at
>>>>> https://bolt.bdxworld.com:13292 for discovery. Fallback to using that
>>>>> endpoint as the base url.
>>>>> >>>>>>>>>> The image service for :regionOne exists but does not have
>>>>> any supported versions.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> glance api.log from controller nodes:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool
>>>>> model set to 'EventletThreadPoolModel'
>>>>> >>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading
>>>>> KWallet
>>>>> >>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading
>>>>> SecretService
>>>>> >>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading
>>>>> Windows
>>>>> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading
>>>>> chainer
>>>>> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading
>>>>> macOS
>>>>> >>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING
>>>>> keystonemiddleware.auth_token [-] AuthToken middleware is set with
>>>>> keystone_authtoken.service_token_roles_required set to False. This is
>>>>> backwards compatible but deprecated behaviour. Please set this to True.
>>>>> >>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-]
>>>>> Could not load 'glance.store.s3.Store': No module named 'boto3':
>>>>> ModuleNotFoundError: No module named 'boto3'
>>>>> >>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-]
>>>>> Could not load 's3': No module named 'boto3': ModuleNotFoundError: No
>>>>> module named 'boto3'
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Is there a patch to fix this issue??
>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230222/26b4ab90/attachment-0001.htm>