Open Stack

I can see RADOS not found error multiple times in podman_logs.

2023-02-22T07:01:01.639404765+00:00 stderr F + echo 'Running command:
'\''/usr/bin/glance-api --config-file /etc/glance/glance-api.conf
--config-file /etc/glance/glance-image-import.conf'\'''
2023-02-22T07:01:01.639420550+00:00 stdout F Running command:
'/usr/bin/glance-api --config-file /etc/glance/glance-api.conf
--config-file /etc/glance/glance-image-import.conf'
2023-02-22T07:01:01.639464013+00:00 stderr F + exec /usr/bin/glance-api
--config-file /etc/glance/glance-api.conf --config-file
/etc/glance/glance-image-import.conf
2023-02-22T07:01:03.662036516+00:00 stderr F ERROR: [errno 2] RADOS object
not found (error calling conf_read_file)

I think this might be the reason glance-api is stuck in a recursive loop.


Thanks & Best Regards,

Abhishek Kekane


On Wed, Feb 22, 2023 at 12:57 PM Swogat Pradhan <swogatpradhan22 at gmail.com>
wrote:

> Attaching podman logs glance_api.
>
> With regards,
> Swogat Pradhan
>
> On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan <swogatpradhan22 at gmail.com>
> wrote:
>
>> Hi All,
>> I have set up glance multi store in the central site and the same issue
>> is present.
>> Attaching the logs.
>>
>> With regards,
>> Swogat Pradhan
>>
>> On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan <
>> swogatpradhan22 at gmail.com> wrote:
>>
>>> Hi,
>>> I am currently working on client approval to run the stack update.
>>> I will share the logs shortly.
>>>
>>> With regards,
>>> Swogat Pradhan
>>>
>>> On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto at redhat.com> wrote:
>>>
>>>> I agree with Pranali and Takashi.
>>>>
>>>> I have also seen "No module named 'boto3'" in my logs while setting up
>>>> DCN with glance successfully. So that is not the cause of your fatal
>>>> error. You should look in your logs for something else as the root
>>>> cause of this issue.
>>>>
>>>>   John
>>>>
>>>> On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam at redhat.com>
>>>> wrote:
>>>> >
>>>> > I looked at tcib definitions but we have never added boto3 even in
>>>> master.
>>>> >
>>>> > However, according to the glance_store code, I expect glance_store
>>>> would be able to ignore the error
>>>> > even if boto3 is not enabled unless you enable s3 backend somehow.
>>>> >
>>>> >
>>>> https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224
>>>> >
>>>> > If you can find the log line "Failed to load driver ..." in
>>>> glance-api.log then the cause would be different
>>>> > and we have to check the full logs(both
>>>> /var/log/containers/glance/glance-api.log and probably
>>>> > /var/log/containers/sdtouts/glance_api.log) to find out what is
>>>> actually causing the crash.
>>>> >
>>>> > On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore at redhat.com>
>>>> wrote:
>>>> >>
>>>> >> This doesn't seem like a fatal error, so the issue might be
>>>> something else.
>>>> >> What podman logs glance_api shows?
>>>> >>
>>>> >> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <
>>>> swogatpradhan22 at gmail.com> wrote:
>>>> >>>
>>>> >>> Hi Abhishek,
>>>> >>> Yes, this feels like the same bug, as per the comment from Lars
>>>> Wiegman (lars-fuga) my glance api container is in reboot loop, when glance
>>>> multi store is configured.
>>>> >>>
>>>> >>> So I have been searching for a patch for tripleo.
>>>> >>>
>>>> >>> With regards,
>>>> >>> Swogat Pradhan
>>>> >>>
>>>> >>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <
>>>> akekane at redhat.com> wrote:
>>>> >>>>
>>>> >>>> Hi Swogat,
>>>> >>>>
>>>> >>>> I think this looks relevant, kindly go through this and confirm.
>>>> >>>> https://bugs.launchpad.net/kolla/+bug/1884259
>>>> >>>>
>>>> >>>>
>>>> >>>> Thanks & Best Regards,
>>>> >>>>
>>>> >>>> Abhishek Kekane
>>>> >>>>
>>>> >>>>
>>>> >>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <
>>>> swogatpradhan22 at gmail.com> wrote:
>>>> >>>>>
>>>> >>>>> Hi pranali,
>>>> >>>>> Those configs('central', 'cinder_store') were added when i had
>>>> started deploying the cloud, those parameters have been removed from the
>>>> yaml now, but exist in the config file.
>>>> >>>>> Right now i have reverted the config so i am not getting any
>>>> erros in the glance api logs.
>>>> >>>>> But when i configure the multistore i get the following errors on
>>>> both sides (central and edge) in glance_api.log:
>>>> >>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model
>>>> set to 'EventletThreadPoolModel'
>>>> >>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet
>>>> >>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading
>>>> SecretService
>>>> >>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows
>>>> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer
>>>> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS
>>>> >>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token
>>>> [-] AuthToken middleware is set with
>>>> keystone_authtoken.service_token_roles_required set to False. This is
>>>> backwards compatible but deprecated behaviour. Please set this to True.
>>>> >>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not
>>>> load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError:
>>>> No module named 'boto3'
>>>> >>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not
>>>> load 's3': No module named 'boto3': ModuleNotFoundError: No module named
>>>> 'boto3'
>>>> >>>>>
>>>> >>>>> I get no error whatsoever other than the above.
>>>> >>>>>
>>>> >>>>> With regards,
>>>> >>>>> Swogat Pradhan
>>>> >>>>>
>>>> >>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore at redhat.com>
>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> Hey Swogat,
>>>> >>>>>>
>>>> >>>>>> By looking at the conf files you've shared, not sure from where
>>>> the extra store sections 'central', 'cinder_store' are added since you have
>>>> not added those in any of your yaml files.
>>>> >>>>>> Could you please share glance-api logs from both the nodes to
>>>> check more details?
>>>> >>>>>> Probably the HA configuration is causing this kind of issue but
>>>> need more information related to deployment & log to understand the root
>>>> cause.
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> Thanks,
>>>> >>>>>> ~Pranali
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <
>>>> swogatpradhan22 at gmail.com> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Hi Pranali,
>>>> >>>>>>> Thank you for your response.
>>>> >>>>>>> The glance.yaml i had shared is for the central site.
>>>> >>>>>>> I am attaching the required files for your reference.
>>>> >>>>>>>
>>>> >>>>>>> With regards,
>>>> >>>>>>> Swogat Pradhan
>>>> >>>>>>>
>>>> >>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <
>>>> pdeore at redhat.com> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>> Hi Swogat,
>>>> >>>>>>>>
>>>> >>>>>>>> The glance.yaml you've shared is for central site or edge site?
>>>> >>>>>>>> Is it possible for you to share your glance-api.conf just to
>>>> check how configuration values are set?
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> Thanks,
>>>> >>>>>>>> Pranali
>>>> >>>>>>>>
>>>> >>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <
>>>> swogatpradhan22 at gmail.com> wrote:
>>>> >>>>>>>>>
>>>> >>>>>>>>> Additional INFO:
>>>> >>>>>>>>> The glance_api container is in reboot loop after adding multi
>>>> store config.
>>>> >>>>>>>>> Which is why the image service has become unreachable.
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <
>>>> swogatpradhan22 at gmail.com> wrote:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Hi,
>>>> >>>>>>>>>> Openstack wallaby | tripleo
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> I am trying to setup DCN where i want to setup a glance
>>>> multistore config in both central and edge sites as per the documentation.
>>>> >>>>>>>>>> But when ever i run stack update with the following config
>>>> for multistore:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> [stack at hkg2director ~]$ cat workplace/glance.yaml
>>>> >>>>>>>>>> parameter_defaults:
>>>> >>>>>>>>>>   GlanceShowMultipleLocations: true
>>>> >>>>>>>>>>   GlanceBackend: rbd
>>>> >>>>>>>>>>   GlanceEnabledImportMethods: web-download,copy-image
>>>> >>>>>>>>>>   GlanceBackendID: ceph
>>>> >>>>>>>>>>   GlanceMultistoreConfig:
>>>> >>>>>>>>>>     dcn01:
>>>> >>>>>>>>>>       GlanceBackend: rbd
>>>> >>>>>>>>>>       GlanceStoreDescription: 'dcn01 rbd glance store'
>>>> >>>>>>>>>>       CephClusterName: dcn01
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> The image service becomes inaccessible. Giving the following
>>>> error:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> (overcloud) [stack at hkg2director ~]$ openstack image list
>>>> >>>>>>>>>> Failed to contact the endpoint at
>>>> https://bolt.bdxworld.com:13292 for discovery. Fallback to using that
>>>> endpoint as the base url.
>>>> >>>>>>>>>> Failed to contact the endpoint at
>>>> https://bolt.bdxworld.com:13292 for discovery. Fallback to using that
>>>> endpoint as the base url.
>>>> >>>>>>>>>> The image service for :regionOne exists but does not have
>>>> any supported versions.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> glance api.log from controller nodes:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool
>>>> model set to 'EventletThreadPoolModel'
>>>> >>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading
>>>> KWallet
>>>> >>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading
>>>> SecretService
>>>> >>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading
>>>> Windows
>>>> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading
>>>> chainer
>>>> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading
>>>> macOS
>>>> >>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING
>>>> keystonemiddleware.auth_token [-] AuthToken middleware is set with
>>>> keystone_authtoken.service_token_roles_required set to False. This is
>>>> backwards compatible but deprecated behaviour. Please set this to True.
>>>> >>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-]
>>>> Could not load 'glance.store.s3.Store': No module named 'boto3':
>>>> ModuleNotFoundError: No module named 'boto3'
>>>> >>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-]
>>>> Could not load 's3': No module named 'boto3': ModuleNotFoundError: No
>>>> module named 'boto3'
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Is there a patch to fix this issue??
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230222/d57a96b8/attachment-0001.htm>