Open Stack

I looked at tcib definitions but we have never added boto3 even in master.

However, according to the glance_store code, I expect glance_store would be
able to ignore the error
even if boto3 is not enabled unless you enable s3 backend somehow.

https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224

If you can find the log line "Failed to load driver ..." in glance-api.log
then the cause would be different
and we have to check the full logs(both
/var/log/containers/glance/glance-api.log and probably
/var/log/containers/sdtouts/glance_api.log) to find out what is actually
causing the crash.

On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore at redhat.com> wrote:

> This doesn't seem like a fatal error, so the issue might be something else.
> What podman logs glance_api shows?
>
> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <swogatpradhan22 at gmail.com>
> wrote:
>
>> Hi Abhishek,
>> Yes, this feels like the same bug, as per the comment from Lars Wiegman
>> (lars-fuga) <https://launchpad.net/~lars-fuga> my glance api container
>> is in reboot loop, when glance multi store is configured.
>>
>> So I have been searching for a patch for tripleo.
>>
>> With regards,
>> Swogat Pradhan
>>
>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane at redhat.com>
>> wrote:
>>
>>> Hi Swogat,
>>>
>>> I think this looks relevant, kindly go through this and confirm.
>>> https://bugs.launchpad.net/kolla/+bug/1884259
>>>
>>>
>>> Thanks & Best Regards,
>>>
>>> Abhishek Kekane
>>>
>>>
>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <
>>> swogatpradhan22 at gmail.com> wrote:
>>>
>>>> Hi pranali,
>>>> Those configs('central', 'cinder_store') were added when i had started
>>>> deploying the cloud, those parameters have been removed from the yaml now,
>>>> but exist in the config file.
>>>> Right now i have reverted the config so i am not getting any erros in
>>>> the glance api logs.
>>>> But when i configure the multistore i get the following errors on both
>>>> sides (central and edge) in glance_api.log:
>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set
>>>> to 'EventletThreadPoolModel'
>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet
>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService
>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows
>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer
>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS
>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-]
>>>> AuthToken middleware is set with
>>>> keystone_authtoken.service_token_roles_required set to False. This is
>>>> backwards compatible but deprecated behaviour. Please set this to True.
>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load
>>>> 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No
>>>> module named 'boto3'
>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load
>>>> 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
>>>>
>>>> I get no error whatsoever other than the above.
>>>>
>>>> With regards,
>>>> Swogat Pradhan
>>>>
>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore at redhat.com>
>>>> wrote:
>>>>
>>>>> Hey Swogat,
>>>>>
>>>>> By looking at the conf files you've shared, not sure from where the
>>>>> extra store sections 'central', 'cinder_store' are added since you
>>>>> have not added those in any of your yaml files.
>>>>> Could you please share glance-api logs from both the nodes to check
>>>>> more details?
>>>>> Probably the HA configuration is causing this kind of issue but need
>>>>> more information related to deployment & log to understand the root cause.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> ~Pranali
>>>>>
>>>>>
>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <
>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>
>>>>>> Hi Pranali,
>>>>>> Thank you for your response.
>>>>>> The glance.yaml i had shared is for the central site.
>>>>>> I am attaching the required files for your reference.
>>>>>>
>>>>>> With regards,
>>>>>> Swogat Pradhan
>>>>>>
>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Swogat,
>>>>>>>
>>>>>>> The glance.yaml you've shared is for central site or edge site?
>>>>>>> Is it possible for you to share your glance-api.conf just to check
>>>>>>> how configuration values are set?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Pranali
>>>>>>>
>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <
>>>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>>
>>>>>>>> Additional INFO:
>>>>>>>> The glance_api container is in reboot loop after adding multi store
>>>>>>>> config.
>>>>>>>> Which is why the image service has become unreachable.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <
>>>>>>>> swogatpradhan22 at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> Openstack wallaby | tripleo
>>>>>>>>>
>>>>>>>>> I am trying to setup DCN where i want to setup a glance multistore
>>>>>>>>> config in both central and edge sites as per the documentation.
>>>>>>>>> But when ever i run stack update with the following config for
>>>>>>>>> multistore:
>>>>>>>>>
>>>>>>>>> [stack at hkg2director ~]$ cat workplace/glance.yaml
>>>>>>>>> parameter_defaults:
>>>>>>>>>   GlanceShowMultipleLocations: true
>>>>>>>>>   GlanceBackend: rbd
>>>>>>>>>   GlanceEnabledImportMethods: web-download,copy-image
>>>>>>>>>   GlanceBackendID: ceph
>>>>>>>>>   GlanceMultistoreConfig:
>>>>>>>>>     dcn01:
>>>>>>>>>       GlanceBackend: rbd
>>>>>>>>>       GlanceStoreDescription: 'dcn01 rbd glance store'
>>>>>>>>>       CephClusterName: dcn01
>>>>>>>>>
>>>>>>>>> The image service becomes inaccessible. Giving the following error:
>>>>>>>>>
>>>>>>>>> (overcloud) [stack at hkg2director ~]$ openstack image list
>>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292
>>>>>>>>> for discovery. Fallback to using that endpoint as the base url.
>>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292
>>>>>>>>> for discovery. Fallback to using that endpoint as the base url.
>>>>>>>>> The image service for :regionOne exists but does not have any
>>>>>>>>> supported versions.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> glance api.log from controller nodes:
>>>>>>>>>
>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model
>>>>>>>>> set to 'EventletThreadPoolModel'
>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet
>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading
>>>>>>>>> SecretService
>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows
>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer
>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS
>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token
>>>>>>>>> [-] AuthToken middleware is set with
>>>>>>>>> keystone_authtoken.service_token_roles_required set to False. This is
>>>>>>>>> backwards compatible but deprecated behaviour. Please set this to True.
>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not
>>>>>>>>> load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError:
>>>>>>>>> No module named 'boto3'
>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not
>>>>>>>>> load 's3': No module named 'boto3': ModuleNotFoundError: No module named
>>>>>>>>> 'boto3'
>>>>>>>>>
>>>>>>>>> Is there a patch to fix this issue??
>>>>>>>>>
>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230221/c106772d/attachment-0001.htm>