Hi,
I am currently working on client approval to run the stack update.
I will share the logs shortly.

With regards,
Swogat Pradhan

On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto@redhat.com> wrote:
I agree with Pranali and Takashi.

I have also seen "No module named 'boto3'" in my logs while setting up
DCN with glance successfully. So that is not the cause of your fatal
error. You should look in your logs for something else as the root
cause of this issue.

  John

On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam@redhat.com> wrote:
>
> I looked at tcib definitions but we have never added boto3 even in master.
>
> However, according to the glance_store code, I expect glance_store would be able to ignore the error
> even if boto3 is not enabled unless you enable s3 backend somehow.
>
> https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224
>
> If you can find the log line "Failed to load driver ..." in glance-api.log then the cause would be different
> and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably
> /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash.
>
> On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore@redhat.com> wrote:
>>
>> This doesn't seem like a fatal error, so the issue might be something else.
>> What podman logs glance_api shows?
>>
>> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
>>>
>>> Hi Abhishek,
>>> Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured.
>>>
>>> So I have been searching for a patch for tripleo.
>>>
>>> With regards,
>>> Swogat Pradhan
>>>
>>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane@redhat.com> wrote:
>>>>
>>>> Hi Swogat,
>>>>
>>>> I think this looks relevant, kindly go through this and confirm.
>>>> https://bugs.launchpad.net/kolla/+bug/1884259
>>>>
>>>>
>>>> Thanks & Best Regards,
>>>>
>>>> Abhishek Kekane
>>>>
>>>>
>>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
>>>>>
>>>>> Hi pranali,
>>>>> Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file.
>>>>> Right now i have reverted the config so i am not getting any erros in the glance api logs.
>>>>> But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log:
>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel'
>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet
>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService
>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows
>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer
>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS
>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.
>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
>>>>>
>>>>> I get no error whatsoever other than the above.
>>>>>
>>>>> With regards,
>>>>> Swogat Pradhan
>>>>>
>>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore@redhat.com> wrote:
>>>>>>
>>>>>> Hey Swogat,
>>>>>>
>>>>>> By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files.
>>>>>> Could you please share glance-api logs from both the nodes to check more details?
>>>>>> Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> ~Pranali
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Pranali,
>>>>>>> Thank you for your response.
>>>>>>> The glance.yaml i had shared is for the central site.
>>>>>>> I am attaching the required files for your reference.
>>>>>>>
>>>>>>> With regards,
>>>>>>> Swogat Pradhan
>>>>>>>
>>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore@redhat.com> wrote:
>>>>>>>>
>>>>>>>> Hi Swogat,
>>>>>>>>
>>>>>>>> The glance.yaml you've shared is for central site or edge site?
>>>>>>>> Is it possible for you to share your glance-api.conf just to check how configuration values are set?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Pranali
>>>>>>>>
>>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Additional INFO:
>>>>>>>>> The glance_api container is in reboot loop after adding multi store config.
>>>>>>>>> Which is why the image service has become unreachable.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Openstack wallaby | tripleo
>>>>>>>>>>
>>>>>>>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation.
>>>>>>>>>> But when ever i run stack update with the following config for multistore:
>>>>>>>>>>
>>>>>>>>>> [stack@hkg2director ~]$ cat workplace/glance.yaml
>>>>>>>>>> parameter_defaults:
>>>>>>>>>>   GlanceShowMultipleLocations: true
>>>>>>>>>>   GlanceBackend: rbd
>>>>>>>>>>   GlanceEnabledImportMethods: web-download,copy-image
>>>>>>>>>>   GlanceBackendID: ceph
>>>>>>>>>>   GlanceMultistoreConfig:
>>>>>>>>>>     dcn01:
>>>>>>>>>>       GlanceBackend: rbd
>>>>>>>>>>       GlanceStoreDescription: 'dcn01 rbd glance store'
>>>>>>>>>>       CephClusterName: dcn01
>>>>>>>>>>
>>>>>>>>>> The image service becomes inaccessible. Giving the following error:
>>>>>>>>>>
>>>>>>>>>> (overcloud) [stack@hkg2director ~]$ openstack image list
>>>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url.
>>>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url.
>>>>>>>>>> The image service for :regionOne exists but does not have any supported versions.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> glance api.log from controller nodes:
>>>>>>>>>>
>>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel'
>>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet
>>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService
>>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows
>>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer
>>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS
>>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.
>>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
>>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
>>>>>>>>>>
>>>>>>>>>> Is there a patch to fix this issue??