<div dir="ltr">Hi Abhishek,<div>I cannot understand why this error is coming though '<span style="font-family:verdana,sans-serif">ERROR: [errno 2] RADOS object not found (error calling conf_read_file)</span>'</div><div><br></div><div>In controller 1 and Edge 1 nodes i have the config files of both sides along with the openstack and admin keyring files.</div><div>And in other controllers and edge nodes i have config files and openstack keyring of other site and admin keyring of the same site.</div><div><br></div><div>When i am trying to connect using cephadm shell --config ceph.conf / dcn01.conf --keyring <admin keyring> --fsid <central/edge> i am able to connect to the cluster and issue commands as well.</div><div><br></div><div>With regards,</div><div>Swogat Pradhan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 22, 2023 at 1:02 PM Abhishek Kekane <<a href="mailto:akekane@redhat.com">akekane@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">I can see RADOS not found error multiple times in podman_logs.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">2023-02-22T07:01:01.639404765+00:00 stderr F + echo 'Running command: '\''/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\'''<br>2023-02-22T07:01:01.639420550+00:00 stdout F Running command: '/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'<br>2023-02-22T07:01:01.639464013+00:00 stderr F + exec /usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf<br>2023-02-22T07:01:03.662036516+00:00 stderr F ERROR: [errno 2] RADOS object not found (error calling conf_read_file)</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">I think this might be the reason glance-api is stuck in a recursive loop.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div><div dir="ltr"><div dir="ltr"><div>Thanks & Best Regards,<br><br></div>Abhishek Kekane<br></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 22, 2023 at 12:57 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Attaching podman logs glance_api.<div><br></div><div>With regards,</div><div>Swogat Pradhan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi All,<div>I have set up glance multi store in the central site and the same issue is present.</div><div>Attaching the logs.</div><div><br></div><div>With regards,</div><div>Swogat Pradhan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div>I am currently working on client approval to run the stack update.</div><div>I will share the logs shortly.</div><div><br></div><div>With regards,</div><div>Swogat Pradhan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 21, 2023 at 7:58 PM John Fulton <<a href="mailto:johfulto@redhat.com" target="_blank">johfulto@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I agree with Pranali and Takashi.<br>
<br>
I have also seen "No module named 'boto3'" in my logs while setting up<br>
DCN with glance successfully. So that is not the cause of your fatal<br>
error. You should look in your logs for something else as the root<br>
cause of this issue.<br>
<br>
John<br>
<br>
On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <<a href="mailto:tkajinam@redhat.com" target="_blank">tkajinam@redhat.com</a>> wrote:<br>
><br>
> I looked at tcib definitions but we have never added boto3 even in master.<br>
><br>
> However, according to the glance_store code, I expect glance_store would be able to ignore the error<br>
> even if boto3 is not enabled unless you enable s3 backend somehow.<br>
><br>
> <a href="https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224" rel="noreferrer" target="_blank">https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224</a><br>
><br>
> If you can find the log line "Failed to load driver ..." in glance-api.log then the cause would be different<br>
> and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably<br>
> /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash.<br>
><br>
> On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <<a href="mailto:pdeore@redhat.com" target="_blank">pdeore@redhat.com</a>> wrote:<br>
>><br>
>> This doesn't seem like a fatal error, so the issue might be something else.<br>
>> What podman logs glance_api shows?<br>
>><br>
>> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>><br>
>>> Hi Abhishek,<br>
>>> Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured.<br>
>>><br>
>>> So I have been searching for a patch for tripleo.<br>
>>><br>
>>> With regards,<br>
>>> Swogat Pradhan<br>
>>><br>
>>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <<a href="mailto:akekane@redhat.com" target="_blank">akekane@redhat.com</a>> wrote:<br>
>>>><br>
>>>> Hi Swogat,<br>
>>>><br>
>>>> I think this looks relevant, kindly go through this and confirm.<br>
>>>> <a href="https://bugs.launchpad.net/kolla/+bug/1884259" rel="noreferrer" target="_blank">https://bugs.launchpad.net/kolla/+bug/1884259</a><br>
>>>><br>
>>>><br>
>>>> Thanks & Best Regards,<br>
>>>><br>
>>>> Abhishek Kekane<br>
>>>><br>
>>>><br>
>>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>> Hi pranali,<br>
>>>>> Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file.<br>
>>>>> Right now i have reverted the config so i am not getting any erros in the glance api logs.<br>
>>>>> But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log:<br>
>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel'<br>
>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet<br>
>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService<br>
>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows<br>
>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer<br>
>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS<br>
>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.<br>
>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'<br>
>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'<br>
>>>>><br>
>>>>> I get no error whatsoever other than the above.<br>
>>>>><br>
>>>>> With regards,<br>
>>>>> Swogat Pradhan<br>
>>>>><br>
>>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <<a href="mailto:pdeore@redhat.com" target="_blank">pdeore@redhat.com</a>> wrote:<br>
>>>>>><br>
>>>>>> Hey Swogat,<br>
>>>>>><br>
>>>>>> By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files.<br>
>>>>>> Could you please share glance-api logs from both the nodes to check more details?<br>
>>>>>> Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause.<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> Thanks,<br>
>>>>>> ~Pranali<br>
>>>>>><br>
>>>>>><br>
>>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>>>>><br>
>>>>>>> Hi Pranali,<br>
>>>>>>> Thank you for your response.<br>
>>>>>>> The glance.yaml i had shared is for the central site.<br>
>>>>>>> I am attaching the required files for your reference.<br>
>>>>>>><br>
>>>>>>> With regards,<br>
>>>>>>> Swogat Pradhan<br>
>>>>>>><br>
>>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <<a href="mailto:pdeore@redhat.com" target="_blank">pdeore@redhat.com</a>> wrote:<br>
>>>>>>>><br>
>>>>>>>> Hi Swogat,<br>
>>>>>>>><br>
>>>>>>>> The glance.yaml you've shared is for central site or edge site?<br>
>>>>>>>> Is it possible for you to share your glance-api.conf just to check how configuration values are set?<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> Thanks,<br>
>>>>>>>> Pranali<br>
>>>>>>>><br>
>>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>>>>>>><br>
>>>>>>>>> Additional INFO:<br>
>>>>>>>>> The glance_api container is in reboot loop after adding multi store config.<br>
>>>>>>>>> Which is why the image service has become unreachable.<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>>>>>>>><br>
>>>>>>>>>> Hi,<br>
>>>>>>>>>> Openstack wallaby | tripleo<br>
>>>>>>>>>><br>
>>>>>>>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation.<br>
>>>>>>>>>> But when ever i run stack update with the following config for multistore:<br>
>>>>>>>>>><br>
>>>>>>>>>> [stack@hkg2director ~]$ cat workplace/glance.yaml<br>
>>>>>>>>>> parameter_defaults:<br>
>>>>>>>>>> GlanceShowMultipleLocations: true<br>
>>>>>>>>>> GlanceBackend: rbd<br>
>>>>>>>>>> GlanceEnabledImportMethods: web-download,copy-image<br>
>>>>>>>>>> GlanceBackendID: ceph<br>
>>>>>>>>>> GlanceMultistoreConfig:<br>
>>>>>>>>>> dcn01:<br>
>>>>>>>>>> GlanceBackend: rbd<br>
>>>>>>>>>> GlanceStoreDescription: 'dcn01 rbd glance store'<br>
>>>>>>>>>> CephClusterName: dcn01<br>
>>>>>>>>>><br>
>>>>>>>>>> The image service becomes inaccessible. Giving the following error:<br>
>>>>>>>>>><br>
>>>>>>>>>> (overcloud) [stack@hkg2director ~]$ openstack image list<br>
>>>>>>>>>> Failed to contact the endpoint at <a href="https://bolt.bdxworld.com:13292" rel="noreferrer" target="_blank">https://bolt.bdxworld.com:13292</a> for discovery. Fallback to using that endpoint as the base url.<br>
>>>>>>>>>> Failed to contact the endpoint at <a href="https://bolt.bdxworld.com:13292" rel="noreferrer" target="_blank">https://bolt.bdxworld.com:13292</a> for discovery. Fallback to using that endpoint as the base url.<br>
>>>>>>>>>> The image service for :regionOne exists but does not have any supported versions.<br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>> glance api.log from controller nodes:<br>
>>>>>>>>>><br>
>>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel'<br>
>>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet<br>
>>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService<br>
>>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows<br>
>>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer<br>
>>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS<br>
>>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.<br>
>>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'<br>
>>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'<br>
>>>>>>>>>><br>
>>>>>>>>>> Is there a patch to fix this issue??<br>
<br>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>