<div dir="ltr">Hi Jhon,<div>Thank you for your response.</div><div>I never checked the /var/lib/tripleo-config/ceph directory.</div><div>I had created the ceph config file and keyring for edge in /etc/ceph/ what i needed to do was copy those config to /var/lib/tripleo-config/ceph directory.</div><div><br></div><div>Now i am able to list the images and glance stores-info gives the desired output.</div><div><br></div><div>Thank you everyone for helping on this issue.</div><div><br></div><div>With regards,</div><div>Swogat Pradhan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 22, 2023 at 6:36 PM John Fulton <<a href="mailto:johfulto@redhat.com">johfulto@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Sounds like you've confirmed that /etc/ceph/ceph.conf and<br>
/etc/ceph/client.openstack.keyring (or similar) are on your container<br>
host and functioning correctly. Next check inside the glance container<br>
that they are mounted correctly inside of it.<br>
<br>
Note this parameter and its default:<br>
<br>
<a href="https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/deployment/cephadm/ceph-client.yaml#L63" rel="noreferrer" target="_blank">https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/deployment/cephadm/ceph-client.yaml#L63</a><br>
<br>
Is the glance container mounting this way?<br>
<br>
/var/lib/tripleo-config/ceph:/etc/ceph<br>
<br>
If so then check the contents of /var/lib/tripleo-config/ceph on your<br>
container host.<br>
<br>
John<br>
<br>
On Wed, Feb 22, 2023 at 3:18 AM Swogat Pradhan<br>
<<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
><br>
> Hi Abhishek,<br>
> I cannot understand why this error is coming though 'ERROR: [errno 2] RADOS object not found (error calling conf_read_file)'<br>
><br>
> In controller 1 and Edge 1 nodes i have the config files of both sides along with the openstack and admin keyring files.<br>
> And in other controllers and edge nodes i have config files and openstack keyring of other site and admin keyring of the same site.<br>
><br>
> When i am trying to connect using cephadm shell --config ceph.conf / dcn01.conf --keyring <admin keyring> --fsid <central/edge> i am able to connect to the cluster and issue commands as well.<br>
><br>
> With regards,<br>
> Swogat Pradhan<br>
><br>
> On Wed, Feb 22, 2023 at 1:02 PM Abhishek Kekane <<a href="mailto:akekane@redhat.com" target="_blank">akekane@redhat.com</a>> wrote:<br>
>><br>
>> I can see RADOS not found error multiple times in podman_logs.<br>
>><br>
>> 2023-02-22T07:01:01.639404765+00:00 stderr F + echo 'Running command: '\''/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\'''<br>
>> 2023-02-22T07:01:01.639420550+00:00 stdout F Running command: '/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'<br>
>> 2023-02-22T07:01:01.639464013+00:00 stderr F + exec /usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf<br>
>> 2023-02-22T07:01:03.662036516+00:00 stderr F ERROR: [errno 2] RADOS object not found (error calling conf_read_file)<br>
>><br>
>> I think this might be the reason glance-api is stuck in a recursive loop.<br>
>><br>
>><br>
>> Thanks & Best Regards,<br>
>><br>
>> Abhishek Kekane<br>
>><br>
>><br>
>> On Wed, Feb 22, 2023 at 12:57 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>><br>
>>> Attaching podman logs glance_api.<br>
>>><br>
>>> With regards,<br>
>>> Swogat Pradhan<br>
>>><br>
>>> On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>><br>
>>>> Hi All,<br>
>>>> I have set up glance multi store in the central site and the same issue is present.<br>
>>>> Attaching the logs.<br>
>>>><br>
>>>> With regards,<br>
>>>> Swogat Pradhan<br>
>>>><br>
>>>> On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>> Hi,<br>
>>>>> I am currently working on client approval to run the stack update.<br>
>>>>> I will share the logs shortly.<br>
>>>>><br>
>>>>> With regards,<br>
>>>>> Swogat Pradhan<br>
>>>>><br>
>>>>> On Tue, Feb 21, 2023 at 7:58 PM John Fulton <<a href="mailto:johfulto@redhat.com" target="_blank">johfulto@redhat.com</a>> wrote:<br>
>>>>>><br>
>>>>>> I agree with Pranali and Takashi.<br>
>>>>>><br>
>>>>>> I have also seen "No module named 'boto3'" in my logs while setting up<br>
>>>>>> DCN with glance successfully. So that is not the cause of your fatal<br>
>>>>>> error. You should look in your logs for something else as the root<br>
>>>>>> cause of this issue.<br>
>>>>>><br>
>>>>>> John<br>
>>>>>><br>
>>>>>> On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <<a href="mailto:tkajinam@redhat.com" target="_blank">tkajinam@redhat.com</a>> wrote:<br>
>>>>>> ><br>
>>>>>> > I looked at tcib definitions but we have never added boto3 even in master.<br>
>>>>>> ><br>
>>>>>> > However, according to the glance_store code, I expect glance_store would be able to ignore the error<br>
>>>>>> > even if boto3 is not enabled unless you enable s3 backend somehow.<br>
>>>>>> ><br>
>>>>>> > <a href="https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224" rel="noreferrer" target="_blank">https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224</a><br>
>>>>>> ><br>
>>>>>> > If you can find the log line "Failed to load driver ..." in glance-api.log then the cause would be different<br>
>>>>>> > and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably<br>
>>>>>> > /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash.<br>
>>>>>> ><br>
>>>>>> > On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <<a href="mailto:pdeore@redhat.com" target="_blank">pdeore@redhat.com</a>> wrote:<br>
>>>>>> >><br>
>>>>>> >> This doesn't seem like a fatal error, so the issue might be something else.<br>
>>>>>> >> What podman logs glance_api shows?<br>
>>>>>> >><br>
>>>>>> >> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>>>> >>><br>
>>>>>> >>> Hi Abhishek,<br>
>>>>>> >>> Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured.<br>
>>>>>> >>><br>
>>>>>> >>> So I have been searching for a patch for tripleo.<br>
>>>>>> >>><br>
>>>>>> >>> With regards,<br>
>>>>>> >>> Swogat Pradhan<br>
>>>>>> >>><br>
>>>>>> >>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <<a href="mailto:akekane@redhat.com" target="_blank">akekane@redhat.com</a>> wrote:<br>
>>>>>> >>>><br>
>>>>>> >>>> Hi Swogat,<br>
>>>>>> >>>><br>
>>>>>> >>>> I think this looks relevant, kindly go through this and confirm.<br>
>>>>>> >>>> <a href="https://bugs.launchpad.net/kolla/+bug/1884259" rel="noreferrer" target="_blank">https://bugs.launchpad.net/kolla/+bug/1884259</a><br>
>>>>>> >>>><br>
>>>>>> >>>><br>
>>>>>> >>>> Thanks & Best Regards,<br>
>>>>>> >>>><br>
>>>>>> >>>> Abhishek Kekane<br>
>>>>>> >>>><br>
>>>>>> >>>><br>
>>>>>> >>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>>>> >>>>><br>
>>>>>> >>>>> Hi pranali,<br>
>>>>>> >>>>> Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file.<br>
>>>>>> >>>>> Right now i have reverted the config so i am not getting any erros in the glance api logs.<br>
>>>>>> >>>>> But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log:<br>
>>>>>> >>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel'<br>
>>>>>> >>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet<br>
>>>>>> >>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService<br>
>>>>>> >>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows<br>
>>>>>> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer<br>
>>>>>> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS<br>
>>>>>> >>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.<br>
>>>>>> >>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'<br>
>>>>>> >>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'<br>
>>>>>> >>>>><br>
>>>>>> >>>>> I get no error whatsoever other than the above.<br>
>>>>>> >>>>><br>
>>>>>> >>>>> With regards,<br>
>>>>>> >>>>> Swogat Pradhan<br>
>>>>>> >>>>><br>
>>>>>> >>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <<a href="mailto:pdeore@redhat.com" target="_blank">pdeore@redhat.com</a>> wrote:<br>
>>>>>> >>>>>><br>
>>>>>> >>>>>> Hey Swogat,<br>
>>>>>> >>>>>><br>
>>>>>> >>>>>> By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files.<br>
>>>>>> >>>>>> Could you please share glance-api logs from both the nodes to check more details?<br>
>>>>>> >>>>>> Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause.<br>
>>>>>> >>>>>><br>
>>>>>> >>>>>><br>
>>>>>> >>>>>><br>
>>>>>> >>>>>> Thanks,<br>
>>>>>> >>>>>> ~Pranali<br>
>>>>>> >>>>>><br>
>>>>>> >>>>>><br>
>>>>>> >>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>>>> >>>>>>><br>
>>>>>> >>>>>>> Hi Pranali,<br>
>>>>>> >>>>>>> Thank you for your response.<br>
>>>>>> >>>>>>> The glance.yaml i had shared is for the central site.<br>
>>>>>> >>>>>>> I am attaching the required files for your reference.<br>
>>>>>> >>>>>>><br>
>>>>>> >>>>>>> With regards,<br>
>>>>>> >>>>>>> Swogat Pradhan<br>
>>>>>> >>>>>>><br>
>>>>>> >>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <<a href="mailto:pdeore@redhat.com" target="_blank">pdeore@redhat.com</a>> wrote:<br>
>>>>>> >>>>>>>><br>
>>>>>> >>>>>>>> Hi Swogat,<br>
>>>>>> >>>>>>>><br>
>>>>>> >>>>>>>> The glance.yaml you've shared is for central site or edge site?<br>
>>>>>> >>>>>>>> Is it possible for you to share your glance-api.conf just to check how configuration values are set?<br>
>>>>>> >>>>>>>><br>
>>>>>> >>>>>>>><br>
>>>>>> >>>>>>>><br>
>>>>>> >>>>>>>> Thanks,<br>
>>>>>> >>>>>>>> Pranali<br>
>>>>>> >>>>>>>><br>
>>>>>> >>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>>>> >>>>>>>>><br>
>>>>>> >>>>>>>>> Additional INFO:<br>
>>>>>> >>>>>>>>> The glance_api container is in reboot loop after adding multi store config.<br>
>>>>>> >>>>>>>>> Which is why the image service has become unreachable.<br>
>>>>>> >>>>>>>>><br>
>>>>>> >>>>>>>>><br>
>>>>>> >>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br>
>>>>>> >>>>>>>>>><br>
>>>>>> >>>>>>>>>> Hi,<br>
>>>>>> >>>>>>>>>> Openstack wallaby | tripleo<br>
>>>>>> >>>>>>>>>><br>
>>>>>> >>>>>>>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation.<br>
>>>>>> >>>>>>>>>> But when ever i run stack update with the following config for multistore:<br>
>>>>>> >>>>>>>>>><br>
>>>>>> >>>>>>>>>> [stack@hkg2director ~]$ cat workplace/glance.yaml<br>
>>>>>> >>>>>>>>>> parameter_defaults:<br>
>>>>>> >>>>>>>>>> GlanceShowMultipleLocations: true<br>
>>>>>> >>>>>>>>>> GlanceBackend: rbd<br>
>>>>>> >>>>>>>>>> GlanceEnabledImportMethods: web-download,copy-image<br>
>>>>>> >>>>>>>>>> GlanceBackendID: ceph<br>
>>>>>> >>>>>>>>>> GlanceMultistoreConfig:<br>
>>>>>> >>>>>>>>>> dcn01:<br>
>>>>>> >>>>>>>>>> GlanceBackend: rbd<br>
>>>>>> >>>>>>>>>> GlanceStoreDescription: 'dcn01 rbd glance store'<br>
>>>>>> >>>>>>>>>> CephClusterName: dcn01<br>
>>>>>> >>>>>>>>>><br>
>>>>>> >>>>>>>>>> The image service becomes inaccessible. Giving the following error:<br>
>>>>>> >>>>>>>>>><br>
>>>>>> >>>>>>>>>> (overcloud) [stack@hkg2director ~]$ openstack image list<br>
>>>>>> >>>>>>>>>> Failed to contact the endpoint at <a href="https://bolt.bdxworld.com:13292" rel="noreferrer" target="_blank">https://bolt.bdxworld.com:13292</a> for discovery. Fallback to using that endpoint as the base url.<br>
>>>>>> >>>>>>>>>> Failed to contact the endpoint at <a href="https://bolt.bdxworld.com:13292" rel="noreferrer" target="_blank">https://bolt.bdxworld.com:13292</a> for discovery. Fallback to using that endpoint as the base url.<br>
>>>>>> >>>>>>>>>> The image service for :regionOne exists but does not have any supported versions.<br>
>>>>>> >>>>>>>>>><br>
>>>>>> >>>>>>>>>><br>
>>>>>> >>>>>>>>>> glance api.log from controller nodes:<br>
>>>>>> >>>>>>>>>><br>
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel'<br>
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet<br>
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService<br>
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows<br>
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer<br>
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS<br>
>>>>>> >>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.<br>
>>>>>> >>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'<br>
>>>>>> >>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'<br>
>>>>>> >>>>>>>>>><br>
>>>>>> >>>>>>>>>> Is there a patch to fix this issue??<br>
>>>>>><br>
<br>
</blockquote></div>