Open Stack

Sounds like you've confirmed that /etc/ceph/ceph.conf and
/etc/ceph/client.openstack.keyring (or similar) are on your container
host and functioning correctly. Next check inside the glance container
that they are mounted correctly inside of it.

Note this parameter and its default:

  https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/deployment/cephadm/ceph-client.yaml#L63

Is the glance container mounting this way?

  /var/lib/tripleo-config/ceph:/etc/ceph

If so then check the contents of /var/lib/tripleo-config/ceph on your
container host.

  John

On Wed, Feb 22, 2023 at 3:18 AM Swogat Pradhan
<swogatpradhan22 at gmail.com> wrote:
>
> Hi Abhishek,
> I cannot understand why this error is coming though 'ERROR: [errno 2] RADOS object not found (error calling conf_read_file)'
>
> In controller 1 and Edge 1 nodes i have the config files of both sides along with the openstack and admin keyring files.
> And in other controllers and edge nodes i have config files and openstack keyring of other site and admin keyring of the same site.
>
> When i am trying to connect using cephadm shell --config ceph.conf / dcn01.conf --keyring <admin keyring> --fsid <central/edge> i am able to connect to the cluster and issue commands as well.
>
> With regards,
> Swogat Pradhan
>
> On Wed, Feb 22, 2023 at 1:02 PM Abhishek Kekane <akekane at redhat.com> wrote:
>>
>> I can see RADOS not found error multiple times in podman_logs.
>>
>> 2023-02-22T07:01:01.639404765+00:00 stderr F + echo 'Running command: '\''/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\'''
>> 2023-02-22T07:01:01.639420550+00:00 stdout F Running command: '/usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'
>> 2023-02-22T07:01:01.639464013+00:00 stderr F + exec /usr/bin/glance-api --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf
>> 2023-02-22T07:01:03.662036516+00:00 stderr F ERROR: [errno 2] RADOS object not found (error calling conf_read_file)
>>
>> I think this might be the reason glance-api is stuck in a recursive loop.
>>
>>
>> Thanks & Best Regards,
>>
>> Abhishek Kekane
>>
>>
>> On Wed, Feb 22, 2023 at 12:57 PM Swogat Pradhan <swogatpradhan22 at gmail.com> wrote:
>>>
>>> Attaching podman logs glance_api.
>>>
>>> With regards,
>>> Swogat Pradhan
>>>
>>> On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan <swogatpradhan22 at gmail.com> wrote:
>>>>
>>>> Hi All,
>>>> I have set up glance multi store in the central site and the same issue is present.
>>>> Attaching the logs.
>>>>
>>>> With regards,
>>>> Swogat Pradhan
>>>>
>>>> On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan <swogatpradhan22 at gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>> I am currently working on client approval to run the stack update.
>>>>> I will share the logs shortly.
>>>>>
>>>>> With regards,
>>>>> Swogat Pradhan
>>>>>
>>>>> On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto at redhat.com> wrote:
>>>>>>
>>>>>> I agree with Pranali and Takashi.
>>>>>>
>>>>>> I have also seen "No module named 'boto3'" in my logs while setting up
>>>>>> DCN with glance successfully. So that is not the cause of your fatal
>>>>>> error. You should look in your logs for something else as the root
>>>>>> cause of this issue.
>>>>>>
>>>>>>   John
>>>>>>
>>>>>> On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam at redhat.com> wrote:
>>>>>> >
>>>>>> > I looked at tcib definitions but we have never added boto3 even in master.
>>>>>> >
>>>>>> > However, according to the glance_store code, I expect glance_store would be able to ignore the error
>>>>>> > even if boto3 is not enabled unless you enable s3 backend somehow.
>>>>>> >
>>>>>> > https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224
>>>>>> >
>>>>>> > If you can find the log line "Failed to load driver ..." in glance-api.log then the cause would be different
>>>>>> > and we have to check the full logs(both /var/log/containers/glance/glance-api.log and probably
>>>>>> > /var/log/containers/sdtouts/glance_api.log) to find out what is actually causing the crash.
>>>>>> >
>>>>>> > On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore at redhat.com> wrote:
>>>>>> >>
>>>>>> >> This doesn't seem like a fatal error, so the issue might be something else.
>>>>>> >> What podman logs glance_api shows?
>>>>>> >>
>>>>>> >> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <swogatpradhan22 at gmail.com> wrote:
>>>>>> >>>
>>>>>> >>> Hi Abhishek,
>>>>>> >>> Yes, this feels like the same bug, as per the comment from Lars Wiegman (lars-fuga) my glance api container is in reboot loop, when glance multi store is configured.
>>>>>> >>>
>>>>>> >>> So I have been searching for a patch for tripleo.
>>>>>> >>>
>>>>>> >>> With regards,
>>>>>> >>> Swogat Pradhan
>>>>>> >>>
>>>>>> >>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane at redhat.com> wrote:
>>>>>> >>>>
>>>>>> >>>> Hi Swogat,
>>>>>> >>>>
>>>>>> >>>> I think this looks relevant, kindly go through this and confirm.
>>>>>> >>>> https://bugs.launchpad.net/kolla/+bug/1884259
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> Thanks & Best Regards,
>>>>>> >>>>
>>>>>> >>>> Abhishek Kekane
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <swogatpradhan22 at gmail.com> wrote:
>>>>>> >>>>>
>>>>>> >>>>> Hi pranali,
>>>>>> >>>>> Those configs('central', 'cinder_store') were added when i had started deploying the cloud, those parameters have been removed from the yaml now, but exist in the config file.
>>>>>> >>>>> Right now i have reverted the config so i am not getting any erros in the glance api logs.
>>>>>> >>>>> But when i configure the multistore i get the following errors on both sides (central and edge) in glance_api.log:
>>>>>> >>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel'
>>>>>> >>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet
>>>>>> >>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService
>>>>>> >>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows
>>>>>> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer
>>>>>> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS
>>>>>> >>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.
>>>>>> >>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
>>>>>> >>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
>>>>>> >>>>>
>>>>>> >>>>> I get no error whatsoever other than the above.
>>>>>> >>>>>
>>>>>> >>>>> With regards,
>>>>>> >>>>> Swogat Pradhan
>>>>>> >>>>>
>>>>>> >>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore at redhat.com> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>> Hey Swogat,
>>>>>> >>>>>>
>>>>>> >>>>>> By looking at the conf files you've shared, not sure from where the extra store sections 'central', 'cinder_store' are added since you have not added those in any of your yaml files.
>>>>>> >>>>>> Could you please share glance-api logs from both the nodes to check more details?
>>>>>> >>>>>> Probably the HA configuration is causing this kind of issue but need more information related to deployment & log to understand the root cause.
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> Thanks,
>>>>>> >>>>>> ~Pranali
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <swogatpradhan22 at gmail.com> wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>> Hi Pranali,
>>>>>> >>>>>>> Thank you for your response.
>>>>>> >>>>>>> The glance.yaml i had shared is for the central site.
>>>>>> >>>>>>> I am attaching the required files for your reference.
>>>>>> >>>>>>>
>>>>>> >>>>>>> With regards,
>>>>>> >>>>>>> Swogat Pradhan
>>>>>> >>>>>>>
>>>>>> >>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore at redhat.com> wrote:
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Hi Swogat,
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> The glance.yaml you've shared is for central site or edge site?
>>>>>> >>>>>>>> Is it possible for you to share your glance-api.conf just to check how configuration values are set?
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Thanks,
>>>>>> >>>>>>>> Pranali
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <swogatpradhan22 at gmail.com> wrote:
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Additional INFO:
>>>>>> >>>>>>>>> The glance_api container is in reboot loop after adding multi store config.
>>>>>> >>>>>>>>> Which is why the image service has become unreachable.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <swogatpradhan22 at gmail.com> wrote:
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> Hi,
>>>>>> >>>>>>>>>> Openstack wallaby | tripleo
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I am trying to setup DCN where i want to setup a glance multistore config in both central and edge sites as per the documentation.
>>>>>> >>>>>>>>>> But when ever i run stack update with the following config for multistore:
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> [stack at hkg2director ~]$ cat workplace/glance.yaml
>>>>>> >>>>>>>>>> parameter_defaults:
>>>>>> >>>>>>>>>>   GlanceShowMultipleLocations: true
>>>>>> >>>>>>>>>>   GlanceBackend: rbd
>>>>>> >>>>>>>>>>   GlanceEnabledImportMethods: web-download,copy-image
>>>>>> >>>>>>>>>>   GlanceBackendID: ceph
>>>>>> >>>>>>>>>>   GlanceMultistoreConfig:
>>>>>> >>>>>>>>>>     dcn01:
>>>>>> >>>>>>>>>>       GlanceBackend: rbd
>>>>>> >>>>>>>>>>       GlanceStoreDescription: 'dcn01 rbd glance store'
>>>>>> >>>>>>>>>>       CephClusterName: dcn01
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> The image service becomes inaccessible. Giving the following error:
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> (overcloud) [stack at hkg2director ~]$ openstack image list
>>>>>> >>>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url.
>>>>>> >>>>>>>>>> Failed to contact the endpoint at https://bolt.bdxworld.com:13292 for discovery. Fallback to using that endpoint as the base url.
>>>>>> >>>>>>>>>> The image service for :regionOne exists but does not have any supported versions.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> glance api.log from controller nodes:
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model set to 'EventletThreadPoolModel'
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading SecretService
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer
>>>>>> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS
>>>>>> >>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.
>>>>>> >>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
>>>>>> >>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not load 's3': No module named 'boto3': ModuleNotFoundError: No module named 'boto3'
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> Is there a patch to fix this issue??
>>>>>>