Open Stack

Hi Jhon,
Thank you for your response.
I never checked the /var/lib/tripleo-config/ceph directory.
I had created the ceph config file and keyring for edge in /etc/ceph/ what
i needed to do was copy those config to /var/lib/tripleo-config/ceph
directory.

Now i am able to list the images and glance stores-info gives the desired
output.

Thank you everyone for helping on this issue.

With regards,
Swogat Pradhan

On Wed, Feb 22, 2023 at 6:36 PM John Fulton <johfulto at redhat.com> wrote:

> Sounds like you've confirmed that /etc/ceph/ceph.conf and
> /etc/ceph/client.openstack.keyring (or similar) are on your container
> host and functioning correctly. Next check inside the glance container
> that they are mounted correctly inside of it.
>
> Note this parameter and its default:
>
>
> https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/deployment/cephadm/ceph-client.yaml#L63
>
> Is the glance container mounting this way?
>
>   /var/lib/tripleo-config/ceph:/etc/ceph
>
> If so then check the contents of /var/lib/tripleo-config/ceph on your
> container host.
>
>   John
>
> On Wed, Feb 22, 2023 at 3:18 AM Swogat Pradhan
> <swogatpradhan22 at gmail.com> wrote:
> >
> > Hi Abhishek,
> > I cannot understand why this error is coming though 'ERROR: [errno 2]
> RADOS object not found (error calling conf_read_file)'
> >
> > In controller 1 and Edge 1 nodes i have the config files of both sides
> along with the openstack and admin keyring files.
> > And in other controllers and edge nodes i have config files and
> openstack keyring of other site and admin keyring of the same site.
> >
> > When i am trying to connect using cephadm shell --config ceph.conf /
> dcn01.conf --keyring <admin keyring> --fsid <central/edge> i am able to
> connect to the cluster and issue commands as well.
> >
> > With regards,
> > Swogat Pradhan
> >
> > On Wed, Feb 22, 2023 at 1:02 PM Abhishek Kekane <akekane at redhat.com>
> wrote:
> >>
> >> I can see RADOS not found error multiple times in podman_logs.
> >>
> >> 2023-02-22T07:01:01.639404765+00:00 stderr F + echo 'Running command:
> '\''/usr/bin/glance-api --config-file /etc/glance/glance-api.conf
> --config-file /etc/glance/glance-image-import.conf'\'''
> >> 2023-02-22T07:01:01.639420550+00:00 stdout F Running command:
> '/usr/bin/glance-api --config-file /etc/glance/glance-api.conf
> --config-file /etc/glance/glance-image-import.conf'
> >> 2023-02-22T07:01:01.639464013+00:00 stderr F + exec /usr/bin/glance-api
> --config-file /etc/glance/glance-api.conf --config-file
> /etc/glance/glance-image-import.conf
> >> 2023-02-22T07:01:03.662036516+00:00 stderr F ERROR: [errno 2] RADOS
> object not found (error calling conf_read_file)
> >>
> >> I think this might be the reason glance-api is stuck in a recursive
> loop.
> >>
> >>
> >> Thanks & Best Regards,
> >>
> >> Abhishek Kekane
> >>
> >>
> >> On Wed, Feb 22, 2023 at 12:57 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>
> >>> Attaching podman logs glance_api.
> >>>
> >>> With regards,
> >>> Swogat Pradhan
> >>>
> >>> On Wed, Feb 22, 2023 at 12:46 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>
> >>>> Hi All,
> >>>> I have set up glance multi store in the central site and the same
> issue is present.
> >>>> Attaching the logs.
> >>>>
> >>>> With regards,
> >>>> Swogat Pradhan
> >>>>
> >>>> On Tue, Feb 21, 2023 at 10:04 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> I am currently working on client approval to run the stack update.
> >>>>> I will share the logs shortly.
> >>>>>
> >>>>> With regards,
> >>>>> Swogat Pradhan
> >>>>>
> >>>>> On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto at redhat.com>
> wrote:
> >>>>>>
> >>>>>> I agree with Pranali and Takashi.
> >>>>>>
> >>>>>> I have also seen "No module named 'boto3'" in my logs while setting
> up
> >>>>>> DCN with glance successfully. So that is not the cause of your fatal
> >>>>>> error. You should look in your logs for something else as the root
> >>>>>> cause of this issue.
> >>>>>>
> >>>>>>   John
> >>>>>>
> >>>>>> On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <
> tkajinam at redhat.com> wrote:
> >>>>>> >
> >>>>>> > I looked at tcib definitions but we have never added boto3 even
> in master.
> >>>>>> >
> >>>>>> > However, according to the glance_store code, I expect
> glance_store would be able to ignore the error
> >>>>>> > even if boto3 is not enabled unless you enable s3 backend somehow.
> >>>>>> >
> >>>>>> >
> https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224
> >>>>>> >
> >>>>>> > If you can find the log line "Failed to load driver ..." in
> glance-api.log then the cause would be different
> >>>>>> > and we have to check the full logs(both
> /var/log/containers/glance/glance-api.log and probably
> >>>>>> > /var/log/containers/sdtouts/glance_api.log) to find out what is
> actually causing the crash.
> >>>>>> >
> >>>>>> > On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore at redhat.com>
> wrote:
> >>>>>> >>
> >>>>>> >> This doesn't seem like a fatal error, so the issue might be
> something else.
> >>>>>> >> What podman logs glance_api shows?
> >>>>>> >>
> >>>>>> >> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>>> >>>
> >>>>>> >>> Hi Abhishek,
> >>>>>> >>> Yes, this feels like the same bug, as per the comment from Lars
> Wiegman (lars-fuga) my glance api container is in reboot loop, when glance
> multi store is configured.
> >>>>>> >>>
> >>>>>> >>> So I have been searching for a patch for tripleo.
> >>>>>> >>>
> >>>>>> >>> With regards,
> >>>>>> >>> Swogat Pradhan
> >>>>>> >>>
> >>>>>> >>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <
> akekane at redhat.com> wrote:
> >>>>>> >>>>
> >>>>>> >>>> Hi Swogat,
> >>>>>> >>>>
> >>>>>> >>>> I think this looks relevant, kindly go through this and
> confirm.
> >>>>>> >>>> https://bugs.launchpad.net/kolla/+bug/1884259
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>> >>>> Thanks & Best Regards,
> >>>>>> >>>>
> >>>>>> >>>> Abhishek Kekane
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>> >>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>>> >>>>>
> >>>>>> >>>>> Hi pranali,
> >>>>>> >>>>> Those configs('central', 'cinder_store') were added when i
> had started deploying the cloud, those parameters have been removed from
> the yaml now, but exist in the config file.
> >>>>>> >>>>> Right now i have reverted the config so i am not getting any
> erros in the glance api logs.
> >>>>>> >>>>> But when i configure the multistore i get the following
> errors on both sides (central and edge) in glance_api.log:
> >>>>>> >>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool
> model set to 'EventletThreadPoolModel'
> >>>>>> >>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading
> KWallet
> >>>>>> >>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading
> SecretService
> >>>>>> >>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading
> Windows
> >>>>>> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading
> chainer
> >>>>>> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading
> macOS
> >>>>>> >>>>> 2023-02-13 13:02:20.571 7 WARNING
> keystonemiddleware.auth_token [-] AuthToken middleware is set with
> keystone_authtoken.service_token_roles_required set to False. This is
> backwards compatible but deprecated behaviour. Please set this to True.
> >>>>>> >>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could
> not load 'glance.store.s3.Store': No module named 'boto3':
> ModuleNotFoundError: No module named 'boto3'
> >>>>>> >>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could
> not load 's3': No module named 'boto3': ModuleNotFoundError: No module
> named 'boto3'
> >>>>>> >>>>>
> >>>>>> >>>>> I get no error whatsoever other than the above.
> >>>>>> >>>>>
> >>>>>> >>>>> With regards,
> >>>>>> >>>>> Swogat Pradhan
> >>>>>> >>>>>
> >>>>>> >>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <
> pdeore at redhat.com> wrote:
> >>>>>> >>>>>>
> >>>>>> >>>>>> Hey Swogat,
> >>>>>> >>>>>>
> >>>>>> >>>>>> By looking at the conf files you've shared, not sure from
> where the extra store sections 'central', 'cinder_store' are added since
> you have not added those in any of your yaml files.
> >>>>>> >>>>>> Could you please share glance-api logs from both the nodes
> to check more details?
> >>>>>> >>>>>> Probably the HA configuration is causing this kind of issue
> but need more information related to deployment & log to understand the
> root cause.
> >>>>>> >>>>>>
> >>>>>> >>>>>>
> >>>>>> >>>>>>
> >>>>>> >>>>>> Thanks,
> >>>>>> >>>>>> ~Pranali
> >>>>>> >>>>>>
> >>>>>> >>>>>>
> >>>>>> >>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>>> >>>>>>>
> >>>>>> >>>>>>> Hi Pranali,
> >>>>>> >>>>>>> Thank you for your response.
> >>>>>> >>>>>>> The glance.yaml i had shared is for the central site.
> >>>>>> >>>>>>> I am attaching the required files for your reference.
> >>>>>> >>>>>>>
> >>>>>> >>>>>>> With regards,
> >>>>>> >>>>>>> Swogat Pradhan
> >>>>>> >>>>>>>
> >>>>>> >>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <
> pdeore at redhat.com> wrote:
> >>>>>> >>>>>>>>
> >>>>>> >>>>>>>> Hi Swogat,
> >>>>>> >>>>>>>>
> >>>>>> >>>>>>>> The glance.yaml you've shared is for central site or edge
> site?
> >>>>>> >>>>>>>> Is it possible for you to share your glance-api.conf just
> to check how configuration values are set?
> >>>>>> >>>>>>>>
> >>>>>> >>>>>>>>
> >>>>>> >>>>>>>>
> >>>>>> >>>>>>>> Thanks,
> >>>>>> >>>>>>>> Pranali
> >>>>>> >>>>>>>>
> >>>>>> >>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>>> >>>>>>>>>
> >>>>>> >>>>>>>>> Additional INFO:
> >>>>>> >>>>>>>>> The glance_api container is in reboot loop after adding
> multi store config.
> >>>>>> >>>>>>>>> Which is why the image service has become unreachable.
> >>>>>> >>>>>>>>>
> >>>>>> >>>>>>>>>
> >>>>>> >>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>>> >>>>>>>>>>
> >>>>>> >>>>>>>>>> Hi,
> >>>>>> >>>>>>>>>> Openstack wallaby | tripleo
> >>>>>> >>>>>>>>>>
> >>>>>> >>>>>>>>>> I am trying to setup DCN where i want to setup a glance
> multistore config in both central and edge sites as per the documentation.
> >>>>>> >>>>>>>>>> But when ever i run stack update with the following
> config for multistore:
> >>>>>> >>>>>>>>>>
> >>>>>> >>>>>>>>>> [stack at hkg2director ~]$ cat workplace/glance.yaml
> >>>>>> >>>>>>>>>> parameter_defaults:
> >>>>>> >>>>>>>>>>   GlanceShowMultipleLocations: true
> >>>>>> >>>>>>>>>>   GlanceBackend: rbd
> >>>>>> >>>>>>>>>>   GlanceEnabledImportMethods: web-download,copy-image
> >>>>>> >>>>>>>>>>   GlanceBackendID: ceph
> >>>>>> >>>>>>>>>>   GlanceMultistoreConfig:
> >>>>>> >>>>>>>>>>     dcn01:
> >>>>>> >>>>>>>>>>       GlanceBackend: rbd
> >>>>>> >>>>>>>>>>       GlanceStoreDescription: 'dcn01 rbd glance store'
> >>>>>> >>>>>>>>>>       CephClusterName: dcn01
> >>>>>> >>>>>>>>>>
> >>>>>> >>>>>>>>>> The image service becomes inaccessible. Giving the
> following error:
> >>>>>> >>>>>>>>>>
> >>>>>> >>>>>>>>>> (overcloud) [stack at hkg2director ~]$ openstack image list
> >>>>>> >>>>>>>>>> Failed to contact the endpoint at
> https://bolt.bdxworld.com:13292 for discovery. Fallback to using that
> endpoint as the base url.
> >>>>>> >>>>>>>>>> Failed to contact the endpoint at
> https://bolt.bdxworld.com:13292 for discovery. Fallback to using that
> endpoint as the base url.
> >>>>>> >>>>>>>>>> The image service for :regionOne exists but does not
> have any supported versions.
> >>>>>> >>>>>>>>>>
> >>>>>> >>>>>>>>>>
> >>>>>> >>>>>>>>>> glance api.log from controller nodes:
> >>>>>> >>>>>>>>>>
> >>>>>> >>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-]
> Threadpool model set to 'EventletThreadPoolModel'
> >>>>>> >>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-]
> Loading KWallet
> >>>>>> >>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-]
> Loading SecretService
> >>>>>> >>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-]
> Loading Windows
> >>>>>> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-]
> Loading chainer
> >>>>>> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-]
> Loading macOS
> >>>>>> >>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING
> keystonemiddleware.auth_token [-] AuthToken middleware is set with
> keystone_authtoken.service_token_roles_required set to False. This is
> backwards compatible but deprecated behaviour. Please set this to True.
> >>>>>> >>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-]
> Could not load 'glance.store.s3.Store': No module named 'boto3':
> ModuleNotFoundError: No module named 'boto3'
> >>>>>> >>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-]
> Could not load 's3': No module named 'boto3': ModuleNotFoundError: No
> module named 'boto3'
> >>>>>> >>>>>>>>>>
> >>>>>> >>>>>>>>>> Is there a patch to fix this issue??
> >>>>>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230222/f40047bd/attachment-0001.htm>