Open Stack

Hi,
I am currently working on client approval to run the stack update.
I will share the logs shortly.

With regards,
Swogat Pradhan

On Tue, Feb 21, 2023 at 7:58 PM John Fulton <johfulto at redhat.com> wrote:

> I agree with Pranali and Takashi.
>
> I have also seen "No module named 'boto3'" in my logs while setting up
> DCN with glance successfully. So that is not the cause of your fatal
> error. You should look in your logs for something else as the root
> cause of this issue.
>
>   John
>
> On Tue, Feb 21, 2023 at 2:20 AM Takashi Kajinami <tkajinam at redhat.com>
> wrote:
> >
> > I looked at tcib definitions but we have never added boto3 even in
> master.
> >
> > However, according to the glance_store code, I expect glance_store would
> be able to ignore the error
> > even if boto3 is not enabled unless you enable s3 backend somehow.
> >
> >
> https://github.com/openstack/glance_store/blob/64e25979a904e468e1e89d8727e9554662596acf/glance_store/backend.py#L222-L224
> >
> > If you can find the log line "Failed to load driver ..." in
> glance-api.log then the cause would be different
> > and we have to check the full logs(both
> /var/log/containers/glance/glance-api.log and probably
> > /var/log/containers/sdtouts/glance_api.log) to find out what is actually
> causing the crash.
> >
> > On Tue, Feb 21, 2023 at 4:05 PM Pranali Deore <pdeore at redhat.com> wrote:
> >>
> >> This doesn't seem like a fatal error, so the issue might be something
> else.
> >> What podman logs glance_api shows?
> >>
> >> On Tue, Feb 21, 2023 at 12:14 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>
> >>> Hi Abhishek,
> >>> Yes, this feels like the same bug, as per the comment from Lars
> Wiegman (lars-fuga) my glance api container is in reboot loop, when glance
> multi store is configured.
> >>>
> >>> So I have been searching for a patch for tripleo.
> >>>
> >>> With regards,
> >>> Swogat Pradhan
> >>>
> >>> On Tue, Feb 21, 2023 at 12:09 PM Abhishek Kekane <akekane at redhat.com>
> wrote:
> >>>>
> >>>> Hi Swogat,
> >>>>
> >>>> I think this looks relevant, kindly go through this and confirm.
> >>>> https://bugs.launchpad.net/kolla/+bug/1884259
> >>>>
> >>>>
> >>>> Thanks & Best Regards,
> >>>>
> >>>> Abhishek Kekane
> >>>>
> >>>>
> >>>> On Tue, Feb 21, 2023 at 11:58 AM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>>
> >>>>> Hi pranali,
> >>>>> Those configs('central', 'cinder_store') were added when i had
> started deploying the cloud, those parameters have been removed from the
> yaml now, but exist in the config file.
> >>>>> Right now i have reverted the config so i am not getting any erros
> in the glance api logs.
> >>>>> But when i configure the multistore i get the following errors on
> both sides (central and edge) in glance_api.log:
> >>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool model
> set to 'EventletThreadPoolModel'
> >>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading KWallet
> >>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading
> SecretService
> >>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading Windows
> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading chainer
> >>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS
> >>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token [-]
> AuthToken middleware is set with
> keystone_authtoken.service_token_roles_required set to False. This is
> backwards compatible but deprecated behaviour. Please set this to True.
> >>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could not
> load 'glance.store.s3.Store': No module named 'boto3': ModuleNotFoundError:
> No module named 'boto3'
> >>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could not
> load 's3': No module named 'boto3': ModuleNotFoundError: No module named
> 'boto3'
> >>>>>
> >>>>> I get no error whatsoever other than the above.
> >>>>>
> >>>>> With regards,
> >>>>> Swogat Pradhan
> >>>>>
> >>>>> On Tue, Feb 21, 2023 at 11:44 AM Pranali Deore <pdeore at redhat.com>
> wrote:
> >>>>>>
> >>>>>> Hey Swogat,
> >>>>>>
> >>>>>> By looking at the conf files you've shared, not sure from where the
> extra store sections 'central', 'cinder_store' are added since you have not
> added those in any of your yaml files.
> >>>>>> Could you please share glance-api logs from both the nodes to check
> more details?
> >>>>>> Probably the HA configuration is causing this kind of issue but
> need more information related to deployment & log to understand the root
> cause.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> ~Pranali
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Feb 20, 2023 at 4:45 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>>>>
> >>>>>>> Hi Pranali,
> >>>>>>> Thank you for your response.
> >>>>>>> The glance.yaml i had shared is for the central site.
> >>>>>>> I am attaching the required files for your reference.
> >>>>>>>
> >>>>>>> With regards,
> >>>>>>> Swogat Pradhan
> >>>>>>>
> >>>>>>> On Mon, Feb 20, 2023 at 4:29 PM Pranali Deore <pdeore at redhat.com>
> wrote:
> >>>>>>>>
> >>>>>>>> Hi Swogat,
> >>>>>>>>
> >>>>>>>> The glance.yaml you've shared is for central site or edge site?
> >>>>>>>> Is it possible for you to share your glance-api.conf just to
> check how configuration values are set?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Pranali
> >>>>>>>>
> >>>>>>>> On Tue, Feb 14, 2023 at 12:35 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Additional INFO:
> >>>>>>>>> The glance_api container is in reboot loop after adding multi
> store config.
> >>>>>>>>> Which is why the image service has become unreachable.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mon, Feb 13, 2023 at 6:39 PM Swogat Pradhan <
> swogatpradhan22 at gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>> Openstack wallaby | tripleo
> >>>>>>>>>>
> >>>>>>>>>> I am trying to setup DCN where i want to setup a glance
> multistore config in both central and edge sites as per the documentation.
> >>>>>>>>>> But when ever i run stack update with the following config for
> multistore:
> >>>>>>>>>>
> >>>>>>>>>> [stack at hkg2director ~]$ cat workplace/glance.yaml
> >>>>>>>>>> parameter_defaults:
> >>>>>>>>>>   GlanceShowMultipleLocations: true
> >>>>>>>>>>   GlanceBackend: rbd
> >>>>>>>>>>   GlanceEnabledImportMethods: web-download,copy-image
> >>>>>>>>>>   GlanceBackendID: ceph
> >>>>>>>>>>   GlanceMultistoreConfig:
> >>>>>>>>>>     dcn01:
> >>>>>>>>>>       GlanceBackend: rbd
> >>>>>>>>>>       GlanceStoreDescription: 'dcn01 rbd glance store'
> >>>>>>>>>>       CephClusterName: dcn01
> >>>>>>>>>>
> >>>>>>>>>> The image service becomes inaccessible. Giving the following
> error:
> >>>>>>>>>>
> >>>>>>>>>> (overcloud) [stack at hkg2director ~]$ openstack image list
> >>>>>>>>>> Failed to contact the endpoint at
> https://bolt.bdxworld.com:13292 for discovery. Fallback to using that
> endpoint as the base url.
> >>>>>>>>>> Failed to contact the endpoint at
> https://bolt.bdxworld.com:13292 for discovery. Fallback to using that
> endpoint as the base url.
> >>>>>>>>>> The image service for :regionOne exists but does not have any
> supported versions.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> glance api.log from controller nodes:
> >>>>>>>>>>
> >>>>>>>>>> 2023-02-13 13:02:19.878 7 INFO glance.async_ [-] Threadpool
> model set to 'EventletThreadPoolModel'
> >>>>>>>>>> 2023-02-13 13:02:19.969 7 INFO keyring.backend [-] Loading
> KWallet
> >>>>>>>>>> 2023-02-13 13:02:19.977 7 INFO keyring.backend [-] Loading
> SecretService
> >>>>>>>>>> 2023-02-13 13:02:19.982 7 INFO keyring.backend [-] Loading
> Windows
> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading
> chainer
> >>>>>>>>>> 2023-02-13 13:02:19.983 7 INFO keyring.backend [-] Loading macOS
> >>>>>>>>>> 2023-02-13 13:02:20.571 7 WARNING keystonemiddleware.auth_token
> [-] AuthToken middleware is set with
> keystone_authtoken.service_token_roles_required set to False. This is
> backwards compatible but deprecated behaviour. Please set this to True.
> >>>>>>>>>> 2023-02-13 13:02:20.711 7 ERROR stevedore.extension [-] Could
> not load 'glance.store.s3.Store': No module named 'boto3':
> ModuleNotFoundError: No module named 'boto3'
> >>>>>>>>>> 2023-02-13 13:02:20.719 7 ERROR stevedore.extension [-] Could
> not load 's3': No module named 'boto3': ModuleNotFoundError: No module
> named 'boto3'
> >>>>>>>>>>
> >>>>>>>>>> Is there a patch to fix this issue??
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230221/534e1594/attachment-0001.htm>